prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.



Report 2: Medicare pays for laboratory tests that are performed individually or in a group  For individual tests, laboratories submit claims to Medicare for each test they perform that is on the CLFS; tests are identified using a billing code  Prior to the implementation of PAMA in 2018, the payment rates on the CLFS were based on rates charged for laboratory tests in 1984 through 1985 adjusted for inflation  Additionally, 57 geographic jurisdictions had their own fee schedules for laboratory tests  CMS used the 57 separate fee schedules to calculate a national limitation amount, which served as the maximum payment for individual laboratory tests  Thus, the payment rate for an individual test was the lesser of the amount claimed by the laboratory, the local fee for a geographic area, or the national limitation amount for a particular test  Medicare pays bundled payment rates for certain laboratory tests that are performed as a group, called panel tests  Panel tests can be divided into two categories—those without billing codes and those with billing codes  Panel tests without billing codes are composed of at least 2 of 23 distinct component tests  Additionally, there are 7 specific combinations of these 23 component tests that are commonly used and have their own billing code  Prior to 2018, Medicare paid for both types of panel tests (those without or with a billing code) using a bundled rate based on the number of tests performed, with modest payment increases for each additional test conducted  For example, in 2017, Medicare paid $7 15 for panel tests with two component tests and $9 12 for panel tests with 3 component tests, with a maximum bundled payment rate of $16 64 for all 23 component tests  Prior to 2018, the Medicare Administrative Contractors would count the number of tests performed before determining the appropriate bundled payment rate  For those panel tests with a billing code, the payment rate was the same if laboratories used the associated billing code for the panel test or listed each of the component tests separately  After PAMA’s implementation in 2018, the 57 separate fee schedules for individual laboratory tests were replaced with a single national fee schedule  The payment rates for this single national fee schedule were based on private-payer rates for laboratory tests paid from January 1, 2016 through June 30, 2016  Specifically, the payment rate for an individual test was generally based on the median private-payer rates for a given test, weighted by test volume  Payment for panel tests also changed in 2018  For panel tests without billing codes, Medicare Administrative Contractors no longer counted the number of component tests performed to determine the bundled payment rate; instead, Medicare paid the separate rate for each component test in the panel  For panel tests with a billing code, the payment rate depended on how the laboratory submitted the claim  If a laboratory used the billing code associated with the panel test, Medicare paid the bundled payment rate for that billing code  If a laboratory submitted a claim for the panel test, but listed each of the component tests separately instead of using the panel test’s billing code, Medicare paid the individual payment rate for each component test  Table 1 below summarizes the changes to payment rates before and after 2018  Multiple types of laboratories receive payment under Medicare  The three laboratory types that received the most revenue from the CLFS in 2016 were independent laboratories, hospital-outreach laboratories, and physician-office laboratories  (See table 2 ) Estimates of the size of the total U S  laboratory market vary  For example, the Healthcare Fraud Prevention Partnership estimated that the laboratory industry received $87 billion in revenue in 2017, while another market report estimated the laboratory industry received $75 billion in revenue in 2016  Similar to Medicare, the three laboratory types that generally receive the most revenue overall are independent laboratories, hospital-outreach laboratories, and physician-office laboratories, when laboratory tests performed in hospital inpatient and outpatient settings were excluded  Estimates of revenue received by these laboratories also vary  For example, in recent years, estimates of the share of laboratory industry revenue generated by independent laboratories ranged from 37 percent to 54 percent  Additionally, estimates of revenue generated by hospital- outreach laboratories recently ranged from 21 to 35 percent, and physician-office laboratories ranged from 4 to 11 percent of total laboratory industry revenue  Private-payer rates for laboratory tests conducted by the three largest laboratory types generally vary by type and other characteristics, according to market reports and the laboratory industry officials we interviewed  Independent laboratories  These laboratories generally receive lower private-payer rates than other types of laboratories, according to industry officials we interviewed  Market reports we reviewed noted that about half of the independent laboratory market is dominated by two national laboratories and that these national laboratories provide more competitive pricing by performing a large volume of tests at one time  Medicare accounted for a smaller proportion of the revenue earned by these two national laboratories (12 percent), compared to other laboratories, according to another market report we reviewed  In contrast, a different market report noted that smaller, independent laboratories tend to earn more of their revenue from Medicare (34 percent)  Hospital-outreach laboratories  These hospital-affiliated laboratories typically receive relatively higher private-payer rates, according to industry officials we interviewed  Although hospital- outreach laboratories perform tests similar to other laboratories, they can obtain above-average payment rates by leveraging the market power of their affiliated hospital when negotiating rates with private payers, according to industry officials and market reports  Hospital-outreach laboratories generally receive about 25 to 30 percent of their revenue from the Medicare CLFS  Physician-office laboratories  Physician-office laboratories typically receive higher private-payer rates than independent laboratories, according to a recent analysis by a laboratory industry association  This industry association also noted that the cost structure to operate in a setting such as a physician-office laboratory is different than in large independent laboratories, as the physician-office laboratory is unable to conduct a large number of tests at one time  Officials from another industry association we interviewed said that payment rates for these laboratories are generally dependent on the size of the physician practice group  These same officials told us that larger physician groups (e g , 10 or more physicians) typically negotiate higher rates from private payers than smaller physician groups  Most physician-office laboratories received less than $25,000 in revenue per year from Medicare, according to CMS  Additionally, in 2013, the Department of Health and Human Services Office of Inspector General found that Medicare’s payment rates on the CLFS were higher than rates paid by some private health insurance plans  Specifically, it found that Medicare rates for laboratory tests were 18 percent to 30 percent higher than rates paid by certain insurers under health benefits plans for federal employees  Definition of Applicable Laboratories Required to Report Private-Payer Data to CMS CMS defined applicable laboratories as those meeting four criteria: (1) they met the definition of laboratory under regulations implementing the Clinical Laboratory Improvement Amendments of 1988; (2) they billed Medicare Part B under their own Medicare billing number, also called the national provider identifier; (3) more than 50 percent of their total Medicare revenues came from the Clinical Laboratory Fee Schedule (CLFS) and/or the Physician Fee Schedule; and (4) they received at least $12,500 in Medicare revenue from the CLFS from January 1, 2016, through June 30, 2016  CMS analyzed private-payer data it collected from about 2,000 laboratories to develop new payment rates for individual laboratory tests on the CLFS  PAMA defined laboratories required to report private-payer data, called applicable laboratories, as laboratories that meet certain criteria  (See sidebar ) Applicable laboratories with their own specific billing number, the NPI, submitted these data to CMS  If one organization operated multiple applicable laboratories, each with its own NPI, then the organization could report data to CMS for multiple applicable laboratories  CMS collected data from applicable laboratories on payments they received from private payers during the first half of 2016  Specifically, CMS collected data on (1) the unique billing code associated with a laboratory test; (2) the private-payer rate for each laboratory test for which final payment was made during the data collection period (January 1, 2016, through June 30, 2016); and (3) the volume of tests performed for each unique billing code at that private- payer rate  For the data CMS collected between January 1, 2017, and May 30, 2017, CMS relied on the entities reporting to CMS to attest to the completeness and accuracy of the data they submitted  CMS relied on each laboratory to identify whether or not it was an applicable laboratory and took steps to assist laboratories in meeting reporting requirements  According to CMS officials, they relied on laboratories to self-identify as applicable laboratories because they were unable to accurately identify the number of laboratories required to report  To assist laboratories, CMS issued multiple guidance documents to the industry outlining the criteria for being an applicable laboratory and describing the type of data CMS intended to collect  CMS also conducted educational calls when the proposed and final rules were issued and prior to the data collection period  CMS officials told us they conducted additional outreach activities, including holding conference calls with national laboratory associations and attending professional conferences  Officials said they used these outreach activities in addition to the guidance issued to inform laboratories of the reporting requirements for applicable laboratories, for example  In addition, CMS established a revenue threshold of $12,500 in an effort to reduce the reporting burden for entities that receive a relatively small amount of revenues under the CLFS  In its final rule, CMS noted that it expected that many of the laboratories that would be below this revenue threshold and, thus exempt from reporting data to CMS, would be physician-office laboratories  CMS also chose to use the NPI in its definition of applicable laboratory in the final rule to allow hospital- outreach laboratories that use their own NPI to submit data to the agency  In its proposed rule, CMS suggested using an alternative identification number to the NPI  However, according to the final rule, CMS chose to use the NPI in its definition of applicable laboratory to allow those hospital-outreach laboratories billing using their own NPI to submit private-payer data to the agency  According to CMS, at the end of the 5-month submission period, the agency had received data from approximately 2,000 applicable laboratories, representing a volume of almost 248 million laboratory tests; these data accounted for about $31 billion in revenue from private payers  CMS reported that the data it collected included private-payer rates for 96 percent of the 1,347 eligible billing codes on the CLFS  CMS used these data to calculate a median, private-payer rate, weighted by volume and phased in this change by limiting payment-rate reductions to 10 percent per year  Beginning in 2018, these new payment rates served as the single, national payment rate for individual laboratory tests  These payment rates were also used for the individual, component tests that make up panel tests and were used when laboratories billed Medicare for panel tests by listing the component tests separately  In general, the median payments rates, weighted for volume, that CMS calculated were lower than Medicare’s previous payment rates for most laboratory tests  According to our analysis, these median payment rates were lower than the corresponding 2017 CLFS national limitation amounts (the maximum that CMS would pay for laboratory tests) for approximately 88 percent of tests  Figure 1 below describes the percentage difference between these median payment rates and Medicare’s 2017 national limitation amounts for laboratory tests  The final payment rates that CMS calculated, which included the 10- percent, phased in, payment-rate reductions, will remain in effect until December 31, 2020; PAMA requires CMS to calculate new payment rates for the CLFS every 3 years  Reporting entities will next be required to submit data on private-payer rates to CMS in early 2020, for final payments made from January 1, 2019 through June 30, 2019  PAMA capped any reductions for the second 3-year cycle after implementation to a maximum of 15 percent per year  CMS did not collect private-payer data from all laboratories required to report this information and did not estimate how much data was not reported by these laboratories, according to agency officials  CMS relied on laboratories to determine whether they met data reporting requirements and submit data accordingly  CMS emphasized the importance of receiving data from all laboratories required to report by stating that it is critical that CMS collect complete data on private-payer rates in order to set accurate Medicare rates  However, agency officials told us that CMS did not receive data from all laboratories required to report  They also told us that CMS did not have the information available to estimate how much data was missing because not all laboratories reported or the extent to which the data collected were representative of all of the data that laboratories were required to report  Prior to collecting private-payer data, CMS estimated that laboratories subject to reporting requirements would receive more than 90 percent of CLFS expenditures to physician-office laboratories and independent laboratories  Specifically, based on its analysis of 2013 Medicare expenditures, CMS estimated that reporting requirements would apply to the laboratories that received 92 percent of CLFS payments to physician- office laboratories and 99 percent of CLFS payments to independent laboratories  After laboratories reported private-payer data, we analyzed the share of CLFS expenditures received by the laboratories that reported  Our analysis found that CMS collected data from laboratories that received the majority of CLFS payments to physician-office, independent, and other non-hospital laboratories in 2016  However, the laboratories that reported private-payer data received less than 70 percent of CLFS expenditures to physician-office, independent, and other non-hospital laboratories  Specifically, using Medicare claims data, we calculated that CMS collected data from laboratories that received 68 percent of 2016 CLFS payments to physician-office, independent, and other non-hospital laboratories  Although it did not collect complete data, CMS concluded that it collected sufficient private-payer data to set Medicare payment rates and that collecting more data from additional laboratories that were required to report would not significantly affect Medicare expenditures  This conclusion was based, in part, on a sensitivity analyses that CMS conducted of the effects that collecting certain types and amounts of additional data would have on weighted median private-payer rates and the effects those rates could have on Medicare payment rates and, thus, expenditures  Results from these analyses showed that Medicare expenditures based on the CLFS would have changed by 2 percent or less after collecting more data from the various types of laboratories  For example, CMS estimated that doubling the amount of private-payer data from physician-office laboratories would increase expenditures by 2 percent and collecting ten times as much data from hospital outreach laboratories would increase expenditures by 1 percent  (See fig  2 ) PAMA’s 10-percent limit on annual payment-rate reductions likely reduced the effect that incomplete private-payer data could have on the CLFS because this limit applied to most Medicare payment rates for laboratory tests  As demonstrated in figure 1, while 59 percent of tests had median private-payer rates that were at least 30 percent less than their respective 2017 national limitation amounts, CMS published Medicare rates for these tests for 2018 through 2020 that were reduced by only 10 percent per year as a result of this limit  For example, a hypothetical laboratory test with a 2017 CLFS national limitation amount of $10 00 and a median private-payer rate of $7 00 would result in CLFS rates of $9 00 in 2018, $8 10 in 2019, and $7 29 in 2020  Changes to median private-payer rates due to collecting more complete data or eliminating inaccurate data would have no effect on Medicare payment rates from 2018 through 2020 for this hypothetical test if they resulted in new median rates of $7 29 or less  Our analysis of the potential effects that collecting data from additional laboratories could have had on Medicare payment rates and expenditures found that the effect of CMS not collecting complete data would likely have been greater absent PAMA’s limits on annual reductions to Medicare payment rates  As a result, CMS may face challenges setting accurate Medicare rates if it does not collect complete data from all laboratories required to report in the future when PAMA allows for greater annual payment-rate reductions  To conduct this analysis, we used the private-payer data CMS collected to analyze the range of effects that collecting additional data could have on Medicare expenditures, assuming 2016 utilization rates remain constant  The extent of these effects depends on the amount of additional data CMS would need to collect to obtain complete data and whether the payment rates in these additional data would have been greater or less than the medians of the rates reported  For example, we estimated that if CMS needed to collect 20 percent more data for its collection to be complete, doing so could increase Medicare CLFS expenditures from 2018 through 2020 by as much as 3 percent or reduce them by as much as 3 percent depending on the payment rates in these additional data  However, if annual limits to Medicare payment-rate reductions were not applied, collecting these additional data could increase CLFS expenditures by as much as 9 percent or reduce them by as much as 9 percent  (See fig  3 and app  II for additional information about these estimates ) As demonstrated in figure 2, CMS did analyze how collecting certain types and amounts of data from additional laboratories would affect Medicare expenditures  However, without valid estimates of how much more data these additional laboratories were required to report and how much these data would change median payment rates, it remains unknown whether CMS’s analyses estimate the actual risk of setting Medicare payment rates that do not reflect private-payer rates from all applicable laboratories, as mandated by PAMA  CMS could have compared the data it collected with independent information on the payment rates laboratories were required to report, for example  The independent information could be estimated by auditing a random sample of laboratories or could be estimated using data from third-party vendors, if these vendors could supply relevant and reliable information  We found that CMS mitigated challenges to setting accurate Medicare payment rates by identifying, analyzing, and responding to potentially inaccurate private-payer data  CMS addressed potentially inaccurate private-payer data and other data that CMS determined did not meet reporting requirements  CMS removed or replaced data from four reporting entities that appeared to have or confirmed having reported revenue—which is the payment rate multiplied by the volume of tests paid at that rate—instead of payment rates  We estimated that if CMS had included these data that CLFS expenditures from 2018 through 2020 would have increased by 7 percent  CMS removed data it determined were reported in error including duplicate submissions and submissions with payment rates of $0 00  We estimated that removing these data will change CLFS expenditures from 2018 through 2020 by less than one percent  CMS identified four other types of potentially inaccurate data that it determined would not significantly impact Medicare payment rates or expenditures and did not exclude them from calculations of median private-payer rates  CMS considered the following potentially inaccurate data to have met its reporting requirements: 1  data from 57 entities that reported particularly high rates in at least 60 percent of their data, 2  data from 12 entities that reported particularly low rates in at least 50 percent of their data, 3  data with payment rates that were 10 times greater than the 2017 national limitation amounts or 10 times less than these amounts, and 4  data from laboratories that may not have met the $12,500 low- expenditure threshold or that reported data from a hospital NPI instead of a laboratory NPI  We found that each of these four types of potentially inaccurate data would have changed estimated Medicare CLFS expenditures from 2018 through 2020 by 1 percent or less if CMS had instead excluded the data  To conduct this analysis, we recalculated Medicare rates after excluding each type of data and estimated Medicare expenditures assuming 2016 rates of utilization  Although weighted median private-payer rates were lower than Medicare’s 2017 national limitation amounts for 88 percent of tests, we estimated the total Medicare expenditures based on the 2018 CLFS would likely increase by 3 percent ($225 million overall) compared to 2016 expenditures, assuming test utilization remained at 2016 levels  This increase in estimated expenditures is due, in part, to CMS’s use of above-average payment rates as a baseline to calculate payment rates for those laboratory tests affected by PAMA’s annual payment-rate reduction limit of 10 percent  (See fig  4 ) When applying the 10-percent payment-rate reduction limit, CMS used as its starting point the 2017 national limitation amounts in order to set a single, national payment rate for each laboratory test  Thus, the Medicare payment rate for a test in 2018 could not be less than 90 percent of the test’s 2017 national limitation amount  However, prior to 2018, some payment rates were commonly lower than the national limitation amounts because they were based on the lesser of (1) the amount billed on claims, (2) the local fee for a geographic area, or (3) a national limitation amount, and because panel tests had different bundled payment rates  As a result, by reducing payment rates from national limitation amounts, CMS did not always reduce rates from what Medicare actually paid  Panel tests, in particular, frequently received bundled payment rates that differed substantially from national limitation amounts associated with their billing codes prior to 2018  We compared national limitation amounts, which represent maximum Medicare payment rates for tests, with the average amounts Medicare allowed for payment in 2016, which reflect actual Medicare payment rates  For example, figure 5 below shows that the 2017 national limitation amount for comprehensive metabolic panel tests ($14 49) was substantially higher than both the average amount Medicare allowed for payment in 2016 ($11 45) and the median payment rate laboratories reported receiving from private payers ($9 08)  As a result, using the 2017 national limitation amount as a basis for payment reductions caused Medicare’s payment rate to increase from an average allowed amount of $11 45 in 2016, to a payment rate of $13 04 in 2018, instead of decreasing towards a lower median private- payer rate of $9 08  By increasing average payment rates rather than phasing in reductions to rates, CMS’s implementation may lead to paying more than necessary for some tests  Federal standards for internal control for information and communications require agency management to use quality information to achieve its objectives  Basing reductions on national limitation amounts rather than more relevant information on how much Medicare actually paid—such as the average allowable amounts in 2016, for example—could result in Medicare paying more than necessary by $733 million from 2018 through 2020, according to our estimates  In implementing PAMA, CMS eliminated bundled rates for panel tests that lack billing codes and started paying separately for each component test instead  CMS also implemented the 2018 CLFS in a manner that could lead to unbundling payment rates for panel tests with billing codes  If payment rates for all panel tests were unbundled, we estimated that Medicare expenditures could increase by $218 million for panel tests that lack billing codes and by as much as $10 1 billion for panel tests with billing codes from 2018 through 2020  CMS also estimated that there could be significant risks of paying more than necessary associated with unbundling and has taken initial steps to monitor these risks and explore possible responses, but had not yet responded to these risks as of July 2018  CMS Unbundled Payment Rates for Panel Tests without Billing Codes Beginning in 2018, CMS no longer uses bundled payment rates for panel tests without billing codes and instead pays laboratories individual payments for each component test that comprises these panel tests  However, CMS staff and members of its advisory panel discussed concerns with this approach  At an advisory panel meeting in 2016, CMS staff relayed concerns from stakeholders that CMS would not be able to collect valid data on private-payer rates for these panel tests  According to agency staff, stakeholders had informed CMS that private payers commonly use bundled payment rates for these panel tests, but laboratories would only be able to report unbundled payment rates for individual component tests  We estimated that unbundling these payment rates would increase Medicare expenditures from 2018 through 2020 by $218 million in comparison to the estimated Medicare expenditures over the same time period based on Medicare’s 2016 utilization and allowable amounts  For example, under the 2016 CLFS, Medicare paid approximately 435,000 claims for panel tests that included the laboratory tests assay of creatinine (HCPCS code 82565) and assay of urea nitrogen (HCPCS code 84520) at an average bundled payment rate of $6 82  In contrast, under the 2018 CLFS, these two component tests are reimbursed individually at $6 33 and $4 88, respectively, or $11 21 combined—a 63 percent increase  Despite concerns about the validity of available private-payer data on component tests for panel tests without billing codes, CMS used these data to set payment rates for component tests  CMS officials told us that they stopped using bundled payment rates for these panel tests because it is not clear that CMS has the authority to combine the individual component tests into groups for bundled payment as it did before 2018 due to PAMA’s reference to payments for each test  However, in July 2018, CMS officials told us the agency was reviewing its authority regarding this issue  CMS officials told us they are exploring alternative approaches that could limit increases to Medicare expenditures but had not yet determined what additional legal authority would be needed, if any, and did not know when CMS would make this determination  Agency officials told us that CMS has taken initial steps to monitor unbundling and explore possible responses, including the following: Monitoring unbundling: CMS has begun monitoring changes in panel test utilization, payment rates, and expenditures associated with its implementation of PAMA, according to officials  For example, CMS officials told us that preliminary data indicated that Medicare payments for individual component tests of panel tests has increased substantially in 2018, but, as of July 2018, it was too early to draw conclusions from these data because laboratories have up to one year to submit claims for tests  Collecting input on alternatives: In 2016, a subcommittee of an advisory panel that CMS established reviewed Medicare’s use of bundled payment rates for panel tests and published different approaches for CMS to consider implementing in combination with other changes to implement PAMA  CMS’s Implementation of PAMA May Have Allowed Unbundling of Payment Rates for Panel Tests with Billing Codes Beginning in 2018, laboratories that submit claims for any of the seven panel tests with billing codes by using the billing codes for the individual component tests now receive the payment rate for each component test, rather than the bundled rate  Prior to 2018, laboratories could submit claims for these panel tests either by using the specific codes for panel tests or by billing separately for each of the component tests, and, regardless of how laboratories submitted claims, Medicare Administrative Contractors would pay bundled payment rates based on how many of the 23 component tests were conducted  However, CMS instructed Medicare Administrative Contractors to stop bundling payment rates for tests that are billed individually on claims rather than billed on claims using codes for panel tests, beginning in 2018  CMS did so because it was not clear that CMS had the authority to combine the individual component tests into groups for bundled payment as it did before 2018 due to PAMA’s reference to payments for individual tests, according to agency officials  This change could potentially have a large effect on Medicare spending  For example, if a laboratory submitted a claim individually for the 14 component tests that comprise a comprehensive metabolic panel it would receive a payment of $81 91, a 528 percent increase from the 2018 Medicare bundled payment rate of $13 04 for this panel test  (See fig  6 ) Improving how reductions to payment rates for panel tests are phased in could mitigate, but not completely counteract, the effect of unbundling these payment rates  For example, for the comprehensive metabolic panel test described in figure 6, basing maximum reductions on 2016 average allowable amounts would result in a 2018 Medicare bundled payment rate of $10 31 instead of $13 04 and individual payment rates for the 14 component tests that total $56 06—a 32 percent decrease from $81 91 that Medicare would otherwise pay  If the payment rate for each panel test with a billing code were unbundled, we estimated that Medicare expenditures for these tests from 2018 through 2020 could reach $13 5 billion, a $10 1 billion increase from the $3 3 billion we estimated Medicare would spend using the bundled payment rates in the CLFS  Similarly, prior to implementing PAMA, CMS estimated that Medicare expenditures to physician-office, independent, and other non-hospital laboratories could potentially increase as much as $2 5 billion in 2018, alone if it paid for the same number of panel tests with billing codes as it did in 2016 but paid for each component test individually  These estimates represent an upper limit on the increased expenditures that could occur if every laboratory stopped using panel test billing codes and instead used the billing codes for individual component tests  We do not know the extent to which laboratories will stop filing claims using panel test billing codes  CMS officials also told us that they were aware of the risks associated with paying for the individual component tests instead of the bundled payment rate for a panel test with a billing code  However, CMS guidance, which was effective in 2018, continued to allow laboratories to use the billing codes for individual component tests rather than the billing code for the panel  CMS officials explained that this was due to PAMA’s reference to payments for individual tests, similar to CMS’s decision to stop paying bundled rates for panel tests without billing codes  At the time we did our work, CMS had not implemented a response to these risks but had taken some initial steps to monitor unbundling and consider alternative approaches to Medicare payment rates for these tests  HHS provided additional information on planned activities to address these risks in its written comments on a draft of this report  (See app  III ) CMS collected data on private-payer rates from laboratories that were required to report these data, but not all laboratories complied with the reporting requirement, and the extent of noncompliance remains unclear  PAMA’s provision directing CMS to phase in payment-rate reductions to Medicare payment rates likely moderates the potential adverse effects of incomplete private-payer data  However, in the future, failing to collect complete data could substantially affect Medicare payment rates because private-payer rates alone will determine Medicare payment rates  In addition, we estimated that Medicare expenditures on laboratory tests will be $733 million higher from 2018 through 2020, because CMS started phasing in payment-rate reductions from national limitation amounts instead of more relevant data on actual payment rates, such as average allowable amounts  Finally, changes to payment rates, billing practices, and testing practices could increase Medicare expenditures by as much as $10 3 billion from 2018 through 2020, if CMS does not address the risks associated with unbundling payment rates for panel tests  Agency officials indicated that it was unclear if PAMA limited CMS’s ability to combine individual component tests into groups for bundled payment, and, as of July 2018, CMS was reviewing this matter but did not know when it would make a determination  We are making the following three recommendations to CMS: The Administrator of CMS should take steps to collect all of the data from all laboratories that are required to report  If only partial data can be collected, CMS should estimate how incomplete data would affect Medicare payment rates and address any significant challenges to setting accurate Medicare rates  (Recommendation 1) The Administrator of CMS should phase in payment-rate reductions that start from the actual payment rates Medicare paid prior to 2018 rather than the national limitation amounts  CMS should revise these rates as soon as practicable to prevent paying more than necessary  (Recommendation 2) The Administrator of CMS should use bundled rates for panel tests, consistent with its practice prior to 2018, rather than paying for them individually; if necessary, the Administrator of CMS should seek legislative authority to do so  (Recommendation 3) We provided a draft of this report to HHS for review and comment  HHs provided written comments, which are reproduced in appendix III  HHS also provided technical comments, which we incorporated as appropriate  HHS concurred with our first recommendation to take steps to collect all data from laboratories required to report and commented that it is evaluating ways to increase reporting  In particular, in a November 2018 final rule, HHS changed the definition of an applicable laboratory, which it expects will increase the number of laboratories required to report data on private-payer rates to the agency  HHS neither agreed nor disagreed with our second recommendation to phase in payment-rate reductions that start from the actual payment rates Medicare paid prior to 2018  HHS noted that any changes to the phasing in of payment-rate reductions would need to be implemented through rulemaking  We estimated that by using the national limitation amounts as a starting point for these reductions, Medicare expenditures would increase by $733 million from 2018 through 2020  For this reason, we continue to believe CMS should revise these rates as soon as practicable and through whatever mechanism CMS determines appropriate  HHS neither agreed nor disagreed with our third recommendation to use bundled rates for panel tests  However, HHS commented that it is taking steps to address this issue  More specifically, for panel tests with billing codes, HHS is working to implement an automated process to identify claims for panel tests that should receive bundled payments, similar to the process used to bundle payment rates for these panel tests prior to PAMA’s implementation and anticipates implementing this change by the summer of 2019  In addition, HHS posted guidance on November 14, 2018, stating that the panel tests with billing codes, laboratories should submit claims using the corresponding code rather than the codes for the separate component tests beginning in 2019  To reduce the potential of paying more than necessary, we believe it is important that CMS implement its proposed automated process to allow for these payments as soon as possible  In contrast, for panel tests without billing codes, HHS commented that it is continuing to review its authority and considering other approaches to payment for these panel tests, such as adding codes to the CLFS  We estimate that unbundling the payment for these panel tests could increase Medicare expenditures by $218 million from 2018 through 2020 compared to expenditures based on Medicare’s 2016 utilization, and the actual amount could be higher if utilization increases  For this reason, we believe CMS should implement bundled payment rates for these panel tests to avoid excess payments  We are sending copies of this report to the appropriate congressional committees and the Administrator of CMS  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix IV  Appendix I: Table of Key Dates Related to Developing the New Payment Rates for the 2018 Clinical Laboratory Fee Schedule Event Centers for Medicare and Medicaid Services (CMS) issued the CLFS proposed rule  CMS issued responses to frequently asked questions regarding the CLFS proposed rule  CMS issued the CLFS final rule  CMS issued responses to frequently asked questions regarding the CLFS final rule  CMS held the joint Annual Laboratory Public Meeting and Medicare Advisory Panel on Clinical Diagnostic Laboratory Tests meeting  CMS issued laboratory billing codes subject to data collection and reporting  CMS issued guidance to laboratories for collecting and reporting data  CMS held a Medicare Advisory Panel on Clinical Diagnostic Laboratory Tests meeting  CMS issued the CLFS data reporting template  CMS collected data on (1) the billing code associated with a laboratory test; (2) the private-payer rate for each laboratory test for which final payment was made during the data collection period (i e , January 1, 2016, through June 30, 2016); and (3) the volume of tests performed for each billing code at that private-payer rate  CMS issued additional guidance for laboratories as the data collection period began  CMS issued the CLFS fee-for-service data collection user’s manual  CMS issued revised guidance to laboratories for collecting and reporting data  CMS held a Medicare Advisory Panel on Clinical Diagnostic Laboratory Tests meeting  CMS released the proposed CLFS rates  CMS held a Medicare Advisory Panel on Clinical Diagnostic Laboratory Tests meeting  Deadline for stakeholders to submit comments on the proposed CLFS rates to CMS  CMS issued the final CLFS rates  New CLFS rates became effective  Table 4 below demonstrates the challenges the Centers for Medicare & Medicaid Services (CMS) faces in setting accurate Medicare payment rates to the extent it does not collect complete data from laboratories on private-payer rates  Specifically, the table shows the potential effect that collecting additional data for each laboratory test could have on Medicare expenditures and how this effect could vary depending on (1) the amount of additional data collected, (2) payment rates in the additional data, and (3) limits to annual reductions in Medicare payment rates  These limits are in place from 2018 through 2023 to phase in changes to payment rates  In addition to the contact named above, Martin T  Gahart, Assistant Director; Gay Hee Lee, Analyst-in-Charge; Kaitlin Farquharson, Sandra George, Dan Lee, Elizabeth T  Morrison, Laurie Pachter, Vikki Porter, and Russell Voth made key contributions to this report 
Report 2:  As part of this exercise, the Secret Service also compared its employee work-life offerings (e g , on-site childcare and telework program) to those available at other agencies  In addition, the Secret Service developed and implemented action plans for improving employee morale  Among these action plans, for example, the component implemented a student loan repayment program and expanded its tuition assistance program’s eligibility requirements  Establish and track metrics of success for improving employee morale, and report to agency leadership on progress improving morale—fully implemented  The Secret Service tracked metrics for improving employee morale and reported the results to leadership  For example, the component tracked metrics on the percentage of the workforce, including IT staff, that participated in the student loan repayment and tuition assistance programs  In addition, the Chief Strategy Officer reported to the Chief Operating Officer the results related to meeting those metrics  Maintain leadership support and commitment to ensure continued progress in improving employee morale, and demonstrate sustained improvement in morale—partly implemented  Secret Service leadership developed and implemented initiatives that demonstrated their commitment to improving the morale of the Secret Service’s workforce  For example, since 2014, the Secret Service had worked with a contractor to identify ways to improve the morale of its entire workforce, including IT staff  However, as of June 2018, the Secret Service was unable to demonstrate that it had sustained improvement in the morale of the component’s IT staff  In particular, the component was only able to provide IT workforce-specific results from one employee morale assessment that was conducted subsequent to the consolidation of this workforce into OCIO in March 2017  These results were from an assessment conducted by the component’s Inspection Division in December 2017 (the assessment found that the majority of the Secret Service’s IT employees rated their morale as “very good” or “excellent ”) While the component also provided certain employee morale results from the Office of Personnel Management’s Federal Employee Viewpoint Survey in 2017, these results were not specific to the IT workforce  Instead, this workforce’s results were combined with those from staff in another Secret Service division  According to OCIO officials, the results were combined because, at the time of the survey, the IT workforce was administratively identified as being part of that other division  OCIO officials stated that, going forward, they plan to continue to assess the morale of the IT workforce on an annual basis as part of the Federal Employee Viewpoint Survey  In addition, the officials stated that OCIO-specific results may be available as part of the 2018 survey results, which the officials expect to receive by September 2018  By measuring employee satisfaction on an annual basis, the Secret Service should have increased knowledge of whether its initiatives that are aimed at improving employee morale are in fact increasing employee satisfaction  Agencies can use performance management systems as a tool to foster a results-oriented organizational culture that links individual performance to organizational goals  We have previously identified numerous leading practices related to performance management that are intended to enhance performance and ensure individual accountability  Among the performance management practices, agencies should (1) establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance, (2) explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals, and (3) periodically provide individuals with regular performance feedback  The Secret Service substantially implemented the selected three leading practices associated with the performance management workforce area  Specifically, the component fully implemented one of the three practices and partly implemented the other two practices  Table 12 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices  Establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance—partly implemented  The Secret Service’s performance management process requires leadership to make meaningful distinctions between levels of staff performance  In particular, the component’s performance plans for IT staff, which are developed by the Office of Human Resources and tailored by OCIO, as necessary, specify the criteria that leadership use to determine if an individual has met or exceeded the expectations associated with each competency identified in their respective performance plan  The performance plans include pre-established, department-wide competencies that are set by DHS, as well as occupational series-specific goals that may be updated by the Secret Service  However, because OCIO did not fully define and document all of its technical competency needs for the IT workforce, as discussed earlier, the Secret Service’s performance plans for IT staff did not include performance expectations related to the full set of technical competencies required for their respective positions  In addition, because OCIO officials were unable to specify a time frame for when they will identify all of the technical competency needs for the IT workforce (as previously discussed), the officials were also unable to specify a time frame for when they would update the IT workforce’s performance plans to include those relevant technical competencies  Until OCIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed, the office will be limited in its ability to provide IT staff with a complete assessment of their performance  In addition, Secret Service management will have limited knowledge of the extent to which IT staff are meeting all relevant technical competencies  Explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals—partly implemented  The Secret Service’s performance plans for IT staff identified certain goals that appeared to be related to organizational goals and objectives  For example, the performance plan for the Telecommunications Specialist occupational series (which is one of the series included in OCIO’s IT workforce) identified a goal for staff to support the voice, wireless, radio, satellite, and video systems serving the Secret Service’s protective and investigative mission  This performance plan goal appeared to be related to the component’s strategic goal on Advanced Technology, which included an objective to create the infrastructure needed to fulfill mission responsibilities  However, the Secret Service was unable to provide documentation that explicitly showed how individual employee performance links to organizational goals, such as a mapping of the goals identified in employee performance plans to organizational goals  Specifically, while Office of Human Resources officials stated that each Secret Service directorate is responsible for ensuring that employee goals map to high-level organizational goals, OCIO officials stated that they did not complete this mapping  The officials were unable to explain why they did not align the goals in their employees’ performance plans to the component’s high-level goals  According to the officials, the Secret Service is in the process of implementing a new automated tool that will require each office to explicitly align individual performance expectations to organizational goals  The officials stated that OCIO plans to use this tool to create employees’ fiscal year 2019 performance plans  By explicitly demonstrating how individual performance expectations align with organizational goals, the Secret Service’s IT staff should have a better understanding of how their daily activities contribute towards achieving the Secret Service’s goals  Periodically provide individuals with regular performance feedback—fully implemented  Secret Service leadership periodically provided their IT staff with performance feedback  Specifically, on an annual basis, OCIO staff received feedback during a mid-year and end-of-year performance feedback assessment  In our prior work, we have stressed that candid and constructive feedback can help individuals maximize their contribution and potential for understanding and realizing the goals and objectives of an organization  Further, this feedback is one of the strongest drivers of employee engagement  According to leading practices of the Software Engineering Institute, effective program oversight includes monitoring program performance and conducting reviews at predetermined checkpoints or milestones  This is done by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels  In addition, the Software Engineering Institute previously identified leading practices for effectively monitoring the performance of agile projects  According to the Institute, agile development methods focus on delivering usable, working software frequently; as such, it is important to measure the value delivered during each iteration of these projects  To that end, the Institute reported that agile projects should be measured on velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  DHS and the Secret Service had fully implemented the selected leading practice for monitoring the performance of one program and three projects within the IITT investment, and conducting reviews of this program and these projects at predetermined checkpoints  In addition, with regard to the selected leading practice for monitoring agile projects, the Secret Service had fully implemented this practice for one of its two projects being implemented using agile and had partially implemented this practice for the other project  Table 13 provides a summary of DHS’s and the Secret Service’s implementation of these leading practices, as relevant for one program and three projects within IITT  Monitor program performance and conduct reviews at predetermined checkpoints or milestones  Consistent with leading practices, DHS and the Secret Service monitored the performance of IITT’s program and projects by comparing actual cost, schedule, and performance information against planned targets and conducting reviews at predetermined checkpoints  For example, within the Secret Service: The Enabling Capabilities program and Multi-Level Security project monitored their contractors’ costs spent to-date on a monthly basis and compared them to the total contract amounts  OCIO used integrated master schedules to monitor the schedule performance of the Enabling Capabilities program and Multi-Level Security project  OCIO also monitored the cost, schedule, and performance of the Uniformed Division Resource Management System and Events Management projects during monthly status reviews  In addition, DHS and the Secret Service conducted acquisition decision event reviews and systems engineering life cycle technical reviews of IITT’s program and projects at predetermined checkpoints and, when applicable, identified deviations from established cost, schedule, and performance targets  For example: Secret Service OCIO met with DHS’s Office of Program Accountability and Risk Management in February 2017, and with DHS’s Acting Under Secretary for Management in June 2017, to discuss a schedule breach for the Enabling Capabilities program  In particular, the Enabling Capabilities program informed DHS that the program needed to change the planned date for acquisition decision event 3 (the point at which a decision is made to fully deploy the system) in order to conduct tests in an operational environment prior to that decision event  This delay was due to the Secret Service misunderstanding the tests that it was required to conduct prior to that decision event  Specifically, the Enabling Capabilities program had conducted tests on “production representative” systems, but these tests were not sufficient to meet the requirements for acquisition decision event 3  The project team for Multi-Level Security identified that certain technical issues they had experienced would delay system deployment and full operational capability (the point at which an investment becomes fully operational)  As such, in October 2017, the project notified the Secret Service Component Acquisition Executive of these expected delays  In particular, the web browser that was intended to provide users on “Sensitive But Unclassified” workstations the ability to view information from different security levels, experienced technical delays in meeting personal identity verification requirements  The project team also described for the executive how the schedule delay would affect the project’s performance metrics and funding, and subsequently updated the project plan accordingly  Measure and monitor agile projects on, among other things, velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of features and user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  Secret Service OCIO measured its two agile projects—Uniformed Division Resource Management System and Events Management— using certain agile metrics  In particular, OCIO officials measured the Uniformed Division Resource Management System and Events Management projects using key metrics related to velocity and development progression  For example, the officials measured development progression for both projects on a daily basis  In addition, OCIO officials monitored each project’s progress against these metrics during bi-weekly reviews that they conducted with each project team  The OCIO officials also tracked product quality metrics for the Uniformed Division Resource Management System  For example, on a monthly basis, the officials tracked the number of helpdesk tickets that had been resolved related to the system  In addition, on a quarterly basis, they tracked the number of Uniformed Division Resource Management System defects that (1) had been fixed and (2) were in the backlog  However, while OCIO officials received certain post-deployment user satisfaction information from end-users of the Uniformed Division Resource Management System by, among other things, tracking the number of helpdesk tickets related to the system and via daily verbal, undocumented feedback from certain Uniformed Division officers, OCIO officials had not fully measured and documented post- deployment user satisfaction with the system, such as via a survey of employees who use the system  The officials stated that they had not conducted and documented a survey because they were focused on (1) addressing software performance issues that occurred after they deployed the system to a limited number of users, and (2) continuing system deployment to the remaining users after they addressed the performance issues  OCIO officials stated that they plan to conduct such a documented survey by the end of September 2018  The results of the user satisfaction survey should provide OCIO with important information on whether the Uniformed Division Resource Management System is meeting users’ needs  The Secret Service’s full implementation of 11 of 14 component-level CIO responsibilities constitutes a significant effort to establish CIO oversight for the component’s IT portfolio  Additional efforts to fully implement the remaining 3 responsibilities, including ensuring that all IT contracts are reviewed, as appropriate; ensuring that the Secret Service’s enterprise governance policy appropriately specifies the CIO’s role in developing and reviewing the component’s IT budget formulation and execution; and ensuring agile projects measure product quality and post-deployment user satisfaction, will further position the CIO to effectively manage the Secret Service’s IT portfolio  When effectively implemented, IT workforce planning and management activities can facilitate the successful accomplishment of an agency’s mission  However, the Secret Service had not fully implemented all of the 15 selected practices for its IT workforce for any of the five areas— strategic planning, recruitment and hiring, training and development, employee morale, and performance management  The Secret Service’s lack of (1) a strategic workforce planning process, including the identification of all required knowledge and skills, assessment of competency gaps, and targeted strategies to address specific gaps in competencies and staffing; (2) targeted recruiting activities, including metrics to monitor the effectiveness of the recruitment program and adjustment of the recruitment program and hiring efforts based on metrics; (3) a training program, including the identification of required training for IT staff, ensuring that staff take required training, and assessment of performance data regarding the training program; and (4) a performance management system that includes all relevant technical competencies, greatly limits its ability to ensure the timely and effective acquisition and maintenance of the Secret Service’s IT infrastructure and services  On the other hand, by monitoring program performance and conducting reviews at predetermined checkpoints for one program and three projects associated with the IITT investment, in accordance with leading practices, the Secret Service and DHS provided important oversight needed to guide that program and those projects  Measuring projects on leading agile metrics also provided the Secret Service CIO with important information on project performance  We are making the following 13 recommendations to the Director of the Secret Service: The Director should ensure that the CIO establishes and documents an IT acquisition review process that ensures the CIO or the CIO’s delegate reviews all contracts containing IT, as appropriate  (Recommendation 1) The Director should update the enterprise governance policy to specify (1) the CIO’s current role and responsibilities on the Executive Resources Board, to include developing and reviewing the IT budget formulation and execution; and (2) the Deputy CIO’s role and responsibilities on the Enterprise Governance Council  (Recommendation 2) The Director should ensure that the Secret Service develops a charter for its Executive Resources Board that specifies the roles and responsibilities of all board members, including the CIO  (Recommendation 3) The Director should ensure that the CIO includes product quality and post-deployment user satisfaction metrics in the modular outcomes and target measures that the CIO sets for monitoring agile projects  (Recommendation 4) The Director should ensure that the CIO identifies all of the required knowledge and skills for the IT workforce  (Recommendation 5) The Director should ensure that the CIO regularly analyzes the IT workforce to identify its competency needs and any gaps it may have  (Recommendation 6) The Director should ensure that, after OCIO completes an analysis of the IT workforce to identify any competency and staffing gaps it may have, the Secret Service updates its recruiting and hiring strategies and plans to address those gaps, as necessary  (Recommendation 7) The Director should ensure that the Office of Human Resources (1) develops and tracks metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce, including their effectiveness at addressing skill and staffing gaps; and (2) reports to component leadership on those metrics  (Recommendation 8) The Director should ensure that the Office of Human Resources and OCIO adjust their recruitment and hiring plans and activities, as necessary, after establishing and tracking metrics for assessing the effectiveness of these activities for the IT workforce  (Recommendation 9) The Director should ensure that the CIO (1) defines the required training for each IT workforce group, (2) determines the activities that OCIO will include in its IT workforce training and development program based on its available training budget, and (3) implements those activities  (Recommendation 10) The Director should ensure that the CIO ensures that the IT workforce completes training specific to their positions (after defining the training required for each workforce group)  (Recommendation 11) The Director should ensure that the CIO collects and assesses performance data (including qualitative or quantitative measures, as appropriate) to determine how the IT training program contributes to improved performance and results (once the training program is implemented)  (Recommendation 12) The Director should ensure that the CIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed  (Recommendation 13) DHS provided written comments on a draft of this report, which are reprinted in appendix III  In its comments, the department concurred with all 13 of our recommendations and provided estimated completion dates for implementing each of them  For example, with regard to recommendation 2, the department stated that the Secret Service would update its enterprise governance policy and related policies to outline the roles and responsibilities of the CIO and Deputy CIO, among others, by March 31, 2019  In addition, for recommendation 13, the department stated that the Secret Service OCIO will include relevant technical competencies in performance plans, as appropriate, in the next performance cycle that starts in July 2019  If implemented effectively, these actions should address the weaknesses we identified  The department also identified a number of other actions that it said had been taken to address our recommendations  For example, in response to recommendation 8, which calls for the Office of Human Resources to (1) develop and track metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce and (2) report to component leadership on those metrics, DHS stated that the Secret Service’s Office of Human Resources’ Outreach Branch provides to the department metrics on recruitment efforts toward designated priority mission-critical occupations  However, for fiscal year 2017, only 1 of the 12 occupational series associated with the Secret Service’s IT workforce was designated as a mission-critical occupation for the component (i e , the 2210 IT Specialist series)  The 11 other occupational series were not designated as mission- critical occupations  In addition, for fiscal year 2018, none of these 12 occupational series were designated as mission-critical occupations  As such, metrics on recruiting for these IT series may not have been reported to DHS leadership  Moreover, while we requested documentation of the recruiting metrics for the Secret Service’s IT workforce and, during the course of our review, had multiple subsequent discussions with the Secret Service regarding such metrics, the component did not provide documentation that demonstrated it had established recruiting metrics for its IT workforce  Tracking such metrics and reporting the results to Secret Service leadership, as we recommended, would provide management with important information necessary to make effective recruitment decisions  Further, in response to recommendation 10, which among other things, calls for the CIO to define the required training for each IT workforce group, the department stated that the Secret Service OCIO recently developed training requirements for each workforce group, which were issued during our audit  However, while during our audit OCIO provided a list of recommended training courses, the office did not identify them as being required courses  Defining training that is required for each IT workforce group, as we recommended, would inform OCIO of the necessary training for each position and enable the office to prioritize this training, to ensure that its staff have the needed knowledge and skills  In addition to the aforementioned comments, we received technical comments from DHS and Secret Service officials, which we incorporated, as appropriate  We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Director of the Secret Service, and other interested parties  In addition, this report is available at no charge on the GAO website at http://www gao gov  Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or HarrisCC@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix IV  Our objectives were to evaluate the extent to which: (1) the U S  Secret Service (Secret Service) Chief Information Officer (CIO) has implemented selected information technology (IT) oversight responsibilities, (2) the Secret Service has implemented leading workforce planning and management practices for its IT workforce, and (3) the Secret Service and the Department of Homeland Security (DHS) have implemented selected performance and progress monitoring practices for the Information Integration and Technology Transformation (IITT) investment  To address the first objective, we analyzed DHS’s policies and guidance on IT management to identify the responsibilities that were to be implemented by the component-level CIO related to overseeing the Secret Service’s IT portfolio, including existing systems, acquisitions, and investments  From the list of 33 responsibilities that we identified, we then excluded the responsibility that was associated with information security, which is expected to be addressed as part of a separate, subsequent GAO review  We also excluded those responsibilities that were significantly large in scope (e g , implement an enterprise architecture) or that, in our professional judgment, lacked specificity (e g , provide timely delivery of mission IT services)  As a result, we excluded from consideration for this review a total of 10 CIO responsibilities  For the 23 that remained, we then combined certain responsibilities that overlapped with other related responsibilities  For example, we combined related responsibilities on the component CIO’s review of IT contracts  As a result, we identified 14 responsibilities for review  We validated with the acting DHS CIO that these responsibilities were key responsibilities for the department’s component-level CIOs  We then included all 14 of the responsibilities in our review  The 14 selected component-level CIO responsibilities were: 1  Develop and review the component IT budget formulation and execution  2  Manage the component IT investment portfolio, including establishing an IT acquisition review process that enables component and DHS review of component acquisitions (i e , contracts) that contain IT  3  Develop, implement, and maintain a detailed IT strategic plan  4  Ensure all component IT policies are in compliance and alignment with DHS IT directives and instructions  5  Concur with each program’s and/or project’s systems engineering life cycle tailoring plan  6  Support the Component Acquisition Executive to ensure processes are established that enable systems engineering life cycle technical reviews and that they are adhered to by programs and/or projects  7  Ensure that all systems engineering life cycle technical review exit criteria are satisfied for each of the component’s IT programs and/or projects  8  Ensure the necessary systems engineering life cycle activities have been satisfactorily completed as planned for each of the component’s IT programs and/or projects  9  Concur with the systems engineering life cycle technical review completion letter for each of the component’s IT programs and/or projects  10  Maintain oversight of their component’s agile development approach for IT by appointing the responsible personnel, identifying investments for adoption, and reviewing artifacts  11  With Component Acquisition Executives, evaluate and approve the application of agile development for IT programs consistent with the component’s agile development approach  12  Set modular outcomes and target measures to monitor the progress in achieving agile implementation for IT programs and/or projects within their component  13  Participate on DHS’s CIO Council, Enterprise Architecture Board, or other councils/boards as appropriate, and appoint employees to serve when necessary  14  Meet the IT competency requirements established by the DHS CIO, as required in the component CIO’s performance plan  To determine the extent to which the Secret Service CIO has implemented these responsibilities, we obtained and assessed relevant component documentation and compared it to the responsibilities  Specifically, we obtained and analyzed documentation including evidence of the CIO’s participation on the Secret Service governance board that has final decision authority and responsibility for enterprise governance, including the IT budget; monthly program management reports showing the CIO’s oversight of IT programs, projects, and systems; monthly status reports on program spending; the Secret Service’s IT strategic plan; the Secret Service’s enterprise governance policy; meeting minutes from the DHS board and council on which the CIO participated (i e , the CIO Council and Enterprise Architecture Board); and documentation demonstrating whether the CIO met the IT competency requirements  In addition, we obtained and analyzed relevant documentation related to the CIO’s oversight of the major IT investments on which the Secret Service was spending development, modernization, and enhancement funds during fiscal year 2017  As of July 2017, the component had one investment—IITT—that met this criterion  IITT is a portfolio investment that, as of July 2017, included two programs (one of which included three projects) and one standalone project (i e , it was not part of another program) that had capabilities that were in planning or development and modernization: the Enabling Capabilities program, Enterprise Resource Management System program (which included three projects, called Uniformed Division Resource Management System, Events Management, and Enterprise-wide Scheduling), and Multi-Level Security project  In particular, we obtained and analyzed documentation related to the CIO’s oversight of the systems engineering life cycles for IITT’s Enabling Capabilities program and the Uniformed Division Resource Management System, Events Management, and Multi-Level Security projects  This documentation included acquisition program baselines, systems engineering life cycle tailoring plans, and systems engineering life cycle technical review briefings and completion letters  We then compared the documentation against the five selected systems engineering life cycle oversight responsibilities (responsibilities 5, 6, 7, 8, and 9)  We also obtained and analyzed documentation related to the CIO’s oversight of two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  Specifically, we obtained and assessed documentation of (1) the CIO’s approval for these projects to be implemented using an agile methodology and (2) the agile development metrics that the CIO established for each of these projects  We then compared this documentation to the three agile development-related component-level CIO responsibilities (responsibilities 10, 11, and 12)  Further, to determine the extent to which the Secret Service CIO had established an IT acquisition (i e , contract) review process that enabled component and DHS review of component contracts that contain IT (which is part of responsibility 2), we first asked Secret Service officials to provide us with a list of all new, unclassified IT contracts that the component awarded between October 1, 2016, and June 30, 2017  The Secret Service officials provided a list of 54 contracts  We validated that these were contracts for IT or IT services by: (1) searching for them in the Federal Procurement Data System – Next Generation; (2) identifying their associated product or service codes, as reported in that system; and (3) determining whether those codes were included in the universe of 79 IT product or service codes identified by the Category Management Leadership Council  In validating the list of 54 contracts provided by the Secret Service, we determined that 5 of the contracts were not associated with an IT product or service code  As such, we removed those contracts from the list  In addition, we found that three other items identified by the component were not in the Federal Procurement Data System – Next Generation  Secret Service officials subsequently confirmed that these three items were not contracts  We therefore removed these three items from the list  As such, the final list of validated contracts identified by the Secret Service included 46 IT contracts  In addition, to identify any IT contracts that were not included in the list provided by the Secret Service, we conducted a search of the Federal Procurement Data System – Next Generation to identify all unclassified contracts that (1) the component awarded between October 1, 2016, and June 30, 2017; (2) were not a modification of a contract; and (3) were associated with 1 of the 79 IT product or service codes identified by the Category Management Leadership Council  Based on these criteria, we identified 144 Secret Service IT contracts in the Federal Procurement Data System – Next Generation (these 144 contracts included the 46 contracts previously identified by Secret Service officials)  We then asked Secret Service officials to validate the accuracy, completeness, and reliability of these data, which they did  From each of these two lists of IT contracts (i e , the list of 46 IT contracts identified by the Secret Service and the list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation), we then selected random, non-generalizable samples of contracts, as described below  First, from the list of 46 IT contracts identified by Secret Service officials, we removed 4 contracts that had total values of less than $10,000  To ensure that we selected across all contract sizes, we randomly selected 12 contracts from the remaining list of 42 contracts, using the following cost ranges: $10,000 to $50,000 (4 contracts), more than $50,000 to less than $250,000 (4 contracts), and more than $250,000 (4 contracts)  Second, from our list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation, we removed the 46 contracts identified by Secret Service officials  We also removed 12 contracts that had total values of less than $10,000  To ensure that we selected across all contract sizes, we randomly selected 21 contracts from the remaining list of 86 contracts, using the following cost ranges: $10,000 to $50,000 (7 contracts), more than $50,000 to less than $250,000 (7 contracts), and more than $250,000 (7 contracts)  In total, we selected 33 IT contracts for review  We separated the contracts into the three cost ranges identified above in order to ensure that contracts of different value levels had been selected  This enabled us to determine the extent to which the CIO appropriately reviewed contracts of all values  To determine the extent to which the CIO had established an IT contract approval process that enabled the Secret Service and DHS, as appropriate, to review IT contracts, we first asked Secret Service Office of the CIO (OCIO) officials for documentation of their IT contract approval process  These officials were unable to provide such documentation  Instead, the officials stated that the Secret Service CIO or the CIO’s delegate approves all IT contracts prior to award  The officials also provided documentation that identified four staff to whom the CIO had delegated his approval authority  Further, the officials stated that, in accordance with DHS’s October 2016 IT acquisition review guidance, they submitted to DHS OCIO for approval any IT contracts that met DHS’s thresholds for review, including those that (1) had total estimated procurement values of $2 5 million or more, and (2) were associated with a major investment  Based on the IT acquisition review process that Secret Service OCIO officials described, we then obtained and analyzed each of the 33 selected IT contracts and associated approval documentation to determine whether or not the Secret Service CIO or the CIO’s delegate had approved each of the contracts  In particular, we (1) reviewed the name of the contract approver on the approval documentation, and (2) compared the signature dates that were on the contracts to the signature dates that were identified on the associated approval documentation  In addition, to determine whether or not the Secret Service CIO submitted to DHS OCIO for approval the IT contracts that (1) had total estimated procurement values of $2 5 million or more, and (2) were associated with major investments, we first analyzed the 144 Secret Service IT contracts that we had previously pulled from the Federal Procurement Data System – Next Generation to determine which contracts met the $2 5 million threshold  We identified 4 contracts that met this threshold  We then requested that OCIO identify the levels (i e , major or non-major) of the investments associated with these contracts  According to OCIO officials, 3 of the 4 contracts were associated with non-major investments and 1 was not associated with an investment  As such, based on DHS’s October 2016 IT acquisition review guidance, none of these contracts needed to be submitted to DHS OCIO for review  We also interviewed Secret Service officials, including the CIO and Deputy CIO, regarding the CIO’s implementation of the 14 selected component-level responsibilities  We assessed the evidence against the selected responsibilities to determine the extent to which the CIO had implemented them  To address the second objective—determining the extent to which the Secret Service had implemented leading workforce planning and management practices for its IT workforce—we first identified seven topic areas associated with human capital management based on the following sources: The Office of Personnel Management’s Human Capital Framework  Office of Personnel Management and the Chief Human Capital Officers Council Subcommittee for Hiring and Succession Planning, End-to-End Hiring Initiative  GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others  GAO, IT Workforce: Key Practices Help Ensure Strong Integrated Program Teams; Selected Departments Need to Assess Skill Gaps  GAO, Department of Homeland Security: Taking Further Action to Better Determine Causes of Morale Problems Would Assist in Targeting Action Plans  GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government  GAO, Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success  DHS acquisition guidance  Secret Service acquisition guidance  Among these topic areas, we then selected five areas that, in our professional judgment, were of particular importance to successful workforce planning and management  They were also previously identified as part of our high-risk and key issues work on human capital management  These areas include: (1) strategic planning, (2) recruitment and hiring, (3) training and development, (4) employee morale, and (5) performance management  We also reviewed these same sources and identified numerous leading practices associated with the five topic areas  Among these leading practices, we then selected three leading practices within each of the five areas (for a total of 15 selected practices)  The selected practices were foundational practices that, in our professional judgment, were of particular importance to successful workforce planning and management  Table 14 identifies the five selected workforce areas and 15 selected associated practices  To determine the extent to which the Secret Service had implemented the selected leading workforce planning and management practices for its IT workforce, we obtained and assessed documentation and compared it against the 15 selected practices  In particular, we analyzed the Secret Service’s human capital strategic plan, human capital staffing plan, IT strategic plan, documentation of the component’s staffing model that it used to determine the number of IT staff needed, an independent verification and validation report on the component’s staffing models, documentation of the current number of IT staff, the Secret Service’s recruitment and outreach plans, documentation of DHS’s hiring authorities (which are applicable to the Secret Service), the Secret Service’s training strategic plan, IT workforce training plan, action plans for improving employee morale, and templates used for measuring and reporting employee performance  We also interviewed Secret Service officials—including the CIO, Deputy CIO, and workforce planning staff—about the component’s workforce- related policies and documentation  Further, we discussed with the officials the Secret Service’s efforts to implement the selected workforce practices for its IT workforce  Regarding our assessments of the Secret Service’s implementation of the 15 selected workforce planning and management practices, we assessed a practice as being fully implemented if component officials provided supporting documentation that demonstrated all aspects of the practice  We assessed a practice as not implemented if the officials did not provide any supporting documentation for that practice, or if the documentation provided did not demonstrate any aspect of the practice  We assessed a practice as being partly implemented if the officials provided supporting documentation that demonstrated some, but not all, aspects of the selected practice  In addition, related to our assessments of the Secret Service’s implementation of the five selected overall workforce areas, we assessed each area as follows, based on the implementation of the three selected practices within each area: Fully implemented: The Secret Service provided evidence that it had fully implemented all three of the selected practices within the workforce area; Substantially implemented: The Secret Service provided evidence that it had either fully implemented two selected practices and partly implemented the remaining one selected practice within the workforce area, or fully implemented one selected practice and partly implemented the remaining two selected practices within the workforce area; Partially implemented: The Secret Service provided evidence that it had partly implemented each of the three selected practices within the workforce area; Minimally implemented: The Secret Service provided evidence that it partly implemented two selected practices and not implemented the remaining one selected practice within the workforce area, or partly implemented one selected practice and not implemented the remaining two selected practices within the workforce area; or Not implemented: The Secret Service did not provide evidence that it had implemented any of the three selected practices within the workforce area  To address the third objective—determining the extent to which the Secret Service and DHS have implemented selected performance and progress monitoring practices for IITT—we reviewed leading project monitoring practices and guidance from the Software Engineering Institute  First, we reviewed the practices within the Project Monitoring and Control process area of the Institute’s Capability Maturity Model Integration® for Acquisition  Based on our review, we identified four practices associated with monitoring program performance and progress  In our professional judgment, all four of these practices were of significance to managing the IITT investment given the phase of the life cycle that the investment was in  As such, we elected to include all four of these practices in our review, and combined them into one practice, as follows: Monitor program performance and conduct reviews at predetermined checkpoints or milestones by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels  Next, given the agile development methodology that the Secret Service was using for certain projects within IITT, we reviewed the Software Engineering Institute’s technical note on the progress monitoring of agile contractors  Based on our review, and in consultation with an internal expert, we selected four agile metrics that the Institute identified as important for successful agile implementations and that, in our professional judgment, were of most significance to monitoring the performance of IITT’s agile projects  We then combined these four metrics into one practice, as follows: Measure and monitor agile projects on velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of features and user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  To determine the extent to which DHS and the Secret Service had implemented the first selected practice, we analyzed relevant program management and governance documentation for IITT’s Enabling Capabilities program, and Multi-Level Security, Uniformed Division Resource Management System, and Events Management projects  In particular, we analyzed acquisition program baselines, DHS acquisition decision event memorandums, artifacts from DHS and Secret Service program oversight reviews, cost monitoring reports, program integrated master schedules, and program status briefings, and compared this documentation to the selected practice  We also interviewed Secret Service OCIO officials regarding the Secret Service’s and DHS’s efforts to monitor the IITT investment’s performance and progress  To determine the extent to which the Secret Service had implemented the second selected practice related to measuring and monitoring agile projects on agile metrics (i e , velocity, development progression, product quality, and post-deployment user satisfaction), we obtained and analyzed agile-related documentation for the two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  Specifically, to determine the extent to which the Secret Service was measuring and monitoring these two projects on metrics for velocity and development progression, we obtained and analyzed documentation, such as sprint burndown charts and monthly program status reports, and compared it to the selected practice  In addition, the agile metrics for product quality and post-deployment user satisfaction were only applicable to projects that had been deployed to users  As such, these metrics were applicable to the Uniformed Division Resource Management System (which the Secret Service had deployed to users) and were not applicable to Events Management (which the Secret Service had not yet deployed to users, as of early May 2018)  We therefore obtained and analyzed documentation demonstrating that Secret Service OCIO measured product defects for the Uniformed Division Resource Management System  We also requested documentation demonstrating that OCIO had measured and monitored post-deployment user satisfaction for this project, including via a survey  OCIO officials stated that they had not conducted such a survey and were unable to provide documentation demonstrating they had measured post- deployment user satisfaction for the Uniformed Division Resource Management System  To assess the reliability of the cost, schedule, and agile-related data that were in DHS and the Secret Service’s program management and governance documentation for the IITT investment, we (1) analyzed related documentation and assessed the data against existing agency records to identify consistency in the information, and (2) examined the data for obvious outliers, incomplete, or unusual entries  We determined that the data in these documents were sufficiently reliable for our purpose, which was to evaluate the extent to which DHS and the Secret Service had implemented processes for monitoring the IITT investment’s performance and progress  We conducted this performance audit from May 2017 to November 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  As of June 2018, the Secret Service’s Information Integration and Technology Transformation (IITT) investment included two programs (one of which included three projects) and one project that had capabilities that were in planning or development and modernization, as described below: Enabling Capabilities  This program is intended to, among other things, (1) modernize and enhance the Secret Service’s information technology (IT) network infrastructure, including increasing bandwidth and improving the speed and reliability of the Secret Service’s IT system performance; (2) enhance cybersecurity to protect against potential intrusions and viruses; and (3) provide counterintelligence and data mining capabilities to improve officials’ ability to perform the Secret Service’s investigative mission  Enterprise Resource Management System  This program comprises three projects that are intended to provide: a system that will enable the Secret Service’s Uniformed Division to efficiently and effectively plan, provision, and schedule missions (this project is referred to as Uniformed Division Resource Management System), a system that will unify the logistical actions (e g , assigning personnel) surrounding special events that Secret Service agents need to protect, such as the United Nations General Assembly (this project is referred to as Events Management), and a capability for creating schedules for Secret Service agents and administrative, professional, and technical staff, as well as the ability to generate reports on information such as monthly hours worked (this project is referred to as Enterprise-wide Scheduling)  Multi-Level Security  This project is intended to enable authorized Secret Service users to view two levels of classified information on a single workstation  Previously, data at various security levels were contained and used in multiple disparate systems  Multi-Level Security is intended to streamline users’ access to information at different security levels in order to enable them to more quickly and effectively perform their duties  Table 15 provides the planned life cycle cost and schedule estimates (threshold values) for each IITT program and project that had capabilities in planning or development and modernization, as of June 2018  In addition, the table describes any changes in those cost and schedule estimates, as well as the key reasons for any changes, as identified by officials from the Secret Service’s Office of the Chief Information Officer  In addition to the contact named above, the following staff made key contributions to this report: Shannin O’Neill (Assistant Director), Emily Kuhn (Analyst-in-Charge), Quintin Dorsey, Rebecca Eyler, Javier Irizarry, and Paige Teigen 

ground_truth: NSF has identified potential benefits and challenges associated with its use of rotators. Benefits include fresh perspectives and close connections to the scientific community, while challenges include staffing turnover and higher costs for some rotators compared with permanent employees. GAO was asked to review NSF's use and management of the IPA and VSEE rotator programs, among other things. This report examines (1) the number, costs, and uses of NSF rotators for fiscal year 2008 through fiscal year 2017; (2) the strategies NSF has used to manage rotator costs and the results of these efforts; and (3) the extent to which NSF has a workforce strategy for using rotators and has evaluated the results of its rotator programs. GAO analyzed summary-level data on NSF's rotators; reviewed key documents; interviewed NSF officials; conducted semistructured interviews with a nongeneralizable sample of rotators and permanent federal employees selected from different scientific directorates within NSF; and compared NSF's management of the program to key principles for effective strategic workforce planning. The numbers of rotators—outside scientists, engineers, and educators on temporary assignment—at the National Science Foundation (NSF) and their costs in proportion to other staff remained relatively stable in fiscal years 2008 through 2017. Most rotators joined NSF under its Intergovernmental Personnel Act (IPA) mobility program. IPA rotators comprised about 12 percent of NSF's workforce and 17 percent of staff costs on average and were not subject to a federal salary cap. They remain employees of their home institutions, with NSF reimbursing the institutions for most of their salaries and benefits. The remaining rotators are considered temporary federal employees under the Visiting Scientist, Engineer, and Educator (VSEE) program; their salaries could not exceed the federal maximum for their positions. Beginning in fiscal year 2017, NSF adopted IPA rotator program cost management strategies expected to achieve the greatest savings with the least harm to recruitment, but NSF officials said it is too soon to determine the full results. For example, for new IPA rotators who had not yet begun negotiating their assignments, NSF began requiring their home institutions to pay for 10 percent of the rotators' salary and benefits. NSF officials told GAO they expect to issue a report evaluating the strategies in December 2018. NSF's IPA program steering committee recommended developing a workforce strategy for balancing the agency's use of rotators with federal staff, but as of June 2018, NSF had not developed a strategy or fully evaluated the IPA and VSEE rotator programs' results, as called for by GAO's key principles for effective strategic workforce planning. NSF officials said they recognized the value of a workforce strategy but were focusing instead on other workforce planning efforts, and they had not fully evaluated program results in part because rotators are blended into the agency's permanent workforce, making a separate evaluation difficult. Without a workforce strategy and evaluation of results, NSF is limited in its ability to manage and, if warranted, adjust its use of rotators. GAO recommends that NSF develop an agency-wide strategy for balancing the agency's use of rotators with permanent staff and evaluate the contributions of its rotator programs toward NSF's human capital goals and programmatic results. NSF agreed with GAO's recommendations.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: NSF relies on two programs for bringing rotators into the agency: (1) the IPA program and (2) the VSEE program  The Office of Personnel Management develops policies on agencies’ use of the IPA program and promulgates program regulations  Rotators in NSF’s IPA and VSEE programs differ in key respects, including their employment status and compensation  IPA rotators  NSF enters into written agreements with rotators’ home institutions for all IPA assignments  The agreements detail rotators’ salaries and health, retirement, and other fringe benefits at their home institutions, as well as the cost-sharing amounts NSF and home institutions are to pay during rotators’ assignments  NSF reimburses its cost-sharing amounts to home institutions, which continue to pay rotators’ full salaries and benefits  NSF does not cap the salaries of IPA rotators; as a result, IPA rotators may receive salaries that exceed the maximum federal salary for the position they hold at NSF  In contrast, if an IPA rotator’s salary is less than the minimum federal salary for the position, NSF will supplement the salary to the minimum rate  VSEE rotators  NSF appoints VSEE rotators as federal employees on a nonpaid leave of absence from their home institutions  VSEE rotators receive their salaries directly from NSF but are not eligible for certain federal benefits, such as retirement; instead, NSF reimburses home institutions for the employer’s share of retirement, life insurance, and health benefits that would otherwise be discontinued  NSF’s policy is to set salaries for VSEE rotators that are generally comparable to the salaries the rotators would receive at their home institutions  In setting salaries, NSF also takes into account other sources of income, such as consulting, and allows for locality pay adjustments applicable to employees in the Washington, D C , metropolitan area  However, because VSEE rotators are federal employees, NSF caps their salaries at the federal maximum for the position they hold at NSF  Both IPA and VSEE rotators are eligible for certain other types of reimbursement  In particular, rotators have the option of having NSF pay their moving expenses to and from Washington, D C , or receiving per diem allowances in accordance with federal travel regulations for up to 2 years  In addition, NSF may reimburse rotators for travel-related expenses related to their participation in NSF’s Independent Research and Development program, which enables NSF staff to maintain their involvement with their professional research and research-related activities at their home institutions  Table 1 shows additional information on IPA and VSEE rotator expenses  Rotators are generally assigned to one of NSF’s seven directorates that support science and engineering research and education (see table 2)  Each directorate is headed by an assistant director and deputy assistant director  Directorates are further subdivided into divisions, offices, or sections  Each division is headed by a division director and typically a deputy division director, and each office is headed by an office director and typically a deputy office director  All these positions are executive positions at NSF  At the staff level, NSF uses program directors—subject matter experts in the scientific areas they manage—to conduct reviews of proposals and recommend which projects the agency should fund  With an annual budget of about $7 5 billion, NSF funds approximately 24 percent of all federally supported basic research conducted by colleges and universities in the United States  In 2016, NSF established the Steering Committee for Policy and Oversight of the IPA Program  The steering committee serves as the primary body for considering policy on NSF’s use of IPA rotators and overseeing common approaches to budgeting and implementation of the IPA program  The committee’s membership includes NSF’s chief human capital officer, who serves as the chair, and several other NSF officials  The steering committee has established strategic principles for management of the IPA program  These principles include maintaining a balance between IPA rotators and federal staff and a commitment to ongoing improvement of the program  NSF officials told us that there is no similar steering committee for overseeing VSEE rotators  Instead, each VSEE rotator is individually overseen by his or her respective supervisor  For the agency as a whole, NSF’s Office of Information and Resource Management and its Division of Human Resource Management conduct human capital management  NSF officials stated that the head of the Office of Information and Resource Management serves as the Chief Human Capital Officer and develops and oversees NSF’s human capital approaches and strategies  These officials also told us that the Deputy Chief Human Capital Officer serves as the division director of Human Resource Management and is responsible for administering the division’s day-to-day operations  The Division of Human Resource Management administers the agency’s human capital policies as set forth in NSF’s personnel manual  The numbers of rotators and their costs to NSF in proportion to other staff have remained relatively stable  Most rotators were IPA rotators, and were used in both executive and program director (staff-level) positions  NSF generally used VSEE rotators in program director positions  Most rotators at NSF were IPA rotators, and the proportion of rotators relative to other staff has remained relatively stable over time (see fig  1)  During the 10-year period we reviewed, from fiscal year 2008 through fiscal year 2017, IPA and VSEE rotators comprised about 12 percent and about 3 percent, respectively, of NSF’s total workforce; and the number of IPA rotators ranged from 162 to 190 (about 11 to 12 percent of total staff), and the number of VSEE rotators ranged from 22 to 52 (about 1 to 3 percent of total staff)  NSF primarily used rotators across its seven scientific directorates, using IPA rotators in executive and program director positions and VSEE rotators in program director positions  The agency used rotators in these positions alongside NSF’s permanent staff to perform day-to-day agency operations, including managing the agency’s merit review process for determining which projects to fund  NSF used IPA rotators in executive positions such as assistant director  According to agency officials, individuals in executive positions at NSF are responsible for setting the direction for the scientific area they are assigned, leading scientific and technical matters, establishing an organizational culture, overseeing outreach and collaboration with NSF stakeholders, and contributing to NSF and national policy development and implementation  For example, an executive IPA rotator that we interviewed told us that he emphasized forming partnerships with industry when setting the direction for his directorate, including issuing joint solicitations for research proposals with industry partners  In addition, according to NSF officials, individuals in executive positions provide guidance and team management for staff  The proportion of IPA rotators to federal employees in executive positions within NSF’s seven scientific directorates and other staff offices has generally increased since fiscal year 2012  As shown in figure 4, from fiscal year 2008 through fiscal year 2017, the number and proportion of executive positions filled by IPA rotators ranged from 18 of 98 (about 18 percent) in 2008 to 30 of 108 (about 28 percent) in fiscal year 2016  In November 2017, IPA rotators filled 29 of 88 (about 33 percent) executive positions within NSF’s seven scientific directorates  At that time, the proportion of executive positions filled by IPA rotators varied among directorates, as shown in table 3  For example, IPA rotators filled 4 of 8 (50 percent) of the executive positions in the Directorate for Social, Behavioral, and Economic Sciences and 2 of 14 (about 14 percent) of the executive positions in the Directorate for Mathematical and Physical Sciences  According to NSF officials, NSF often pairs IPA rotators and federal employees at the executive level so that each can benefit from the other’s experience and perspective  For example, in all but one directorate, an IPA rotator filled the assistant director position and a federal employee filled the corresponding deputy assistant director position  Two NSF executives we interviewed, including an IPA rotator and a federal employee, commented positively on the pairing of IPA rotators and federal employees at the executive level  For example, they said that rotators maintain close ties to the research community and federal employees may have more experience with NSF’s institutional history  One NSF executive told us that IPA rotators help keep the agency at the forefront of science because they have deep ties with the research community and regularly publish their own research  Additionally, a federal program director we interviewed told us that in one previous instance in which an IPA rotator filled an executive position without being paired with a federal employee, the rotator’s lack of institutional knowledge of NSF and the steep learning curve for the position caused inefficiencies during the rotator’s first year at NSF  The agency, however, does not require pairing IPA rotators and federal employees at the executive level, according to NSF officials  For example, in November 2017, IPA rotators filled both the division director and deputy division director positions in the Division of Behavioral and Cognitive Science and Division of Undergraduate Education  In our interviews with a nongeneralizable sample of NSF employees and rotators, we found mixed perceptions about the effect of NSF’s use of IPA rotators on opportunities for advancement for permanent employees  For example, in response to a question about this effect, one permanent NSF employee told us that she advanced to an executive position and that opportunities exist for advancement within the agency  In contrast, another NSF employee we interviewed told us that she did not feel there were opportunities for advancement because, in her view, executive vacancies created by the departure of rotators were exclusively filled with other rotators  NSF officials said that the agency has no policy that restricts repeatedly filling certain executive positions with rotators and that such a situation is a common practice  Nevertheless, NSF officials told us 32 of the 88 executives (about 36 percent) in NSF’s seven scientific directorates in November 2017 had held staff-level positions within the agency before becoming executives  NSF uses both IPA and VSEE rotators in program director positions, which are staff-level positions  In fiscal year 2016, NSF had a total of 506 program directors, including 139 IPA rotators (about 27 percent) and 39 VSEE rotators (about 8 percent)  According to NSF officials, program directors are responsible for conducting long-range planning and developing budgets for the areas of science represented by their program and for administrating the merit review process  In particular, IPA and VSEE rotators who serve as program directors help determine the projects that NSF funds  To do so, they review proposals, identify experts in their field to serve as external reviewers, and make funding recommendations to their respective division directors  NSF officials told us that, similar to the pairing of IPA rotators and federal employees at the executive level, permanent and rotating program directors frequently work together on a shared program so that each can benefit from the other’s experience and perspective  For example, a rotating program director we interviewed told us that she worked under the guidance of a program lead, who is typically a permanent employee  Another rotating program director told us that NSF’s permanent federal employees are good at training incoming rotators  Beginning in fiscal year 2017, NSF adopted rotator program cost- management strategies expected to achieve the greatest savings with the least harm to recruitment, but NSF officials said it is too soon to determine the full results because these new strategies are being phased in for new IPA agreements only  NSF considered other strategies to manage rotator costs, but it did not adopt them, generally because NSF anticipated negative effects on rotator recruitment or because it estimated the resulting cost savings would be small  NSF has adopted three strategies to manage rotators’ costs in fiscal year 2017, but, NSF officials said it is too soon to determine the full results because these new strategies are being phased in for new IPA agreements only  All three of these strategies relate to IPA rotators; NSF officials told us that they have not considered or adopted any cost- management strategies related to VSEE rotators  The officials explained that any such strategies could affect NSF’s entire federal workforce because VSEE rotators are federal employees  The three strategies are: (1) obtaining a minimum 10 percent cost-share from each IPA rotator’s home institution, (2) limiting IPA rotators’ paid trips to their home institutions to 12 per year, and (3) no longer reimbursing IPA rotators for consulting income that they forgo while at NSF  NSF officials told us they expect to issue a report with the results of evaluations of all three strategies in December 2018  In October 2016, NSF implemented a cost-sharing pilot program that requires institutions covered by the program—those who entered into negotiations for new IPA agreements in fiscal year 2017—to pay for at least 10 percent of the IPA rotators’ salaries and fringe benefits  Implementing this cost-management strategy, and the other strategies that NSF adopted, was consistent with recommendations from NSF’s steering committee for oversight of IPA rotators  This cost-management strategy targeted NSF’s costs for IPA rotators’ salary and fringe benefits, which constitute the largest component of IPA rotators’ costs  For example, these costs were about $34 7 million, or about 89 percent of IPA rotator costs in fiscal year 2017  Previously, according to NSF officials, the agency requested an optional cost-share amount of 15 percent from rotators’ home institutions, but it typically received less because of variations in the amounts that home institutions provided  According to an October 2016 report from the task force on fiscal oversight, NSF decided on 10 percent for the cost-sharing pilot program because, historically, few home institutions provided the full 15 percent and NSF believed a requirement of 10 percent would not significantly affect its ability to recruit and hire IPA rotators  If a home institution is unable to provide the full 10 percent, the institution may request that NSF waive the cost-sharing requirement  According to NSF officials, such requests must be signed by a senior administrator at the rotator’s home institution and include the rationale for not being able to provide the required amount, the financial impact on the institution if it were to provide the full 10 percent, and associated documentation, among other things  Changes made in implementing this strategy, and the other strategies that NSF adopted, applied to new IPA agreements made in fiscal year 2017  These changes did not apply to IPA rotators with agreements made prior to 2017—even if those agreements are subsequently extended or renewed—or that were being negotiated at the time of the policy change, provided that the rotators’ appointment memoranda were already being reviewed by NSF’s Division of Human Resource Management  NSF officials told us that as of March 2018, the agency had not conducted full evaluations of this strategy or the other strategies because it was too soon to determine their full effects and NSF had not yet collected enough data to do so  Instead, NSF issued reports in January and March 2018 containing its preliminary analyses  In general, these preliminary reports found that the cost-management strategies resulted in savings to NSF  Similarly, our analysis of data from NSF found that cost sharing as a percentage of IPA rotators’ salary and fringe benefits increased from about 7 percent in fiscal year 2016 to about 8 percent in fiscal year 2017  NSF officials told us that of the 55 IPA rotators who were subject to the cost-sharing requirement in fiscal year 2017: the home institutions for 54 rotators met or exceeded the 10 percent cost-share requirement, and of those, 16 exceeded the cost-share requirement; and the home institution for 1 rotator did not cost-share because the rotator was from a Federally Funded Research and Development Center and NSF waived the cost-share requirement because cost- sharing would not decrease the overall federal cost  In November 2017, NSF decided to extend the cost-sharing pilot through at least the end of fiscal year 2018, to ensure a full evaluation could be conducted  In particular, NSF officials told us that they need more data and experience with this pilot program to better understand its effects, such as the ability to recruit potential IPA rotators  For example, one IPA rotator that we interviewed expressed concern with the cost-sharing requirement’s potential effect on small or publicly funded universities, which may lack funds to contribute to the cost of an IPA assignment  According to NSF officials, their evaluation will include an analysis of the cost of IPA rotators under the cost-sharing requirement and its effect on the IPA program, including recruitment  Beginning in fiscal year 2017, for IPA rotators who entered into negotiations for new agreements in that fiscal year, NSF placed a limit of 12 agency-funded trips per year that rotators may take to their home institutions under the Independent Research and Development program  In our analysis of data from NSF, we found that NSF’s costs for IPA rotators under this program decreased from about $1 5 million (about 3 percent of IPA rotator costs) in fiscal year 2016 to $1 1 million (about 3 percent of IPA rotator costs) in fiscal year 2017  NSF officials told us that the new limit applies only to an IPA rotator’s trips to their home institution and does not limit travel to other locations for fieldwork or scientific conferences, among other things  These officials explained that NSF chose not to limit trips to these other locations because they are considered fundamental to IPA rotators’ research and are infrequent—occurring one to three times per year, on average, per IPA rotator  Additionally, rotators are permitted to use annual leave, leave without pay, or flexitime to take trips using non-NSF funds for activities performed on a rotator’s own time  In adopting this cost-management strategy, NSF sought to balance the benefits of IPA rotators’ travel with the travel costs  According to the Task Force on Fiscal Oversight’s October 2016 report, NSF’s support for travel benefits the agency by providing a way for program directors and executives to stay current in their scientific fields, conduct outreach with scientific communities, and provide oversight and stewardship of NSF’s programs and awards  NSF officials told us that the agency sought to control travel costs under the Independent Research and Development program by setting a reasonable limit to NSF-funded trips that would cause the least harm to rotators’ research so as not to discourage them from coming to NSF  As a result, NSF decided on a maximum of 12 trips per year under this program because, historically, more than 80 percent of the IPA rotator participants traveled to their home institution less than once per month  In fiscal year 2017, for IPA rotators who entered into new agreements in that fiscal year, NSF ended reimbursements for consulting income that the rotators forgo as a result of their assignment to NSF  Previously, when an IPA rotator discontinued consulting activities during an IPA assignment, NSF would reimburse the rotator up to $10,000 a year  IPA rotators who entered into negotiations or agreements with NSF prior to this change may still receive this reimbursement  In fiscal year 2017, NSF’s cost for lost consulting reimbursements to IPA rotators was $150,000  This amount represented a decrease of about $160,000, or about 52 percent, from fiscal year 2016  NSF made this change because it determined that doing so would not negatively affect the IPA program  In particular, NSF found that other federal science agencies typically did not reimburse IPA rotators for lost consulting income and it concluded that IPA rotators typically do not expect NSF to offer reimbursement  In addition to the three adopted strategies, NSF’s Task Force on Fiscal Oversight identified other potential cost management strategies for the IPA program  The task force reviewed various data on the costs that make up the IPA program, such as the number of IPA rotators who received a particular form of compensation or who would be affected by the potential strategies  In addition, the task force took into account anecdotal and other evidence on how IPA rotators might react to the strategies  Using input from the task force, NSF opted against the other potential strategies because it either (1) expected the resulting cost savings to be small or (2) anticipated potential negative effects from implementing them, such as increased difficulty in hiring IPA rotators  These potential cost-management strategies primarily related to IPA rotator compensation, as described below  Capping IPA rotators’ salaries  NSF decided against establishing a salary cap for IPA rotators at various levels between about $185,000 and $240,000 annually  The task force found that salary caps at lower levels would have greater cost savings because of the higher number of individuals covered by the cap, but that the caps would also pose a significant risk to NSF’s ability to recruit IPA rotators  In particular, the task force found that salary caps at lower levels would disproportionately affect IPA rotators in two of its directorates—the Directorate for Computer and Information Science and Engineering and the Directorate for Engineering—because of the higher salaries of individuals in positions associated with those fields  As a result, the task force recommended that NSF first assess the effects of its cost-sharing pilot program before proceeding with any cap on IPA rotators’ salaries  Reducing or eliminating IPA rotators’ supplemental pay  NSF decided against reducing or eliminating the supplemental pay that IPA rotators receive when their salary at their home institution is below the minimum for their NSF position  In fiscal year 2017, NSF’s cost for IPA rotators’ supplemental pay was $1 0 million (about 3 percent of IPA rotator costs)  The task force recommended against this potential cost-management strategy because it would disproportionately affect IPA rotators in two of its directorates—the Directorate for Biological Sciences and the Directorate for Geosciences  In addition, the task force expected that any cost savings associated with this strategy would be small  Reducing IPA rotators’ per diem payments  NSF decided against reducing or eliminating per diem payments for lodging (excluding taxes), meals, and incidental expenses incurred during the length of rotators’ assignments  In fiscal year 2017, NSF’s cost for per diem payments was $3 1 million (about 8 percent of IPA costs)  The task force concluded, based on its analysis of per diem costs and anecdotal evidence, that many IPA rotators would opt to depart NSF if NSF did not provide per diem payments  As a result, the task force recommended against this strategy  As of June 2018, NSF had not developed an agency-wide workforce strategy for using rotators, as its IPA program steering committee recommended  In addition, NSF has not fully evaluated or developed plans to evaluate both IPA and VSEE rotator program results in terms of progress toward NSF’s human capital goals or programmatic results  As of June 2018, NSF had not developed an agency-wide workforce strategy that includes use of rotators, as NSF’s IPA program steering committee had recommended  In an August 2016 report on the IPA program, the steering committee stated that NSF did not have an agency- wide workforce strategy; instead, each directorate made decisions on its own about when and how to use IPA rotators in executive and program director positions  According to the report, an agency-wide framework would enable NSF to ensure an optimal balance of federal and rotator executives and program directors, which is a strategic principle that the steering committee developed for the IPA program  In February 2017, the committee issued an internal report to agency leadership that recommended expanding what was originally envisioned as a workforce strategy for the IPA program into a comprehensive agency-wide workforce strategy  The report stated that expanding the scope of the workforce strategy would have the greatest impact across the agency and would help NSF leadership in making strategic human capital decisions  The report outlined a process for developing a workforce strategy with various steps, including the following: Job analyses  The report recommended job analyses to review the roles and responsibilities of executive and staff-level positions and to identify the skills and capabilities required for successful performance of the work  According to the report, the steering committee’s working group for developing a workforce strategy found, based on its initial efforts to review position descriptions and roles and responsibilities, that some functions may be better served if performed by permanent federal employees and other functions by rotators  However, the working group concluded that NSF should obtain additional input and evidence before initiating large-scale changes in its workforce  Analysis of workforce gaps and surpluses  The report stated that identifying gaps and surpluses in the demand and supply for federal and rotator scientific staff would inform opportunities to optimize recruitment and retention efforts  The report recommended separate analyses for executive and scientific staff- level positions  Development of strategies to close workforce gaps and address surpluses  According to the steering committee’s report, examples of strategies include succession planning and rebalancing the mix of permanent federal staff and rotators to ensure an optimal workforce with the skills, experience, and capabilities to accomplish NSF’s science-related work  According to NSF officials, the agency’s Division of Human Resource Management was responsible for implementing the steering committee’s recommendation  In particular, it undertook an effort to work with senior leadership to develop a broad strategic workforce plan for the agency  However, in June 2018, NSF officials told us that they shifted their focus from developing a separate workforce strategy in order to focus instead on (1) development of a human capital operating plan, which agencies are required to develop and approve annually, and update as needed, under OPM regulations that went into effect on April 11, 2017; and (2) an Office of Management and Budget (OMB) memorandum issued in April 2017 directing agency heads to develop reform plans that identify ways to improve the efficiency, effectiveness, and accountability of their respective agencies  The NSF officials explained that they recognized the value in having a workforce strategy, but they did not consider it appropriate for the Division of Human Resource Management to develop a workforce strategy at the same time that the agency was completing the OPM and OMB plans  NSF did not specify how its efforts to complete the OPM and OMB plans would address the need the steering committee identified for an agency- wide framework that would enable NSF to ensure an optimal balance of federal and rotator executives and program directors  In particular, NSF’s human capital operating plan, which it approved in April 2018, does not discuss NSF’s use of rotators or include information on balancing the agency’s use of rotators with permanent staff  Furthermore, NSF has not yet determined how it will address its use of rotators as part of its agency reform plan  In particular, NSF officials told us in June 2018 that they may address the agency’s use of rotators under the workforce focus area of its reform plan, but that they were only just beginning to identify and select initiatives under this focus area and that these initiatives have not yet been finalized  The process the NSF steering committee laid out in its internal report, when implemented, would align with two key principles GAO has identified for effective strategic workforce planning  Specifically, it would align with the principles of (1) determining the skills and competencies that are critical to successfully achieving missions and goals, and (2) developing human capital strategies to address gaps and enable the contribution of critical skills and competencies needed for mission success  By incorporating the NSF’s steering committee’s recommendation for a workforce strategy—and the process outlined by the steering committee for developing this strategy—into its human capital operating plan or agency reform plan, NSF could better manage its use of rotators and balance them with its permanent staff  We have previously found that high-performing organizations recognize the fundamental importance of measuring both the outcomes of human capital strategies and how these outcomes have helped the organizations accomplish their missions and programmatic goals  However, as of May 2018, NSF had not fully evaluated and did not have plans to evaluate the results of its IPA and VSEE rotator programs in terms of progress toward human capital goals and the contributions the programs made toward achieving programmatic results  One of GAO’s key principles for effective strategic workforce planning states that agencies should monitor and evaluate progress toward the agencies’ human capital goals and the contribution that human capital results have made toward achieving programmatic results  In particular, we previously found that evaluation activities can improve the effectiveness of workforce strategies by identifying shortfalls in performance and other improvement opportunities  OPM also requires agencies to develop a human capital operating plan that will support the evaluation of the agency’s human capital strategies  In March 2014, NSF published a summary of the results of focus groups with IPA rotators and their supervisors  This summary outlined benefits and challenges of the program from the perspectives of both groups, such as the benefit of bringing fresh perspective and new ideas to NSF and the challenge of recruiting and retaining qualified IPA rotators  However, the summary did not provide the agency’s assessment of progress towards programmatic results and human capital goals  For example, it summarized the benefits of the program from the standpoint of rotators and did not provide NSF’s assessment of how individual IPA rotators or the program as a whole contributed to NSF’s scientific mission  In addition, the summary did not provide an assessment of the extent to which the current workforce balance of federal and rotator executives and program directors is aligned with NSF’s work  In our semistructured interviews with federal staff and rotators in executive and staff-level positions at NSF, most were comfortable with the current balance, but three individuals raised concerns about the use of rotators in executive positions, suggesting that NSF could benefit from further analysis of its balance of rotators and federal staff  In April 2018, NSF adopted its human capital operating plan which identifies specific, short-term actions that the agency will take to achieve its human capital goals  In its plan, NSF identified strategies derived from NSF’s commitment to ongoing improvement, such as reviewing and realigning its workforce to meet future needs  Also, NSF’s process for developing a workforce strategy, outlined in the steering committee’s February 2017 internal report, included recommendations to conduct an assessment of the outcomes of workforce strategies and the impact of these outcomes on helping NSF accomplish its scientific mission and related programmatic goals  However, plans for this assessment did not include an evaluation of the agency’s rotator programs  Moreover, neither the steering committee’s February 2017 internal report nor NSF’s April 2018 report committed to conducting such an evaluation or specified how assessments described in its reports would address NSF’s rotator programs  For example, neither report specified how NSF would evaluate the extent to which the rotator programs have achieved NSF’s objectives, which we identified through our review of NSF documentation and interviews with NSF officials  These objectives include: bringing fresh perspectives from across the country and across all fields of science and engineering supported by NSF; helping influence new directions for research in science, engineering, and education, including emerging interdisciplinary fields; providing scientific leadership and management of NSF’s research and education programs; and providing opportunities for researchers to gain first-hand knowledge of the philosophy and mechanisms of federal support for research and bring this knowledge back to their home institutions  According to NSF officials, the agency has not separately evaluated the results of its rotator programs in part because rotators are blended into its permanent federal workforce, making it difficult to evaluate the results of its rotator programs separately from those of its overall workforce  In our December 2003 report on key principles for effective strategic workforce planning, we found that federal agencies in general have experienced difficulties in defining practical and meaningful measures that assess the effects human capital strategies have on programmatic results  However, without an evaluation of the extent of the rotator programs’ contributions toward NSF’s human capital goals or programmatic results, NSF is limited in its ability to demonstrate the programs’ benefits to external stakeholders, such as the Congress, and to adjust the programs, if warranted  Such adjustments could include increasing or decreasing the use of rotators overall or in certain types of positions, such as executive or staff-level positions  In recent years, NSF has recognized the need to think more strategically about its use of rotators and has taken positive steps to manage its rotator programs  For example, beginning in fiscal year 2017, NSF adopted several strategies to manage the cost of rotators  However, as of June 2018, NSF had decided against developing a separate agency-wide strategy for balancing its use of IPA rotators and federal staff, as NSF’s steering committee for the IPA program recommended in February 2017  NSF officials said that they recognized the value in having a workforce strategy but wanted to focus instead on addressing OPM and OMB requirements related to workforce planning  By following through on the steering committee’s recommendation for a workforce strategy, NSF could better manage its use of rotators and balance them with its permanent staff  Moreover, as of June 2018, NSF had not fully evaluated the results of the rotator programs, as called for by key principles for effective strategic workforce planning  NSF officials told us they have not done so, in part, because rotators are blended into NSF’s permanent federal workforce, making it difficult to evaluate the results of its rotator program separately from those of its overall workforce  However, without an evaluation of the extent of the rotator programs’ contributions toward NSF’s human capital goals or programmatic results, NSF is limited in its ability to demonstrate the programs’ benefits to external stakeholders, such as the Congress, and to adjust the programs, if warranted  We are making the following two recommendations to NSF: The NSF Director of Human Resource Management should complete the development of an agency-wide workforce strategy for balancing the agency’s use of IPA and VSEE rotators with permanent staff as part of NSF’s current agency reform planning efforts or updates to its human capital operating plan  (Recommendation 1) The NSF Director of Human Resource Management should evaluate the contributions of the IPA and VSEE rotator programs toward NSF’s human capital goals and the contributions the programs have made toward achieving programmatic results  (Recommendation 2) We provided a draft of this report to NSF for comment  In its written comments, which are reproduced in appendix I, NSF concurred with our recommendations and stated that implementation of the recommendations will enhance efforts to fulfill the agency’s mission and strengthen its workforce  NSF also provided technical comments, which we incorporated as appropriate  We are sending copies to the appropriate Congressional Committees, the Director of the National Science Foundation, and other interested parties  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-3841 or neumannj@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix II  In addition to the individual named above, Joseph Cook (Assistant Director), Nkenge Gibson, Kathryn Smith, and Douglas Hunker made key contributions to this report  Also contributing to this report were Antoinette Capaccio, Serena Lo, Timothy Guinane, Cynthia Norris, and Sara Sullivan 


Report 2:  As part of this exercise, the Secret Service also compared its employee work-life offerings (e g , on-site childcare and telework program) to those available at other agencies  In addition, the Secret Service developed and implemented action plans for improving employee morale  Among these action plans, for example, the component implemented a student loan repayment program and expanded its tuition assistance program’s eligibility requirements  Establish and track metrics of success for improving employee morale, and report to agency leadership on progress improving morale—fully implemented  The Secret Service tracked metrics for improving employee morale and reported the results to leadership  For example, the component tracked metrics on the percentage of the workforce, including IT staff, that participated in the student loan repayment and tuition assistance programs  In addition, the Chief Strategy Officer reported to the Chief Operating Officer the results related to meeting those metrics  Maintain leadership support and commitment to ensure continued progress in improving employee morale, and demonstrate sustained improvement in morale—partly implemented  Secret Service leadership developed and implemented initiatives that demonstrated their commitment to improving the morale of the Secret Service’s workforce  For example, since 2014, the Secret Service had worked with a contractor to identify ways to improve the morale of its entire workforce, including IT staff  However, as of June 2018, the Secret Service was unable to demonstrate that it had sustained improvement in the morale of the component’s IT staff  In particular, the component was only able to provide IT workforce-specific results from one employee morale assessment that was conducted subsequent to the consolidation of this workforce into OCIO in March 2017  These results were from an assessment conducted by the component’s Inspection Division in December 2017 (the assessment found that the majority of the Secret Service’s IT employees rated their morale as “very good” or “excellent ”) While the component also provided certain employee morale results from the Office of Personnel Management’s Federal Employee Viewpoint Survey in 2017, these results were not specific to the IT workforce  Instead, this workforce’s results were combined with those from staff in another Secret Service division  According to OCIO officials, the results were combined because, at the time of the survey, the IT workforce was administratively identified as being part of that other division  OCIO officials stated that, going forward, they plan to continue to assess the morale of the IT workforce on an annual basis as part of the Federal Employee Viewpoint Survey  In addition, the officials stated that OCIO-specific results may be available as part of the 2018 survey results, which the officials expect to receive by September 2018  By measuring employee satisfaction on an annual basis, the Secret Service should have increased knowledge of whether its initiatives that are aimed at improving employee morale are in fact increasing employee satisfaction  Agencies can use performance management systems as a tool to foster a results-oriented organizational culture that links individual performance to organizational goals  We have previously identified numerous leading practices related to performance management that are intended to enhance performance and ensure individual accountability  Among the performance management practices, agencies should (1) establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance, (2) explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals, and (3) periodically provide individuals with regular performance feedback  The Secret Service substantially implemented the selected three leading practices associated with the performance management workforce area  Specifically, the component fully implemented one of the three practices and partly implemented the other two practices  Table 12 lists these selected leading practices and provides our assessment of the Secret Service’s implementation of the practices  Establish a performance management system that differentiates levels of staff performance and defines competencies in order to provide a fuller assessment of performance—partly implemented  The Secret Service’s performance management process requires leadership to make meaningful distinctions between levels of staff performance  In particular, the component’s performance plans for IT staff, which are developed by the Office of Human Resources and tailored by OCIO, as necessary, specify the criteria that leadership use to determine if an individual has met or exceeded the expectations associated with each competency identified in their respective performance plan  The performance plans include pre-established, department-wide competencies that are set by DHS, as well as occupational series-specific goals that may be updated by the Secret Service  However, because OCIO did not fully define and document all of its technical competency needs for the IT workforce, as discussed earlier, the Secret Service’s performance plans for IT staff did not include performance expectations related to the full set of technical competencies required for their respective positions  In addition, because OCIO officials were unable to specify a time frame for when they will identify all of the technical competency needs for the IT workforce (as previously discussed), the officials were also unable to specify a time frame for when they would update the IT workforce’s performance plans to include those relevant technical competencies  Until OCIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed, the office will be limited in its ability to provide IT staff with a complete assessment of their performance  In addition, Secret Service management will have limited knowledge of the extent to which IT staff are meeting all relevant technical competencies  Explicitly align individual performance expectations with organizational goals to help individuals see the connection between their daily activities and organizational goals—partly implemented  The Secret Service’s performance plans for IT staff identified certain goals that appeared to be related to organizational goals and objectives  For example, the performance plan for the Telecommunications Specialist occupational series (which is one of the series included in OCIO’s IT workforce) identified a goal for staff to support the voice, wireless, radio, satellite, and video systems serving the Secret Service’s protective and investigative mission  This performance plan goal appeared to be related to the component’s strategic goal on Advanced Technology, which included an objective to create the infrastructure needed to fulfill mission responsibilities  However, the Secret Service was unable to provide documentation that explicitly showed how individual employee performance links to organizational goals, such as a mapping of the goals identified in employee performance plans to organizational goals  Specifically, while Office of Human Resources officials stated that each Secret Service directorate is responsible for ensuring that employee goals map to high-level organizational goals, OCIO officials stated that they did not complete this mapping  The officials were unable to explain why they did not align the goals in their employees’ performance plans to the component’s high-level goals  According to the officials, the Secret Service is in the process of implementing a new automated tool that will require each office to explicitly align individual performance expectations to organizational goals  The officials stated that OCIO plans to use this tool to create employees’ fiscal year 2019 performance plans  By explicitly demonstrating how individual performance expectations align with organizational goals, the Secret Service’s IT staff should have a better understanding of how their daily activities contribute towards achieving the Secret Service’s goals  Periodically provide individuals with regular performance feedback—fully implemented  Secret Service leadership periodically provided their IT staff with performance feedback  Specifically, on an annual basis, OCIO staff received feedback during a mid-year and end-of-year performance feedback assessment  In our prior work, we have stressed that candid and constructive feedback can help individuals maximize their contribution and potential for understanding and realizing the goals and objectives of an organization  Further, this feedback is one of the strongest drivers of employee engagement  According to leading practices of the Software Engineering Institute, effective program oversight includes monitoring program performance and conducting reviews at predetermined checkpoints or milestones  This is done by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels  In addition, the Software Engineering Institute previously identified leading practices for effectively monitoring the performance of agile projects  According to the Institute, agile development methods focus on delivering usable, working software frequently; as such, it is important to measure the value delivered during each iteration of these projects  To that end, the Institute reported that agile projects should be measured on velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  DHS and the Secret Service had fully implemented the selected leading practice for monitoring the performance of one program and three projects within the IITT investment, and conducting reviews of this program and these projects at predetermined checkpoints  In addition, with regard to the selected leading practice for monitoring agile projects, the Secret Service had fully implemented this practice for one of its two projects being implemented using agile and had partially implemented this practice for the other project  Table 13 provides a summary of DHS’s and the Secret Service’s implementation of these leading practices, as relevant for one program and three projects within IITT  Monitor program performance and conduct reviews at predetermined checkpoints or milestones  Consistent with leading practices, DHS and the Secret Service monitored the performance of IITT’s program and projects by comparing actual cost, schedule, and performance information against planned targets and conducting reviews at predetermined checkpoints  For example, within the Secret Service: The Enabling Capabilities program and Multi-Level Security project monitored their contractors’ costs spent to-date on a monthly basis and compared them to the total contract amounts  OCIO used integrated master schedules to monitor the schedule performance of the Enabling Capabilities program and Multi-Level Security project  OCIO also monitored the cost, schedule, and performance of the Uniformed Division Resource Management System and Events Management projects during monthly status reviews  In addition, DHS and the Secret Service conducted acquisition decision event reviews and systems engineering life cycle technical reviews of IITT’s program and projects at predetermined checkpoints and, when applicable, identified deviations from established cost, schedule, and performance targets  For example: Secret Service OCIO met with DHS’s Office of Program Accountability and Risk Management in February 2017, and with DHS’s Acting Under Secretary for Management in June 2017, to discuss a schedule breach for the Enabling Capabilities program  In particular, the Enabling Capabilities program informed DHS that the program needed to change the planned date for acquisition decision event 3 (the point at which a decision is made to fully deploy the system) in order to conduct tests in an operational environment prior to that decision event  This delay was due to the Secret Service misunderstanding the tests that it was required to conduct prior to that decision event  Specifically, the Enabling Capabilities program had conducted tests on “production representative” systems, but these tests were not sufficient to meet the requirements for acquisition decision event 3  The project team for Multi-Level Security identified that certain technical issues they had experienced would delay system deployment and full operational capability (the point at which an investment becomes fully operational)  As such, in October 2017, the project notified the Secret Service Component Acquisition Executive of these expected delays  In particular, the web browser that was intended to provide users on “Sensitive But Unclassified” workstations the ability to view information from different security levels, experienced technical delays in meeting personal identity verification requirements  The project team also described for the executive how the schedule delay would affect the project’s performance metrics and funding, and subsequently updated the project plan accordingly  Measure and monitor agile projects on, among other things, velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of features and user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  Secret Service OCIO measured its two agile projects—Uniformed Division Resource Management System and Events Management— using certain agile metrics  In particular, OCIO officials measured the Uniformed Division Resource Management System and Events Management projects using key metrics related to velocity and development progression  For example, the officials measured development progression for both projects on a daily basis  In addition, OCIO officials monitored each project’s progress against these metrics during bi-weekly reviews that they conducted with each project team  The OCIO officials also tracked product quality metrics for the Uniformed Division Resource Management System  For example, on a monthly basis, the officials tracked the number of helpdesk tickets that had been resolved related to the system  In addition, on a quarterly basis, they tracked the number of Uniformed Division Resource Management System defects that (1) had been fixed and (2) were in the backlog  However, while OCIO officials received certain post-deployment user satisfaction information from end-users of the Uniformed Division Resource Management System by, among other things, tracking the number of helpdesk tickets related to the system and via daily verbal, undocumented feedback from certain Uniformed Division officers, OCIO officials had not fully measured and documented post- deployment user satisfaction with the system, such as via a survey of employees who use the system  The officials stated that they had not conducted and documented a survey because they were focused on (1) addressing software performance issues that occurred after they deployed the system to a limited number of users, and (2) continuing system deployment to the remaining users after they addressed the performance issues  OCIO officials stated that they plan to conduct such a documented survey by the end of September 2018  The results of the user satisfaction survey should provide OCIO with important information on whether the Uniformed Division Resource Management System is meeting users’ needs  The Secret Service’s full implementation of 11 of 14 component-level CIO responsibilities constitutes a significant effort to establish CIO oversight for the component’s IT portfolio  Additional efforts to fully implement the remaining 3 responsibilities, including ensuring that all IT contracts are reviewed, as appropriate; ensuring that the Secret Service’s enterprise governance policy appropriately specifies the CIO’s role in developing and reviewing the component’s IT budget formulation and execution; and ensuring agile projects measure product quality and post-deployment user satisfaction, will further position the CIO to effectively manage the Secret Service’s IT portfolio  When effectively implemented, IT workforce planning and management activities can facilitate the successful accomplishment of an agency’s mission  However, the Secret Service had not fully implemented all of the 15 selected practices for its IT workforce for any of the five areas— strategic planning, recruitment and hiring, training and development, employee morale, and performance management  The Secret Service’s lack of (1) a strategic workforce planning process, including the identification of all required knowledge and skills, assessment of competency gaps, and targeted strategies to address specific gaps in competencies and staffing; (2) targeted recruiting activities, including metrics to monitor the effectiveness of the recruitment program and adjustment of the recruitment program and hiring efforts based on metrics; (3) a training program, including the identification of required training for IT staff, ensuring that staff take required training, and assessment of performance data regarding the training program; and (4) a performance management system that includes all relevant technical competencies, greatly limits its ability to ensure the timely and effective acquisition and maintenance of the Secret Service’s IT infrastructure and services  On the other hand, by monitoring program performance and conducting reviews at predetermined checkpoints for one program and three projects associated with the IITT investment, in accordance with leading practices, the Secret Service and DHS provided important oversight needed to guide that program and those projects  Measuring projects on leading agile metrics also provided the Secret Service CIO with important information on project performance  We are making the following 13 recommendations to the Director of the Secret Service: The Director should ensure that the CIO establishes and documents an IT acquisition review process that ensures the CIO or the CIO’s delegate reviews all contracts containing IT, as appropriate  (Recommendation 1) The Director should update the enterprise governance policy to specify (1) the CIO’s current role and responsibilities on the Executive Resources Board, to include developing and reviewing the IT budget formulation and execution; and (2) the Deputy CIO’s role and responsibilities on the Enterprise Governance Council  (Recommendation 2) The Director should ensure that the Secret Service develops a charter for its Executive Resources Board that specifies the roles and responsibilities of all board members, including the CIO  (Recommendation 3) The Director should ensure that the CIO includes product quality and post-deployment user satisfaction metrics in the modular outcomes and target measures that the CIO sets for monitoring agile projects  (Recommendation 4) The Director should ensure that the CIO identifies all of the required knowledge and skills for the IT workforce  (Recommendation 5) The Director should ensure that the CIO regularly analyzes the IT workforce to identify its competency needs and any gaps it may have  (Recommendation 6) The Director should ensure that, after OCIO completes an analysis of the IT workforce to identify any competency and staffing gaps it may have, the Secret Service updates its recruiting and hiring strategies and plans to address those gaps, as necessary  (Recommendation 7) The Director should ensure that the Office of Human Resources (1) develops and tracks metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce, including their effectiveness at addressing skill and staffing gaps; and (2) reports to component leadership on those metrics  (Recommendation 8) The Director should ensure that the Office of Human Resources and OCIO adjust their recruitment and hiring plans and activities, as necessary, after establishing and tracking metrics for assessing the effectiveness of these activities for the IT workforce  (Recommendation 9) The Director should ensure that the CIO (1) defines the required training for each IT workforce group, (2) determines the activities that OCIO will include in its IT workforce training and development program based on its available training budget, and (3) implements those activities  (Recommendation 10) The Director should ensure that the CIO ensures that the IT workforce completes training specific to their positions (after defining the training required for each workforce group)  (Recommendation 11) The Director should ensure that the CIO collects and assesses performance data (including qualitative or quantitative measures, as appropriate) to determine how the IT training program contributes to improved performance and results (once the training program is implemented)  (Recommendation 12) The Director should ensure that the CIO updates the performance plans for each occupational series within the IT workforce to include the relevant technical competencies, once identified, against which IT staff performance should be assessed  (Recommendation 13) DHS provided written comments on a draft of this report, which are reprinted in appendix III  In its comments, the department concurred with all 13 of our recommendations and provided estimated completion dates for implementing each of them  For example, with regard to recommendation 2, the department stated that the Secret Service would update its enterprise governance policy and related policies to outline the roles and responsibilities of the CIO and Deputy CIO, among others, by March 31, 2019  In addition, for recommendation 13, the department stated that the Secret Service OCIO will include relevant technical competencies in performance plans, as appropriate, in the next performance cycle that starts in July 2019  If implemented effectively, these actions should address the weaknesses we identified  The department also identified a number of other actions that it said had been taken to address our recommendations  For example, in response to recommendation 8, which calls for the Office of Human Resources to (1) develop and track metrics to monitor the effectiveness of the Secret Service’s recruitment activities for the IT workforce and (2) report to component leadership on those metrics, DHS stated that the Secret Service’s Office of Human Resources’ Outreach Branch provides to the department metrics on recruitment efforts toward designated priority mission-critical occupations  However, for fiscal year 2017, only 1 of the 12 occupational series associated with the Secret Service’s IT workforce was designated as a mission-critical occupation for the component (i e , the 2210 IT Specialist series)  The 11 other occupational series were not designated as mission- critical occupations  In addition, for fiscal year 2018, none of these 12 occupational series were designated as mission-critical occupations  As such, metrics on recruiting for these IT series may not have been reported to DHS leadership  Moreover, while we requested documentation of the recruiting metrics for the Secret Service’s IT workforce and, during the course of our review, had multiple subsequent discussions with the Secret Service regarding such metrics, the component did not provide documentation that demonstrated it had established recruiting metrics for its IT workforce  Tracking such metrics and reporting the results to Secret Service leadership, as we recommended, would provide management with important information necessary to make effective recruitment decisions  Further, in response to recommendation 10, which among other things, calls for the CIO to define the required training for each IT workforce group, the department stated that the Secret Service OCIO recently developed training requirements for each workforce group, which were issued during our audit  However, while during our audit OCIO provided a list of recommended training courses, the office did not identify them as being required courses  Defining training that is required for each IT workforce group, as we recommended, would inform OCIO of the necessary training for each position and enable the office to prioritize this training, to ensure that its staff have the needed knowledge and skills  In addition to the aforementioned comments, we received technical comments from DHS and Secret Service officials, which we incorporated, as appropriate  We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Director of the Secret Service, and other interested parties  In addition, this report is available at no charge on the GAO website at http://www gao gov  Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or HarrisCC@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix IV  Our objectives were to evaluate the extent to which: (1) the U S  Secret Service (Secret Service) Chief Information Officer (CIO) has implemented selected information technology (IT) oversight responsibilities, (2) the Secret Service has implemented leading workforce planning and management practices for its IT workforce, and (3) the Secret Service and the Department of Homeland Security (DHS) have implemented selected performance and progress monitoring practices for the Information Integration and Technology Transformation (IITT) investment  To address the first objective, we analyzed DHS’s policies and guidance on IT management to identify the responsibilities that were to be implemented by the component-level CIO related to overseeing the Secret Service’s IT portfolio, including existing systems, acquisitions, and investments  From the list of 33 responsibilities that we identified, we then excluded the responsibility that was associated with information security, which is expected to be addressed as part of a separate, subsequent GAO review  We also excluded those responsibilities that were significantly large in scope (e g , implement an enterprise architecture) or that, in our professional judgment, lacked specificity (e g , provide timely delivery of mission IT services)  As a result, we excluded from consideration for this review a total of 10 CIO responsibilities  For the 23 that remained, we then combined certain responsibilities that overlapped with other related responsibilities  For example, we combined related responsibilities on the component CIO’s review of IT contracts  As a result, we identified 14 responsibilities for review  We validated with the acting DHS CIO that these responsibilities were key responsibilities for the department’s component-level CIOs  We then included all 14 of the responsibilities in our review  The 14 selected component-level CIO responsibilities were: 1  Develop and review the component IT budget formulation and execution  2  Manage the component IT investment portfolio, including establishing an IT acquisition review process that enables component and DHS review of component acquisitions (i e , contracts) that contain IT  3  Develop, implement, and maintain a detailed IT strategic plan  4  Ensure all component IT policies are in compliance and alignment with DHS IT directives and instructions  5  Concur with each program’s and/or project’s systems engineering life cycle tailoring plan  6  Support the Component Acquisition Executive to ensure processes are established that enable systems engineering life cycle technical reviews and that they are adhered to by programs and/or projects  7  Ensure that all systems engineering life cycle technical review exit criteria are satisfied for each of the component’s IT programs and/or projects  8  Ensure the necessary systems engineering life cycle activities have been satisfactorily completed as planned for each of the component’s IT programs and/or projects  9  Concur with the systems engineering life cycle technical review completion letter for each of the component’s IT programs and/or projects  10  Maintain oversight of their component’s agile development approach for IT by appointing the responsible personnel, identifying investments for adoption, and reviewing artifacts  11  With Component Acquisition Executives, evaluate and approve the application of agile development for IT programs consistent with the component’s agile development approach  12  Set modular outcomes and target measures to monitor the progress in achieving agile implementation for IT programs and/or projects within their component  13  Participate on DHS’s CIO Council, Enterprise Architecture Board, or other councils/boards as appropriate, and appoint employees to serve when necessary  14  Meet the IT competency requirements established by the DHS CIO, as required in the component CIO’s performance plan  To determine the extent to which the Secret Service CIO has implemented these responsibilities, we obtained and assessed relevant component documentation and compared it to the responsibilities  Specifically, we obtained and analyzed documentation including evidence of the CIO’s participation on the Secret Service governance board that has final decision authority and responsibility for enterprise governance, including the IT budget; monthly program management reports showing the CIO’s oversight of IT programs, projects, and systems; monthly status reports on program spending; the Secret Service’s IT strategic plan; the Secret Service’s enterprise governance policy; meeting minutes from the DHS board and council on which the CIO participated (i e , the CIO Council and Enterprise Architecture Board); and documentation demonstrating whether the CIO met the IT competency requirements  In addition, we obtained and analyzed relevant documentation related to the CIO’s oversight of the major IT investments on which the Secret Service was spending development, modernization, and enhancement funds during fiscal year 2017  As of July 2017, the component had one investment—IITT—that met this criterion  IITT is a portfolio investment that, as of July 2017, included two programs (one of which included three projects) and one standalone project (i e , it was not part of another program) that had capabilities that were in planning or development and modernization: the Enabling Capabilities program, Enterprise Resource Management System program (which included three projects, called Uniformed Division Resource Management System, Events Management, and Enterprise-wide Scheduling), and Multi-Level Security project  In particular, we obtained and analyzed documentation related to the CIO’s oversight of the systems engineering life cycles for IITT’s Enabling Capabilities program and the Uniformed Division Resource Management System, Events Management, and Multi-Level Security projects  This documentation included acquisition program baselines, systems engineering life cycle tailoring plans, and systems engineering life cycle technical review briefings and completion letters  We then compared the documentation against the five selected systems engineering life cycle oversight responsibilities (responsibilities 5, 6, 7, 8, and 9)  We also obtained and analyzed documentation related to the CIO’s oversight of two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  Specifically, we obtained and assessed documentation of (1) the CIO’s approval for these projects to be implemented using an agile methodology and (2) the agile development metrics that the CIO established for each of these projects  We then compared this documentation to the three agile development-related component-level CIO responsibilities (responsibilities 10, 11, and 12)  Further, to determine the extent to which the Secret Service CIO had established an IT acquisition (i e , contract) review process that enabled component and DHS review of component contracts that contain IT (which is part of responsibility 2), we first asked Secret Service officials to provide us with a list of all new, unclassified IT contracts that the component awarded between October 1, 2016, and June 30, 2017  The Secret Service officials provided a list of 54 contracts  We validated that these were contracts for IT or IT services by: (1) searching for them in the Federal Procurement Data System – Next Generation; (2) identifying their associated product or service codes, as reported in that system; and (3) determining whether those codes were included in the universe of 79 IT product or service codes identified by the Category Management Leadership Council  In validating the list of 54 contracts provided by the Secret Service, we determined that 5 of the contracts were not associated with an IT product or service code  As such, we removed those contracts from the list  In addition, we found that three other items identified by the component were not in the Federal Procurement Data System – Next Generation  Secret Service officials subsequently confirmed that these three items were not contracts  We therefore removed these three items from the list  As such, the final list of validated contracts identified by the Secret Service included 46 IT contracts  In addition, to identify any IT contracts that were not included in the list provided by the Secret Service, we conducted a search of the Federal Procurement Data System – Next Generation to identify all unclassified contracts that (1) the component awarded between October 1, 2016, and June 30, 2017; (2) were not a modification of a contract; and (3) were associated with 1 of the 79 IT product or service codes identified by the Category Management Leadership Council  Based on these criteria, we identified 144 Secret Service IT contracts in the Federal Procurement Data System – Next Generation (these 144 contracts included the 46 contracts previously identified by Secret Service officials)  We then asked Secret Service officials to validate the accuracy, completeness, and reliability of these data, which they did  From each of these two lists of IT contracts (i e , the list of 46 IT contracts identified by the Secret Service and the list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation), we then selected random, non-generalizable samples of contracts, as described below  First, from the list of 46 IT contracts identified by Secret Service officials, we removed 4 contracts that had total values of less than $10,000  To ensure that we selected across all contract sizes, we randomly selected 12 contracts from the remaining list of 42 contracts, using the following cost ranges: $10,000 to $50,000 (4 contracts), more than $50,000 to less than $250,000 (4 contracts), and more than $250,000 (4 contracts)  Second, from our list of 144 IT contracts that we identified from the Federal Procurement Data System – Next Generation, we removed the 46 contracts identified by Secret Service officials  We also removed 12 contracts that had total values of less than $10,000  To ensure that we selected across all contract sizes, we randomly selected 21 contracts from the remaining list of 86 contracts, using the following cost ranges: $10,000 to $50,000 (7 contracts), more than $50,000 to less than $250,000 (7 contracts), and more than $250,000 (7 contracts)  In total, we selected 33 IT contracts for review  We separated the contracts into the three cost ranges identified above in order to ensure that contracts of different value levels had been selected  This enabled us to determine the extent to which the CIO appropriately reviewed contracts of all values  To determine the extent to which the CIO had established an IT contract approval process that enabled the Secret Service and DHS, as appropriate, to review IT contracts, we first asked Secret Service Office of the CIO (OCIO) officials for documentation of their IT contract approval process  These officials were unable to provide such documentation  Instead, the officials stated that the Secret Service CIO or the CIO’s delegate approves all IT contracts prior to award  The officials also provided documentation that identified four staff to whom the CIO had delegated his approval authority  Further, the officials stated that, in accordance with DHS’s October 2016 IT acquisition review guidance, they submitted to DHS OCIO for approval any IT contracts that met DHS’s thresholds for review, including those that (1) had total estimated procurement values of $2 5 million or more, and (2) were associated with a major investment  Based on the IT acquisition review process that Secret Service OCIO officials described, we then obtained and analyzed each of the 33 selected IT contracts and associated approval documentation to determine whether or not the Secret Service CIO or the CIO’s delegate had approved each of the contracts  In particular, we (1) reviewed the name of the contract approver on the approval documentation, and (2) compared the signature dates that were on the contracts to the signature dates that were identified on the associated approval documentation  In addition, to determine whether or not the Secret Service CIO submitted to DHS OCIO for approval the IT contracts that (1) had total estimated procurement values of $2 5 million or more, and (2) were associated with major investments, we first analyzed the 144 Secret Service IT contracts that we had previously pulled from the Federal Procurement Data System – Next Generation to determine which contracts met the $2 5 million threshold  We identified 4 contracts that met this threshold  We then requested that OCIO identify the levels (i e , major or non-major) of the investments associated with these contracts  According to OCIO officials, 3 of the 4 contracts were associated with non-major investments and 1 was not associated with an investment  As such, based on DHS’s October 2016 IT acquisition review guidance, none of these contracts needed to be submitted to DHS OCIO for review  We also interviewed Secret Service officials, including the CIO and Deputy CIO, regarding the CIO’s implementation of the 14 selected component-level responsibilities  We assessed the evidence against the selected responsibilities to determine the extent to which the CIO had implemented them  To address the second objective—determining the extent to which the Secret Service had implemented leading workforce planning and management practices for its IT workforce—we first identified seven topic areas associated with human capital management based on the following sources: The Office of Personnel Management’s Human Capital Framework  Office of Personnel Management and the Chief Human Capital Officers Council Subcommittee for Hiring and Succession Planning, End-to-End Hiring Initiative  GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others  GAO, IT Workforce: Key Practices Help Ensure Strong Integrated Program Teams; Selected Departments Need to Assess Skill Gaps  GAO, Department of Homeland Security: Taking Further Action to Better Determine Causes of Morale Problems Would Assist in Targeting Action Plans  GAO, Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government  GAO, Results-Oriented Cultures: Creating a Clear Linkage between Individual Performance and Organizational Success  DHS acquisition guidance  Secret Service acquisition guidance  Among these topic areas, we then selected five areas that, in our professional judgment, were of particular importance to successful workforce planning and management  They were also previously identified as part of our high-risk and key issues work on human capital management  These areas include: (1) strategic planning, (2) recruitment and hiring, (3) training and development, (4) employee morale, and (5) performance management  We also reviewed these same sources and identified numerous leading practices associated with the five topic areas  Among these leading practices, we then selected three leading practices within each of the five areas (for a total of 15 selected practices)  The selected practices were foundational practices that, in our professional judgment, were of particular importance to successful workforce planning and management  Table 14 identifies the five selected workforce areas and 15 selected associated practices  To determine the extent to which the Secret Service had implemented the selected leading workforce planning and management practices for its IT workforce, we obtained and assessed documentation and compared it against the 15 selected practices  In particular, we analyzed the Secret Service’s human capital strategic plan, human capital staffing plan, IT strategic plan, documentation of the component’s staffing model that it used to determine the number of IT staff needed, an independent verification and validation report on the component’s staffing models, documentation of the current number of IT staff, the Secret Service’s recruitment and outreach plans, documentation of DHS’s hiring authorities (which are applicable to the Secret Service), the Secret Service’s training strategic plan, IT workforce training plan, action plans for improving employee morale, and templates used for measuring and reporting employee performance  We also interviewed Secret Service officials—including the CIO, Deputy CIO, and workforce planning staff—about the component’s workforce- related policies and documentation  Further, we discussed with the officials the Secret Service’s efforts to implement the selected workforce practices for its IT workforce  Regarding our assessments of the Secret Service’s implementation of the 15 selected workforce planning and management practices, we assessed a practice as being fully implemented if component officials provided supporting documentation that demonstrated all aspects of the practice  We assessed a practice as not implemented if the officials did not provide any supporting documentation for that practice, or if the documentation provided did not demonstrate any aspect of the practice  We assessed a practice as being partly implemented if the officials provided supporting documentation that demonstrated some, but not all, aspects of the selected practice  In addition, related to our assessments of the Secret Service’s implementation of the five selected overall workforce areas, we assessed each area as follows, based on the implementation of the three selected practices within each area: Fully implemented: The Secret Service provided evidence that it had fully implemented all three of the selected practices within the workforce area; Substantially implemented: The Secret Service provided evidence that it had either fully implemented two selected practices and partly implemented the remaining one selected practice within the workforce area, or fully implemented one selected practice and partly implemented the remaining two selected practices within the workforce area; Partially implemented: The Secret Service provided evidence that it had partly implemented each of the three selected practices within the workforce area; Minimally implemented: The Secret Service provided evidence that it partly implemented two selected practices and not implemented the remaining one selected practice within the workforce area, or partly implemented one selected practice and not implemented the remaining two selected practices within the workforce area; or Not implemented: The Secret Service did not provide evidence that it had implemented any of the three selected practices within the workforce area  To address the third objective—determining the extent to which the Secret Service and DHS have implemented selected performance and progress monitoring practices for IITT—we reviewed leading project monitoring practices and guidance from the Software Engineering Institute  First, we reviewed the practices within the Project Monitoring and Control process area of the Institute’s Capability Maturity Model Integration® for Acquisition  Based on our review, we identified four practices associated with monitoring program performance and progress  In our professional judgment, all four of these practices were of significance to managing the IITT investment given the phase of the life cycle that the investment was in  As such, we elected to include all four of these practices in our review, and combined them into one practice, as follows: Monitor program performance and conduct reviews at predetermined checkpoints or milestones by, among other things, comparing actual cost, schedule, and performance data with estimates in the program plan and identifying significant deviations from established targets or thresholds for acceptable performance levels  Next, given the agile development methodology that the Secret Service was using for certain projects within IITT, we reviewed the Software Engineering Institute’s technical note on the progress monitoring of agile contractors  Based on our review, and in consultation with an internal expert, we selected four agile metrics that the Institute identified as important for successful agile implementations and that, in our professional judgment, were of most significance to monitoring the performance of IITT’s agile projects  We then combined these four metrics into one practice, as follows: Measure and monitor agile projects on velocity (i e , number of story points completed per sprint or release), development progression (e g , the number of features and user stories planned and accepted), product quality (e g , number of defects), and post-deployment user satisfaction  To determine the extent to which DHS and the Secret Service had implemented the first selected practice, we analyzed relevant program management and governance documentation for IITT’s Enabling Capabilities program, and Multi-Level Security, Uniformed Division Resource Management System, and Events Management projects  In particular, we analyzed acquisition program baselines, DHS acquisition decision event memorandums, artifacts from DHS and Secret Service program oversight reviews, cost monitoring reports, program integrated master schedules, and program status briefings, and compared this documentation to the selected practice  We also interviewed Secret Service OCIO officials regarding the Secret Service’s and DHS’s efforts to monitor the IITT investment’s performance and progress  To determine the extent to which the Secret Service had implemented the second selected practice related to measuring and monitoring agile projects on agile metrics (i e , velocity, development progression, product quality, and post-deployment user satisfaction), we obtained and analyzed agile-related documentation for the two projects that the Secret Service was implementing using an agile methodology—Uniformed Division Resource Management System and Events Management  Specifically, to determine the extent to which the Secret Service was measuring and monitoring these two projects on metrics for velocity and development progression, we obtained and analyzed documentation, such as sprint burndown charts and monthly program status reports, and compared it to the selected practice  In addition, the agile metrics for product quality and post-deployment user satisfaction were only applicable to projects that had been deployed to users  As such, these metrics were applicable to the Uniformed Division Resource Management System (which the Secret Service had deployed to users) and were not applicable to Events Management (which the Secret Service had not yet deployed to users, as of early May 2018)  We therefore obtained and analyzed documentation demonstrating that Secret Service OCIO measured product defects for the Uniformed Division Resource Management System  We also requested documentation demonstrating that OCIO had measured and monitored post-deployment user satisfaction for this project, including via a survey  OCIO officials stated that they had not conducted such a survey and were unable to provide documentation demonstrating they had measured post- deployment user satisfaction for the Uniformed Division Resource Management System  To assess the reliability of the cost, schedule, and agile-related data that were in DHS and the Secret Service’s program management and governance documentation for the IITT investment, we (1) analyzed related documentation and assessed the data against existing agency records to identify consistency in the information, and (2) examined the data for obvious outliers, incomplete, or unusual entries  We determined that the data in these documents were sufficiently reliable for our purpose, which was to evaluate the extent to which DHS and the Secret Service had implemented processes for monitoring the IITT investment’s performance and progress  We conducted this performance audit from May 2017 to November 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  As of June 2018, the Secret Service’s Information Integration and Technology Transformation (IITT) investment included two programs (one of which included three projects) and one project that had capabilities that were in planning or development and modernization, as described below: Enabling Capabilities  This program is intended to, among other things, (1) modernize and enhance the Secret Service’s information technology (IT) network infrastructure, including increasing bandwidth and improving the speed and reliability of the Secret Service’s IT system performance; (2) enhance cybersecurity to protect against potential intrusions and viruses; and (3) provide counterintelligence and data mining capabilities to improve officials’ ability to perform the Secret Service’s investigative mission  Enterprise Resource Management System  This program comprises three projects that are intended to provide: a system that will enable the Secret Service’s Uniformed Division to efficiently and effectively plan, provision, and schedule missions (this project is referred to as Uniformed Division Resource Management System), a system that will unify the logistical actions (e g , assigning personnel) surrounding special events that Secret Service agents need to protect, such as the United Nations General Assembly (this project is referred to as Events Management), and a capability for creating schedules for Secret Service agents and administrative, professional, and technical staff, as well as the ability to generate reports on information such as monthly hours worked (this project is referred to as Enterprise-wide Scheduling)  Multi-Level Security  This project is intended to enable authorized Secret Service users to view two levels of classified information on a single workstation  Previously, data at various security levels were contained and used in multiple disparate systems  Multi-Level Security is intended to streamline users’ access to information at different security levels in order to enable them to more quickly and effectively perform their duties  Table 15 provides the planned life cycle cost and schedule estimates (threshold values) for each IITT program and project that had capabilities in planning or development and modernization, as of June 2018  In addition, the table describes any changes in those cost and schedule estimates, as well as the key reasons for any changes, as identified by officials from the Secret Service’s Office of the Chief Information Officer  In addition to the contact named above, the following staff made key contributions to this report: Shannin O’Neill (Assistant Director), Emily Kuhn (Analyst-in-Charge), Quintin Dorsey, Rebecca Eyler, Javier Irizarry, and Paige Teigen 

ground_truth: Medicare paid $7.1 billion for 433 million laboratory tests in 2017. These tests help health care providers prevent, diagnose, and treat diseases. PAMA included a provision for GAO to review CMS's implementation of new payment rates for these tests. This report addresses, among other objectives, (1) how CMS developed the new payment rates; (2) challenges CMS faced in setting accurate payment rates and what factors may have mitigated these challenges; and (3) the potential effect of the new payment rates on Medicare expenditures. GAO analyzed 2016 Medicare claims data (the most recent data available when GAO started its work and the year on which new payment rates were based) and private-payer data CMS collected. GAO also interviewed CMS and industry officials. The Centers for Medicare & Medicaid Services (CMS) within the Department of Health and Human Services (HHS) revised the Clinical Laboratory Fee Schedule (CLFS) for 2018, establishing new Medicare payment rates for laboratory services. Prior to 2018, these rates were based on historical laboratory fees and were typically higher than the rates paid by private payers. The Protecting Access to Medicare Act of 2014 (PAMA) required CMS to develop a national fee schedule for laboratory tests based on private-payer data. To revise the rates, CMS collected data on private-payer rates from approximately 2,000 laboratories and calculated median payment rates, weighted by volume. GAO found that the median private-payer rates were lower than Medicare's maximum payment rates in 2017 for 88 percent of tests. CMS is gradually phasing in reductions to Medicare payment rates, limited annually at 10 percent over a 3-year period (2018 through 2020), as outlined in PAMA. CMS relied on laboratories to determine whether they met data reporting requirements, but agency officials told GAO that CMS did not receive data from all laboratories required to report. CMS did not estimate the amount of data it should have received from laboratories that were required to report but did not. CMS took steps to exclude inaccurate private-payer data and estimated how collecting certain types and amounts of additional private-payer data could affect Medicare expenditures. However, it is not known whether CMS's estimates reflect the actual risk of incomplete data resulting in inaccurate Medicare payment rates. GAO found that PAMA's phased in reductions to new Medicare payment rates likely mitigated this risk of inaccurate Medicare payment rates from 2018 through 2020. However, GAO found that collecting incomplete data could have a larger effect on the accuracy of Medicare payment rates in future years when PAMA allows for greater payment-rate reductions. CMS's implementation of the new payment rates could lead Medicare to pay billions of dollars more than is necessary and result in CLFS expenditures increasing from what Medicare paid prior to 2018 for two reasons. First, CMS used the maximum Medicare payment rates in 2017 as a baseline to start the phase in of payment-rate reductions instead of using actual Medicare payment rates. This resulted in excess payments for some laboratory tests and, in some cases, higher payment rates than those Medicare previously paid, on average. GAO estimated that Medicare expenditures from 2018 through 2020 may be $733 million more than if CMS had phased in payment-rate reductions based on the average payment rates in 2016. Second, CMS stopped paying a bundled payment rate for certain panel tests (groups of laboratory tests generally performed together), as was its practice prior to 2018, because CMS had not yet clarified its authority to do so under PAMA, according to officials. CMS is currently reviewing whether it has the authority to bundle payment rates for panel tests to reflect the efficiency of conducting a group of tests. GAO estimated that if the payment rate for each panel test were unbundled, Medicare expenditures could increase by as much as $10.3 billion from 2018 through 2020 compared to estimated Medicare expenditures using lower bundled payment rates for panel tests. GAO recommends that the Administrator of CMS (1) collect complete private-payer data from all laboratories required to report or address the estimated effects of incomplete data, (2) phase in payment-rate reductions that start from the actual payment rates rather than the maximum payment rates Medicare paid prior to 2018, and (3) use bundled rates for panel tests. HHS concurred with GAO's first recommendation, neither agreed nor disagreed with the other two, and has since issued guidance to help address the third. GAO believes CMS should fully address these recommendations to prevent Medicare from paying more than is necessary.