prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.



Report 2: Concerned that the federal government was more focused on program activities and processes than the results to be achieved, Congress passed the Government Performance and Results Act of 1993 (GPRA)  GPRA sought to focus federal agencies on performance by requiring agencies to develop long-term and annual goals, and measure and report on progress towards those goals annually  Based on our analyses of the act’s implementation, we concluded in March 2004 that GPRA’s requirements had laid a solid foundation for results-oriented management  At that time, we found that performance planning and measurement had slowly yet increasingly become a part of agencies’ cultures  For example, managers reported having significantly more performance measures in 2003 than in 1997, when GPRA took effect government-wide  However, the benefit of collecting performance information is fully realized only when that information is actually used by managers to make decisions aimed at improving results  Although our 2003 survey found greater reported availability of performance information than in 1997, it also showed managers’ use of that information for various management activities generally had remained unchanged  Based on those results, and in response to a request from Congress, in September 2005, we developed a framework intended to help agencies better incorporate performance information into their decision making  As shown in figure 1, we identified five leading practices that can promote the use of performance information for policy and program decisions; and four ways agency managers can use performance information to make program decisions aimed at improving results  Our September 2005 report also highlighted examples of how agencies had used performance information to improve results  For example, we described how the Department of Transportation’s National Highway Traffic Safety Administration used performance information to identify, develop, and share effective strategies that increased national safety belt usage—which can decrease injuries and fatalities from traffic accidents— from 11 percent in 1985 to 80 percent in 2004  Subsequently, the GPRA Modernization Act of 2010 (GPRAMA) was enacted, which significantly expanded and enhanced the statutory framework for federal performance management  The Senate Committee on Homeland Security and Governmental Affairs report accompanying the bill that would become GPRAMA stated that agencies were not consistently using performance information to improve their management and results  The report cited the results of our 2007 survey of federal managers  That survey continued to show little change in managers’ use of performance information  The report further stated that provisions in GPRAMA are intended to address those findings and increase the use of performance information to improve performance and results  For example, GPRAMA requires certain agencies to designate a subset of their respective goals as their highest priorities—known as agency priority goals—and to measure and assess progress toward those goals at least quarterly through data-driven reviews  Our recent work and surveys suggest that data-driven reviews are having their intended effect  For example, in July 2015, we found that agencies reported that their reviews had positive effects on progress toward agency goals and efforts to improve the efficiency of operations, among other things  In addition, for those managers who were familiar with their agencies’ data-driven reviews, our 2013 and 2017 surveys showed that the more managers viewed their programs as being subject to a review, the more likely they were to report their agencies’ reviews were driving results and conducted in line with our leading practices  Recognizing the important role these reviews were playing in improving data-driven decision making, our management agenda for the presidential and congressional transition in 2017 included a key action to expand the use of data-driven reviews beyond agency priority goals to other agency goals  More broadly, our recent surveys of federal managers have continued to show that reported government-wide uses of performance information generally have not changed or in some cases have declined  As we found in September 2017, and as illustrated in figure 2, the 2017 update to our index suggests that government-wide use of performance information did not improve between 2013 and 2017  In addition, it is statistically significantly lower relative to our 2007 survey, when we created the index  Moreover, in looking at the government-wide results on the 11 individual survey questions that comprise the index, we found few statistically significant changes in 2017 when compared to (1) our 2013 survey or (2) the year each question was first introduced  For example, in comparing 2013 and 2017 results, two questions had results that were statistically significantly different: The percentage of managers who reported that employees who report to them pay attention to their agency’s use of performance information was statistically significantly higher (from 40 to 46 percent)  The percentage of managers who reported using performance information to adopt new program approaches or change work processes was statistically significantly lower (from 54 to 47 percent)  As we stated in our September 2017 report, the decline on the latter question was of particular concern as agencies were developing plans to improve their efficiency, effectiveness, and accountability, as called for by an April 2017 memorandum from OMB  In early 2017, the administration announced several efforts intended to improve government performance  OMB issued several memorandums detailing the administration’s plans to improve government performance by reorganizing the government, reducing the federal workforce, and reducing federal agency burden  As part of the reorganization efforts, OMB and agencies were to develop government-wide and agency reform plans, respectively, designed to leverage various GPRAMA provisions  For instance, the April 2017 memorandum mentioned above stated that OMB intends to monitor implementation of the reforms using, among other things, agency priority goals  While many agency-specific organizational improvements were included in the President’s fiscal year 2019 budget, released in February 2018, OMB published additional government-wide and agency reform proposals in June 2018  The President’s Management Agenda (PMA), released in March 2018, outlines a long-term vision for modernizing federal operations and improving the ability of agencies to achieve outcomes  To address the issues outlined in the PMA, the administration established a number of cross-agency priority (CAP) goals  CAP goals, required by GPRAMA, are to address issues in a limited number of policy areas requiring action across multiple agencies, or management improvements that are needed across the government  The PMA highlights several root causes for the challenges the federal government faces  Among them is that agencies do not consistently apply data-driven decision-making practices  The PMA states that smarter use of data and evidence is needed to orient decisions and accountability around service and results  To that end, in March 2018, the administration established the Leveraging Data as a Strategic Asset CAP goal to improve the use of data in decision making to increase the federal government’s effectiveness  Over the past 25 years, various organizations, roles, and responsibilities have been created by executive action or in law to provide leadership in federal performance management  At individual agencies and across the federal government, these organizations and officials have key responsibilities for improving performance, as outlined below  OMB: At least every four years, OMB is to coordinate with other agencies to develop CAP goals—such as the one described earlier on leveraging data as an asset—to improve the performance and management of the federal government  OMB is also required to coordinate with agencies to develop annual federal government performance plans to define, among other things, the level of performance to be achieved toward the CAP goals  Following GPRAMA’s enactment, OMB issued guidance for initial implementation, as required by the act, and continues to provide updated guidance in its annual Circular No  A-11, additional memorandums, and other means  Chief Operating Officer (COO): The deputy agency head, or equivalent, is designated as the COO, with overall responsibility for improving agency management and performance through, among other things, the use of performance information  President’s Management Council (PMC): The PMC is comprised of OMB’s Deputy Director for Management and the COOs of major departments and agencies, among other individuals  Its responsibilities include improving overall executive branch management and implementing the PMA  Performance Improvement Officer (PIO): Agency heads designate a senior executive as the PIO, who reports directly to the COO  The PIO is responsible for assisting the head of the agency and COO to ensure that agency goals are achieved through, among other things, the use of performance information  Performance Improvement Council (PIC): The PIC is charged with assisting OMB to improve the performance of the federal government  It is chaired by the Deputy Director for Management at OMB and includes PIOs from each of the 24 Chief Financial Officers Act agencies, as well as other PIOs and individuals designated by the chair  Among its responsibilities, the PIC is to work to resolve government-wide or cross-cutting performance issues, and facilitate the exchange among agencies of practices that have led to performance improvements  Previously, the General Service Administration’s (GSA) Office of Executive Councils provided analytical, management, and administrative support for the PIC, the PMC, and other government-wide management councils  In January 2018, the office was abolished and its functions, staff, and authorities, along with those of the Unified Shared Services Management Office, were reallocated to GSA’s newly created Shared Solutions and Performance Improvement Office  As at the government-wide level—where, as described earlier, the use of performance information did not change from 2013 to 2017—managers’ reported use of performance information at most agencies also did not improve since 2013 (illustrated in figure 3)  At the agency level, 3 of the 24 agencies had statistically significant changes in their index scores—1 increase (National Science Foundation) and 2 decreases (Social Security Administration and the Office of Personnel Management)  Also, in 2017, 6 agencies had results that were statistically significantly different—4 higher and 2 lower—than the government-wide average (see sidebar)  Throughout the report, we highlight two different types of statistically significant results—changes from our last survey in 2013 and differences from the 2017 government-wide average  The former indicates when an agency’s reported use of performance information or leading practices has measurably improved or declined  The latter indicates when it is statistically significantly higher or lower than the rest of government  These results suggest agencies have taken actions that led to improvements in their use of performance information  For example, when a result is a statistically significant increase since 2013, as with the National Science Foundation index score in 2017, this suggests that the agency has adopted practices that led to a measurable increase in the use of performance information by managers  When a result is statistically significantly higher than the government-wide average, like GSA’s 2017 index score, this suggests that the agency’s use of performance information is among the highest results when compared to the rest of government  These agencies could also have insights into practices that led to relatively high levels of performance information use  Finally, when a result is a statistically significant decrease since 2013, as with the Social Security Administration’s index score in 2017, or statistically significantly lower than the government-wide average, like the Department of Homeland Security’s 2017 index score, this suggests the agencies face challenges that are hampering their ability to use performance information  Appendix III provides each agency’s index scores from 2007, 2013, and 2017 to show changes between survey years  When we disaggregated the index and analyzed responses from the 11 questions that comprise the index—which could help pinpoint particular actions that improved the use of performance information—we similarly found relatively few changes in agencies’ recent results  Specifically, we identified 16 instances where agency responses on individual questions were statistically significantly different from 2013 to 2017—10 increases and 6 decreases  This represents about 6 percent of the total possible responses to the 11 survey questions from each of the agencies  In addition, we found 12 instances where an agency’s result on a question was statistically significantly higher (11) or lower (1) than the government-wide average in 2017  For example, the percentage of Social Security Administration (SSA) managers reporting that their peers use performance information to share effective approaches was statistically significantly higher than the government-wide average  Although SSA’s index score had a statistically significant decline in 2017 compared to 2013, the agency’s index score remains relatively high, as it has in prior years  The scope of our work has not allowed us to determine definitively what factors caused the decline in SSA’s index score and whether the decline is likely to continue, although its result on this particular question may indicate a continued strength  Each agency’s results on the 11 questions that comprise the index are presented in appendix I  The agencies’ respective statistically significant results are identified in figure 4  While some agencies had statistically significant improvements on individual questions, and could point to actions that led to improvements in their use of performance information, these improvements should be considered in relation to the range of agency results and the government- wide average  In figure 4, there are five agencies with statistically significant increases on responses to individual questions, where those results were not statistically significantly higher than the government-wide average (see arrows without plus signs for the Departments of Agriculture, Defense, and Justice; the Environmental Protection Agency; and the National Science Foundation)  While these represent improvements, they should be considered in relation to the range of agency results and the government-wide average (provided in detail in the agency summaries in appendix I)  For example, in 2017, the percentage of managers at the Department of Agriculture who reported that upper management use performance information to inform decisions about program changes was statistically significantly higher than in 2013  However, the department’s 2017 result (37 percent) was relatively lower when compared to the maximum agency result on that question (60 percent)  Appendix I presents the results on the index and the 11 questions that comprise it for each of the 24 agencies  When we compared government-wide and agency-level results on selected survey questions that reflect practices that promote the use of performance information, we found that results between 2013 and 2017 generally remained unchanged  As described earlier, there are 10 survey questions that both reflect the five leading practices identified in our past work and had statistically significant associations with higher index scores  As shown in figure 5, government-wide results on 2 of the 10 questions were statistically significantly different, both increases, from 2013 to 2017  Despite these two increases, the overall results suggest these practices are not widely followed government-wide  On most of the 10 questions, only about half (or fewer) of the managers reported their agencies were following them to a “great” or “very great” extent  When we analyzed agency-level responses to these 10 questions, we also found relatively few changes in recent results  Specifically, our analysis found 20 instances—16 increases and 4 decreases—where agencies’ responses on individual questions were statistically significantly different from 2013 to 2017  This represents about 8 percent of the total possible responses to the 10 survey questions from each of the agencies  In addition, we found 10 instances where an agency’s result on a question was statistically significantly higher (8) or lower (2) than the government-wide average in 2017  Each agency’s results on these 10 questions are presented in appendix I, and the statistically significant results are identified in figure 6  Those agencies with results on individual questions that are either statistically significantly higher than 2013, higher than the 2017 government-wide average, or both may have taken actions in line with our leading practices for promoting the use of performance information  For example, the National Science Foundation had both types of statistically significant results on a question about having sufficient information on the validity of their performance data  Here, the agency’s result increased 27 percentage points from 2013 to 2017  While the scope of our review does not allow us to definitively determine the reasons for the National Science Foundation’s higher results, they suggest the agency has taken recent actions that greatly improved the availability and accessibility of information on the validity of performance data  In both 2013 and 2017, our analyses found this particular question to be the strongest predictor of higher performance information use when we tested for associations between the questions that reflect leading practices and our index  Our 2017 survey results show that managers who reported their programs were subject to data-driven reviews also were more likely to report using performance information in decision making to a greater extent (see figure 7)  For the 35 percent of managers who reported being familiar with data-driven reviews, those who reported their programs had been subject to data-driven reviews to a “great” or “very great” extent had index scores that were statistically significantly higher than those whose programs were subject to these reviews to a lesser extent  Similarly, we found that being subject to data-driven reviews to a greater extent was also related to greater reporting of agencies following practices that can promote the use of performance information  As figure 8 shows, managers who reported their programs were subject to these reviews to a “great” or “very great” extent more frequently reported that their agencies followed the five leading practices that promote the use of performance information, as measured by the 10 related survey questions associated with higher scores on the index  For example, of the estimated 48 percent of managers who reported their programs were subject to data-driven reviews to a “great” or “very great” extent, 72 percent also reported that managers at their level (peers) effectively communicate performance information on a routine basis to a “great” or “very great” extent  Conversely, for the 24 percent of managers who reported their programs were subject to data-driven reviews to a “small” or “no” extent, only 30 percent reported that managers at their level do this to a “great” or “very great” extent  Our past work has found that the Executive Branch has taken steps to improve the use of performance information in decision making by senior leaders at federal agencies  However, our survey results indicate those steps have not led to similar improvements in use by managers at lower levels  Through its guidance to implement GPRAMA, OMB developed a framework for performance management in the federal government that involves agencies setting goals and priorities, measuring performance, and regularly reviewing and reporting on progress  This includes expectations for how agency senior leaders should use performance information to assess progress towards achieving agency priority goals through data-driven reviews, and strategic objectives through strategic reviews  For example, GPRAMA requires, and OMB’s guidance reinforces, that data-driven reviews should involve the agency head, Chief Operating Officer, Performance Improvement Officer, and other senior officials responsible for leading efforts to achieve each goal  OMB’s guidance also identifies ways in which agency leaders should use the results of those reviews to inform various decision-making activities, such as revising strategies, formulating budgets, and managing risks  Our past work also found that agencies made progress in implementing these reviews and using performance information  In July 2015, we found that agencies generally were conducting their data-driven reviews in line with GPRAMA requirements and our related leading practices, including that agency leaders used the reviews to drive performance improvement  In addition, in September 2017, we reported on selected agencies’ experiences in implementing strategic reviews and found that the reviews helped direct leadership attention to progress on strategic objectives  Despite those findings, our survey results continue to show that the reported use of performance information by federal managers has generally not improved, and actually declined at some agencies  This could be because of the two different groups of agency officials covered by our work  GPRAMA’s requirements, and the federal performance management framework established by OMB’s guidance, apply at the agency-wide level and generally involve senior leaders  Our past work reviewing implementation of the act therefore focused on improvements in the use of performance information by senior leaders at the agency- wide level  In contrast, our surveys covered random samples of mid- and upper-level managers within those agencies, including at lower organizational levels such as component agencies  Their responses indicate that the use of performance information more broadly within agencies—at lower organizational levels—generally has not improved over time  The exception to this was managers whose programs were subject to the data-driven reviews required by GPRAMA  As described above, those managers were more likely to report greater use of performance information in their agencies  This reinforces the value of the processes and practices put in place by GPRAMA  Our survey results suggest that limited actions have been taken to diffuse processes and practices related to the use of performance information to lower levels within federal agencies, where mid-level and senior managers make decisions about managing programs and operations  Although OMB staff agreed that diffusing processes and practices to lower levels could lead to improved use of performance information, they told us they have not directed agencies to do so for a few reasons  First, OMB staff expressed concerns about potentially imposing a “one-size-fits- all” approach on agencies  They stated that agencies are best positioned to improve their managers’ use of performance information, given their individual and unique missions and cultures, and the environments in which they operate  We agree that it makes sense for agencies to be able to tailor their approaches for those reasons  OMB’s existing guidance provides an overarching framework that recognizes the need for flexibility and for agencies to tailor their approaches  Moreover, given the long- standing and cross-cutting nature of this challenge, a government-wide approach also would provide a consistent focus on improving the use of performance information more extensively within agencies  OMB staff also told us that they believed it would go beyond their mandate to direct agencies to extend GPRAMA requirements to lower levels  GPRAMA requires OMB to provide guidance to agencies to implement its requirements, which only apply at the agency-wide level  As noted earlier, however, GPRAMA also requires OMB to develop cross- agency priority (CAP) goals to improve the performance and management of the federal government  The President’s Management Agenda established a CAP goal to leverage data as a strategic asset, in part, to improve the use of data for decision making and accountability throughout the federal government  This new CAP goal presents an opportunity for OMB and agencies to identify actions to expand the use of performance information in decision making throughout agencies  As of June 2018, the action plan for implementing the Leveraging Data as a Strategic Asset CAP goal is limited  According to the President’s Management Agenda and initial CAP goal action plan, the goal primarily focuses on developing and implementing a long-term, enterprise-wide federal data strategy to better govern and leverage the federal government’s data  It is through this strategy that, among other things, the administration intends to improve the use of data for decision making and accountability  However, the strategy is under development and not expected to be released until January 2019, with a related plan to implement it expected in April 2019  The existing action plan, released in March 2018 and updated in June 2018, does not yet include specific steps needed to improve the use of data—including performance information—more extensively within agencies  According to the action plan for the goal, potential actions currently under consideration focus on establishing agency “learning agendas” that prioritize the development and use of data and other evidence for decision-making; building agency capacity to use data and other evidence; and improving the timeliness of performance information and other data, and making that information available to decision makers and the public  Although developing learning agendas and building capacity could help improve the use of performance information in agencies, improving availability of data may be less effective  For example, as our past survey results have shown, increasing the availability of performance information has not resulted in corresponding increases in its use in decision making  We recognize that the CAP goal was created in March 2018  Nonetheless, it is important that OMB and its fellow goal leaders develop the action plan and related federal data strategy consistent with all key requirements to better ensure successful implementation  The action plan does not yet include complete information related to the following GPRAMA requirements: performance goals that define the level of performance to be achieved each year for the CAP goal; the various federal agencies, organizations, programs, and other activities that contribute to the CAP goal; performance measures to assess overall progress towards the goal as well as the progress of each agency, program, and other activity contributing to the goal; and clearly defined quarterly targets  Consistent with GPRAMA, Standards for Internal Control in the Federal Government identifies information that agencies are required to include in their plans to help ensure they achieve their goals  The standards state that objectives—such as improving the use of data in decision making— should be clearly defined to enable the identification of risks  Objectives are to be defined in specific terms so they can be understood at all levels of the entity—in this case, government-wide as well as within individual agencies  This involves defining what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement  Ensuring that future updates to the new CAP goal’s action plan includes all required elements is particularly important, as our previous work has found that some past CAP goal teams did not meet all planning and reporting requirements  For example, in May 2016 we found that most of the CAP goal teams we reviewed had not established targets for all performance measures they were tracking  This limited the transparency of their efforts and the ability to track progress toward established goals  We recommended that OMB, working with the Performance Information Council (PIC), report on actions that CAP goal teams are taking, or plan to take, to develop such targets and performance measures  OMB staff generally agreed and, in July 2017, told us they were working, where possible, to assist the development of measures for CAP goals  However, the recommendation has not been addressed and OMB staff said the next opportunity to address it would be when the administration established new CAP goals (which took place in March 2018)  Following the initial release of the new CAP goals, CAP goal teams are to more fully develop the related action plans through quarterly updates  Given the ongoing importance of meeting these planning and reporting requirements, we will continue to monitor the status of actions to address this recommendation as implementation of the new CAP goals proceeds  While the PIC, which is chaired by OMB, has contributed to efforts to enhance the use of performance information, our survey results identify additional opportunities to further those efforts  The PIC’s past efforts have included hosting various working groups and learning events for agency officials to provide performance management guidance, and developing resources with relevant practices  For example, the PIC created a working group focused on agency performance reviews, which was used to share recommendations for how agencies can implement reviews, along with a guide with practices for effectively implementing strategic reviews  In January 2018, staff supporting the PIC joined with staff from another GSA office to create a new group called Fed2Fed Solutions  This group consults with agencies and provides tailored support, such as data analysis and performance management training for agency officials, to help them address specific challenges related to organizational transformation, data-driven decision making, and other management improvement efforts  Our survey results identify useful information related to potential promising practices and challenges that OMB and the PIC could use to inform efforts to enhance the use of performance information more extensively within agencies (e g , at lower levels)  As was previously described, the PIC has responsibilities to (1) facilitate the exchange among agencies of proven practices, and (2) work to resolve government- wide or cross-cutting performance issues, such as challenges  Our analyses of 2017 survey results identified instances where agencies may have found effective ways to enhance the use of performance information by agency leaders and managers in decision making, as well as instances where agencies (and their managers) face challenges in doing so  Specifically, based on analyses of our survey responses, we identified 14 agencies that may have insights into specific practices that led to recent improvements in managers’ use of performance information, or ways that they maintain relatively high levels of use by their managers when compared to the rest of the government  Figure 9 summarizes the agencies identified earlier in the report that had statistically significant increases, or results higher than the government-wide average, on our index or individual survey questions  As the figure shows, several agencies had statistically significant results across all three sets of analyses and therefore may have greater insights to offer: the General Services Administration, National Aeronautics and Space Administration, and the National Science Foundation  In addition, our analyses identified nine agencies where results suggest managers face challenges that have hampered their ability to use performance information  Figure 10 summarizes the agencies identified earlier in the report that had statistically significant decreases, or results lower than the government-wide average, on our index or individual survey questions  As the figure shows, the Office of Personnel Management had statistically significant decreases in all three sets of analyses  Four agencies—the Departments of the Treasury and Veterans Affairs, the Nuclear Regulatory Commission, and the Social Security Administration—were common to both of the figures above  That is, they had results that indicate they may have insights on some aspects of using performance information and face challenges in other aspects  As was mentioned earlier, to provide proper context, these results should be considered in relation to the range of agency results and the government- wide average (provided in detail in the agency summaries in appendix I)  Given the prioritization of other activities, such as the recent creation of the Fed2Fed Solutions program, the PIC has not yet undertaken a systematic approach that could improve the use of performance information by managers at lower levels within agencies  Such an approach would involve identifying and sharing practices that have led to improved use, as well as identifying common or cross-cutting challenges that have hampered such use  The results of our analyses could help the PIC do so, and in a more targeted manner  By identifying and sharing proven practices, the PIC could further ensure that agency leaders and managers are aware of effective or proven ways they can use performance information to inform their decisions across the spectrum of activities they manage within their agencies  Those proven practices also may help agency leaders and managers resolve any identified challenges  Furthermore, in September 2017, we found that, for the estimated 35 percent of managers who reported familiarity with data-driven reviews, the more they viewed their programs being subject to a review, the more likely they were to report the reviews were driving results and were conducted in line with our leading practices for using performance information  Despite the reported benefits of and results achieved through data-driven reviews, they were not necessarily widespread  As noted above, GPRAMA requires agencies to conduct such reviews for agency priority goals, which represent a small subset of goals, and they are required at the departmental level  These reasons may explain why most managers reported they were not familiar with the reviews  As a result, we recommended that OMB should work with the PIC to identify and share among agencies practices for expanding the use of data-driven reviews  OMB staff agreed with our recommendation but have yet to address it  In June 2018, OMB updated its annual guidance to agencies to explicitly encourage them to expand data-driven reviews to include other goals, priorities, and management areas as applicable to improve organizational performance  However, as of June 2018, OMB and the PIC have yet to take any steps to identify and share practices for expanding the use of these reviews in line with our recommendation  Given the additional analyses we conducted for this report—which show that being subject to data-driven reviews is related to greater reported use of performance information and leading practices that promote such use—we continue to believe these further actions would help agencies implement these reviews more extensively  We reiterate the importance of the September 2017 recommendation and will continue to monitor OMB’s progress to address it  For more than 20 years, our work has highlighted weaknesses in the use of performance information in federal decision making  While the Executive Branch has taken some actions in recent years, such as establishing a framework for performance management across the federal government, our survey results underscore that more needs to be done to improve the use of performance information more extensively within agencies and government-wide  The President’s Management Agenda and its related CAP goal to leverage data as a strategic asset present an opportunity to do so, as it aims to improve data-driven decision making  As OMB and its fellow goal leaders more fully develop the action plan for achieving this goal, providing additional details for its plans to improve data-driven decision making would help provide assurance that it can be achieved  As part of those initiatives, our survey results could provide a useful guide for targeting efforts  Officials at each agency could use these results to identify areas for additional analysis and potential actions that could help improve the use of performance information across the agency and at lower levels  Similarly, OMB and the PIC could use the results to identify broader issues in need of government-wide attention  It will also be important, however, for OMB and the PIC to go beyond this analysis and work with agencies to identify and share proven practices for increasing the use of performance information at lower levels within agencies, as well as challenges that may be hampering agencies’ ability to do so  We are making the following two recommendations to OMB: The Director of OMB should direct the leaders of the Leveraging Data as a Strategic Asset CAP Goal to ensure future updates to the action plan, and the resulting federal data strategy, provide additional details on improving the use of data, including performance information, more extensively within federal agencies  The action plan should identify performance goals; contributing agencies, organizations, programs, and other activities; those responsible for leading implementation within these contributors; planned actions; time frames; and means to assess progress  (Recommendation 1) The Director of OMB, in coordination with the PIC, should prioritize efforts to identify and share among agencies proven practices for increasing, and challenges that hamper, the use of performance information in decision making more extensively within agencies
Report 2:  Program offices responsible for 7 of the 8 selected mission-essential systems did not conduct such a review  According to OCIO officials, the offices were not aware of the need for a review  However, without reviewing physical access logs, CDC has reduced assurance that the agency would detect suspicious physical access activities  Further, CDC had not routinely monitored the configuration settings of its systems to ensure that the configurations were securely set  For example, for at least 41 of 94 technical control deficiencies we identified, OCIO officials cited quality control gaps where the change management process or system administrators had not discovered deficiencies resulting from insecure configuration settings  Without an effective monitoring process in place for system configurations, the agency was not aware of insecure system configurations  FISMA requires each agency to periodically test and evaluate the effectiveness of its information security policies, procedures, and practices  The law also requires agencies to test the management, operational, and technical controls for every system identified in the agency’s required inventory of major information systems at a frequency depending on risk, but no less than annually  In addition, NIST SP 800- 53A identifies three assessment methods—interview, examine, and test— and describes the potential depth and coverage for each  Assessing a control’s effectiveness based on an interview is likely less rigorous than examining a control; similarly, examining a control is likely less rigorous than testing the control’s functionality  CDC had not sufficiently tested or assessed the effectiveness of the security controls for the 8 mission-essential systems that we reviewed  Although CDC annually assessed security controls of selected systems, the agency had only examined control descriptions in security plans to ensure accuracy  At least once every 3 years, the agency selected controls for a more in-depth assessment of the 8 mission-essential systems we reviewed  However, CDC had assessed only 191 (about 7 percent) of 2,818 controls described in the security plans for the selected systems  In addition, the agency used methods for assessing controls that were often not rigorous enough to identify the control deficiencies that we identified  For example, as depicted in figure 1, CDC relied exclusively on interviews—a less rigorous method—to assess 20 percent of the 191 controls it assessed for the selected systems  The security control tests and assessments were insufficient in part because CDC had not developed comprehensive security assessment plans or had not consistently implemented the plans for the 8 selected mission-essential systems we reviewed  For example, one system’s assessment plan indicated that five controls should be assessed using a testing methodology; instead, however, the assessor conducted interviews to determine whether controls were effective or not  OCIO officials stated that the security control test and assessment process is manual and staffing is limited  They stated that the agency intends to rely increasingly on automated tools—such as the tools implemented by the Continuous Diagnostics and Mitigation program—for performing the assessments  Nevertheless, by not assessing controls in an in-depth and comprehensive manner, CDC has limited assurance that the security controls are in place and operating as intended  Further, without developing and implementing comprehensive assessment plans, assessments may not be performed with sufficient rigor to identify control deficiencies  The respond core security function is intended to support the ability to contain the impact of a potential cybersecurity event  Controls associated with this function include implementing an incident response capability and remediating newly-identified deficiencies  Although CDC had implemented controls for incident response to detect cybersecurity events, we reported in June 2018 that the agency had not maintained adequate information to support its incident response capability or taken timely corrective actions to remediate identified control deficiencies  NIST SP 800-53 and SP 800-61 state that agencies should develop and document an incident response policy with corresponding implementation procedures and an incident response plan, and keep them updated according to agency requirements  NIST also states that agencies should implement an incident handling capability, including an incident response team that consists of forensic/malicious code analysts  In addition, agencies are to provide incident response training for the team and test the incident response capability to determine the effectiveness of the response  Further, NIST states that agencies are to monitor incidents by tracking and documenting them and maintain records about each incident, including forensic analysis  Finally, National Archives and Records Administration guidance states that records and data relevant to security incident investigations should be retained for 3 years  CDC had implemented an incident response capability  The agency had developed policy, procedures, and a plan that addressed incident response, and updated them annually  CDC had an incident response team that managed all of the incident handling and response efforts for the agency, and conducted forensic analyses for reported security incidents  Team members had undergone training, such as an advanced network forensic and analysis course offered by a private firm  In addition, the agency had periodically tested its incident handling capability by conducting penetration testing exercises  These exercises allowed the team to test its real-time response capabilities  CDC’s incident response procedures state that incident tickets should include a description of actions taken, response time, and whether actions have been completed or not  The agency’s procedures also require that computers affected by an incident be removed from the network immediately  Nevertheless, CDC had shortcomings in implementing its incident response capability and monitoring procedures  For the 11 security incidents CDC considered most significant over a 19-month period ending in March 2017, the agency had not consistently described the actions taken, the response times, or whether remedial actions had been completed  The agency also had not maintained audit log records for its security incidents  For example, the agency described recommended actions for 10 of the 11 incidents, but did not describe the actions that had been taken  In addition, although incident response team officials told us that all incident ticket records had been saved, CDC had not retained system log data that supported incident resolution for at least five of the incidents  The agency’s policy did not address record retention in accordance with National Archives and Records Administration guidance  Further, for two of the security incidents, the security incident tickets did not clearly indicate when two compromised workstations had been removed from the network  According to OCIO officials, shortcomings in fully documenting incidents resulted from the organization being understaffed, primarily due to budget limitations and the inability to hire qualified personnel  Without effectively tracking and documenting information system security incidents, CDC’s systems are at increased risk that the impact of security incidents would not be fully addressed  FISMA requires each agency to develop, document, and implement an information security program that, among other things, includes a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in information security policies, procedures, or practices  NIST SP 800-53 states that agencies are to develop a plan of action and milestones (POA&M) for an information system to document the agency’s planned remedial actions to correct identified deficiencies  CDC policy was consistent with the NIST guidelines  CDC had developed POA&Ms for deficiencies identified by its security control assessments, but had not remediated the deficiencies in a timely manner  For each of the 8 selected mission-essential systems, the agency had created plans for correcting control deficiencies  However, the agency did not implement several remedial actions by their due date  For example, expected completion dates had passed for correcting deficiencies associated with 4 of the 8 selected mission-essential systems  For these 4 systems, the completion dates were 1 to 8 months beyond the due dates at the time of our review in September 2017  According to Office of the Chief Information Security Officer officials, program offices that own the systems did not always communicate updates on the status of remedial actions for their respective systems, noting that deficiencies may have been corrected  Without effective communication to update its POA&Ms, CDC was not in a position to effectively manage its remedial actions and correct known deficiencies in a timely manner  The recover core security function is intended to support timely recovery of normal operations to reduce the impact from a cybersecurity event  Controls associated with this function include developing and testing contingency plans to ensure that, when unexpected events occur, critical operations can continue without interruption or can be promptly resumed, and that information resources are protected  Losing the capability to process, retrieve, and protect electronically maintained information can significantly affect an agency’s ability to accomplish its mission  If contingency planning is inadequate, even relatively minor interruptions can result in lost or incorrectly processed data, which can cause financial losses, expensive recovery efforts, and inaccurate or incomplete information  NIST SP 800-53 states that agency systems should have a contingency plan that includes the identification of key personnel and the systems’ essential mission functions and addresses full information system restoration  For high-impact systems, NIST specifies that agencies test contingency plans at an alternate processing site that is separated from the primary processing site to reduce susceptibility to the same threats  In addition, NIST states that organizations should initiate corrective actions based on testing if they are needed  As we reported in June 2018, CDC had developed and fully tested contingency plans for each of the 8 selected mission-essential systems that we reviewed  Each plan identified key personnel and their contact information, essential mission functions of the systems, and instructions on how to fully restore the systems in the event of a disruption  Additionally, between January 2015 and May 2017, CDC had tested whether the 8 systems could be recovered at their respective alternate sites, and had initiated corrective actions based on the results of the tests  However, the alternate site for 6 of the 8 selected mission-essential systems was located in relatively close proximity to the main processing site  Although 2 systems had alternate sites located in another state, the alternate site for the other 6 systems was within the same metropolitan area  As a result, an event such as a natural disaster or substantial power outage could affect both the main and alternate sites for these systems, potentially rendering CDC unable to complete functions associated with its mission  Prompt restoration of service is necessary because the required recovery time for these systems ranged from 4 to 24 hours  Security plans for 3 of the systems recognized the hazards of having the sites within the same geographical region, but stated that CDC had accepted this risk  According to OCIO officials, having a site further away was cost prohibitive; however, the officials had not documented this analysis or the associated risk of having the agency’s processing sites located within the same geographical area  Without documenting the analysis and associated risk, CDC had less assurance that senior leadership was aware of the risk of agency systems being unavailable  As a consequence, senior leadership may not agree whether acceptance of the risk was warranted  An underlying reason for the information security deficiencies in selected systems was that, although the agency had developed and documented an agency-wide information security program, it had not consistently or effectively implemented elements of the program  FISMA requires each agency to develop, document, and implement an information security program that, among other things, includes the following elements: periodic assessments of the risk and magnitude of the harm that could result from the unauthorized access, use, disclosure, disruption, modification, or destruction of information and information systems that support the operations and assets of the agency; policies and procedures that (1) are based on risk assessments, (2) cost-effectively reduce information security risks to an acceptable level, (3) ensure that information security is addressed throughout the life cycle of each system, and (4) ensure compliance with applicable requirements; plans for providing adequate information security for networks, facilities, and systems or group of information systems, as appropriate; security awareness training to inform personnel of information security risks and of their responsibilities in complying with agency policies and procedures, as well as training personnel with significant security responsibilities for information security; periodic testing and evaluation of the effectiveness of information security policies, procedures, and practices, to be performed with a frequency depending on risk, but no less than annually, and that includes testing of management, operational, and technical controls for every system identified in the agency’s required inventory of major information systems; a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in the information security policies, procedures, or practices of the agency; and plans and procedures to ensure continuity of operations for information systems  As discussed previously in this report, CDC had implemented aspects of each of these elements  For example, the agency had conducted risk assessments, developed security plans, assessed security controls, developed remedial action plans, and developed and tested contingency plans for each of the 8 selected mission-essential systems  In addition, the agency had documented numerous policies and procedures and ensured that staff had completed annual security awareness training  However, CDC’s program had shortcomings  For example, as discussed earlier in this report, CDC had not consistently or effectively: addressed threats, technical constraints, and the changing threat environment in its system risk assessments, or assessed the risk of having alternate processing sites within close proximity to each other; documented detailed technical requirements in policies and procedures, or facility controls in facility security plans; tracked and trained staff with significant security responsibilities; monitored configuration settings and comprehensively assessed remediated deficiencies in a timely manner; or documented its cost analysis and associated risk of having an alternate processing site within the same geographical region as its primary processing site  Until CDC addresses these shortcomings and consistently and effectively implements all elements of its information security program, the agency will lack reasonable assurance that its computing resources are protected from inadvertent or deliberate misuse  In our June 2018 report, we made 195 recommendations to CDC to strengthen its technical security controls and bolster its agency-wide information security program  Specifically, we recommended that the agency take 184 actions to resolve technical control deficiencies by implementing stronger access controls, encrypting sensitive data, configuring devices securely, applying patches in a timely manner, strengthening firewall rules, and implementing logging and monitoring controls more effectively, among other actions  We also made 11 recommendations for CDC to improve its information security program by, among other things, assessing risks as needed, documenting more detailed technical requirements, monitoring and assessing controls more comprehensively, and remediating deficiencies in a timely manner  Since the issuance of our June 2018 report, CDC has made significant progress in implementing the recommendations we made to resolve the technical security control deficiencies in the information systems we reviewed and to improve its information security program  In this regard, the agency has implemented many of the recommendations for improving technical security controls for the systems we reviewed and has developed plans to implement recommendations for enhancing its information security program  Specifically, as of August 3, 2018, CDC had fully implemented 102 (55 percent) of the 184 recommendations we made to fortify the technical security controls over the systems we reviewed  In addition, the agency had partially implemented 20 (11 percent) of the 184 recommendations  In these instances, CDC had made progress toward implementing the recommendations, but had not completed all of the necessary corrective actions for us to close the recommendations  Therefore, these recommendations remain open  Further, CDC did not provide any evidence that it had implemented the remaining 62 technical control- related recommendations  Table 3 summarizes the status of CDC’s efforts to implement the 184 recommendations that we made to resolve the technical control deficiencies, as of August 3, 2018  By implementing 102 recommendations, CDC (as of August 3, 2018) reduced some of the risks associated with certain key activities  Specifically, these efforts included protecting network boundaries and logging and monitoring security events for indications of inappropriate or unusual activity on systems—that we highlighted in our June 2018 report as being particularly vulnerable and requiring the agency’s greater priority and attention  In addition, the agency had implemented several of our recommendations to rectify a number of the security control deficiencies  These efforts included strengthening firewall rules, implementing stronger access controls, configuring devices securely, and expanding its audit monitoring capabilities  In addition, CDC had developed a plan of action and milestones (POA&M) for each of the identified technical control deficiencies and related recommendations that remained open as of August 3, 2018  The POA&Ms assigned organization responsibilities, identified estimated costs, identified points of contact, and established time frames for resolving the deficiencies and closing the related recommendations  The agency’s plans called for it to implement the majority of the remaining open technical control-related recommendations by September 2019, and all recommendations by September 2020, as shown in figure 2  Our June 2018 report also included 11 recommendations to CDC to improve its information security program  In particular, we recommended that the agency, among other things, evaluate system impact level categorizations to ensure they reflect the current operating environment; update risk assessments to identify threats and the likelihood of impact of the threat on the environment; and update the facility risk assessments  In addition, we recommended that the agency take the necessary steps to make sure staff with significant security roles and responsibilities are appropriately identified and receive role-based training; monitor the configuration settings of agency systems to ensure the settings are set as intended; update security control assessments to include an assessment of controls using an appropriate level of rigor; and remediate POA&Ms in a timely manner  Further, we recommended that the agency document the cost-benefit analysis with associated risk of having an alternate site within the same geographical region as the main site  As of August 3, 2018, the agency had partially implemented 1 of the 11 information security program-related recommendations, but had not provided any evidence that it had implemented the remaining 10 recommendations  Regarding the partially implemented recommendation, CDC had provided role-based training to all personnel performing significant security responsibilities  However, the agency still needed to establish and automate the identification process and the tracking of training records for individuals needing specialized security role-based training  CDC had developed plans to fully implement this recommendation and each of the remaining 10 information security program-related recommendations by July 2019  Fully implementing the open recommendations is essential to ensuring that the agency’s systems and sensitive information are not at increased and unnecessary risk of unauthorized use, disclosure, modification, or disruption  We received written comments on a draft of this report from CDC  In its comments, which are reprinted in appendix III, the agency stated that it recognizes the risks associated with operating a large, global information technology enterprise and has implemented processes, procedures, and tools to better ensure the prevention, detection, and correction of potential incidents  CDC also said cybersecurity remains a high priority and that it takes the responsibilities for protecting public health information and data entrusted to it seriously  To strengthen its cybersecurity program, the agency stated that it is restructuring and streamlining the cyber program and IT infrastructure of its Office of the Chief Information Officer  Further, CDC stated that it has leveraged GAO’s limited official use only report, issued in June 2018, to accelerate its implementation, infrastructure, and software deployments to complete phrases one and two of DHS’s Continuous Diagnostics and Mitigation program  The agency also said it concurred with, and highlighted a number of actions that it had planned or begun taking to remediate, the 11 security program recommendations that we made to CDC in our June 2018 report  We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and the department’s Office of the Inspector General, the Director of CDC, and interested congressional parties  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact Gregory C  Wilshusen at (202) 512-6244 or wilshuseng@gao gov, or Dr  Nabajyoti Barkakati at (202) 512-4499 or barkakatin@gao gov  GAO staff who made key contributions to this report are listed in appendix IV  Our objective was to assess the extent to which CDC had effectively implemented an information security program and controls to protect the confidentiality, integrity, and availability of its information on selected information systems  In June 2018, we issued a report which detailed the findings from our work in response to this objective  In the report, we made 184 recommendations to CDC to resolve the technical security control deficiencies in the information systems we reviewed and 11 additional recommendations to improve its information security program  We designated that report as “limited official use only” (LOUO) and did not release it to the general public because of the sensitive information it contained  This report publishes the findings discussed in our June 2018 report, but we have removed all references to the sensitive information  Specifically, we deleted the names of the information systems and computer networks that we examined, disassociated identified control deficiencies from named systems, deleted certain details about information security controls and control deficiencies, and omitted an appendix that was contained in the LOUO report  The appendix contained sensitive details about the technical security control deficiencies in the CDC’s information systems and computer networks that we reviewed, and the 184 recommendations we made to mitigate those deficiencies  We also provided a draft of this report to CDC officials to review and comment on the sensitivity of the information contained herein and to affirm that the report can be made available to the public without jeopardizing the security of CDC’s information systems and networks  In addition, this report addresses a second objective that was not included in the June 2018 report  Specifically, this objective was to determine the extent to which CDC had taken corrective actions to address the previously identified security program and technical control deficiencies and related recommendations for improvement that we identified in the earlier report  As noted in our June 2018 report, we determined the extent to which CDC had effectively implemented an information security program and controls to protect the confidentiality, integrity, and availability of its information on selected information systems  To do this, we initially gained an understanding of the overall network environment, identified interconnectivity and control points, and examined controls for the agency’s networks and facilities  We conducted site visits at two CDC facilities in Atlanta, Georgia  To evaluate CDC’s controls over its information systems, we used our Federal Information System Controls Audit Manual, which contains guidance for reviewing information system controls that affect the confidentiality, integrity, and availability of computerized information  We based our assessment of controls on requirements identified by the Federal Information Security Modernization Act of 2014 (FISMA), which establishes key elements for an effective agency-wide information security program; NIST guidelines and standards; Department of Health and Human Services and CDC policies, procedures, and standards; and standards and guidelines from relevant security organizations, such as the National Security Agency, the Center for Internet Security, and the Interagency Security Committee  We had reviewed a non-generalizable sample of the agency’s information systems, focusing on those systems that (1) collect, process, and maintain private or potentially-sensitive proprietary business, medical, and personally identifiable information; (2) are essential to CDC’s mission; and (3) were assigned a Federal Information Processing Standard rating of moderate or high impact  Based on these criteria, we had selected eight mission-essential systems for our review  Of these systems, the agency had categorized 7 as high-impact systems and 1 as a moderate-impact system  For these 8 selected mission- essential systems, we had reviewed information security program-related controls associated with risk assessments, security plans, security control assessments, remedial action plans, and contingency plans  To assess the safeguards CDC implemented for its systems, we had examined technical security controls for 24 CDC systems, including systems the agency designated as high-value assets  These included 10 key systems, 8 of which were high- and moderate-impact mission- essential systems just described, 1 additional high-impact system, 1 additional moderate-impact system, and 14 general support systems  We selected the additional high-impact system because the agency re- categorized it as a high-impact system during our review  We selected the additional moderate-impact system because the agency used it to control physical access to highly sensitive CDC biologic lab facilities, including facilities that handle dangerous and exotic substances that cause incurable and deadly diseases  We selected 10 key systems, 8 of which were mission-essential systems, for review that (1) collect, process, and maintain private or potentially sensitive proprietary business, medical, and personally identifiable information; (2) are essential to CDC’s mission; (3) could have a catastrophic or severe impact on operations if compromised; or (4) could be of particular interest to potential adversaries  We also selected 14 general support systems that were part of the agency’s network infrastructure supporting the 10 key systems  To review controls over the 10 key systems and 14 general support systems, we had examined the agency’s network infrastructure and assessed the controls associated with system access, encryption, configuration management, and logging and monitoring  For reporting purposes, we had categorized the security controls that we assessed into the five core security functions described in the National Institute of Standards and Technology’s (NIST) cybersecurity framework  The five core security functions are: Identify: Develop the organizational understanding to manage cybersecurity risk to systems, assets, data, and capabilities  Protect: Develop and implement the appropriate safeguards to ensure delivery of critical infrastructure services  Detect: Develop and implement the appropriate activities to identify the occurrence of a cybersecurity event  Respond: Develop and implement the appropriate activities to take action regarding a detected cybersecurity event  Recover: Develop and implement the appropriate activities to maintain plans for resilience and to restore any capabilities or services that were impaired due to a cybersecurity event  These core security functions are described in more detail in appendix II  For the identify core security function, we had examined CDC’s reporting for its hardware and software assets; analyzed risk assessments for the eight selected mission-essential systems to determine whether threats and vulnerabilities were being identified; reviewed risk assessments for two facilities; analyzed CDC policies, procedures, and practices to determine their effectiveness in providing guidance to personnel responsible for securing information and information systems; and analyzed security plans for the eight selected systems to determine if those plans had been documented and updated according to federal guidance  We also evaluated the risk assessments for two facilities that housed the 8 mission-essential selected systems  For the protect core security function, we had examined access controls for the 24 systems  These controls included the complexity and expiration of password settings to determine if password management was being enforced; administrative users’ system access permissions to determine whether their authorizations exceeded the access necessary to perform their assigned duties; firewall configurations, among other things, to determine whether system boundaries had been adequately protected; and physical security controls to determine if computer facilities and resources were being protected from espionage, sabotage, damage, and theft  We also had examined configurations for providing secure data transmissions across the network to determine whether sensitive data were being encrypted  In addition, we had examined configuration settings for routers, network management servers, switches, firewalls, and workstations to determine if settings adhered to configuration standards, and inspected key servers and workstations to determine if critical patches had been installed and/or were up-to-date  Further, we had examined training records to determine if employees and contractors had received security awareness training according to federal requirements, and whether personnel who have significant security responsibilities had received training commensurate with those responsibilities  For the detect core security function, we had analyzed centralized logging and network traffic monitoring capabilities for key assets connected to the network; analyzed CDC’s procedures and results for assessing security controls to determine whether controls for the eight selected mission- essential systems had been sufficiently tested at least annually and based on risk  We also had reviewed the agency’s implementation of continuous monitoring practices to determine whether the agency had developed and implemented a continuous monitoring strategy to manage its information technology assets and monitor the security configurations and vulnerabilities for those assets  For the respond core security function, we had reviewed CDC’s implementation of incident response practices, including an examination of incident tickets for 11 incidents; and had examined the agency’s process for correcting identified deficiencies for the eight selected mission-essential systems  For the recover core security function, we had examined contingency plans for eight selected mission-essential systems to determine whether those plans had been developed and tested  In assessing CDC’s controls associated with this function, as well as the other four core functions, we had interviewed Office of the Chief Information Officer officials, as needed  Within the core security functions, as appropriate, we had evaluated the elements of CDC’s information security program based on elements required by FISMA  For example, we analyzed risk assessments, security plans, security control assessments, and remedial action plans for each of the 8 selected mission-essential systems  In addition, we had assessed whether the agency had ensured staff had completed security awareness training and whether those with significant security responsibilities received commensurate training  We also had evaluated CDC’s security policies and procedures  To determine the reliability of CDC’s computer-processed data for training and incident response records, we had evaluated the materiality of the data to our audit objective and assessed the data by various means, including reviewing related documents, interviewing knowledgeable agency officials, and reviewing internal controls  Through a combination of methods, we concluded that the data were sufficiently reliable for the purposes of our work  To accomplish our second objective—on CDC’s actions to address the previously identified security program and technical control deficiencies and related recommendations—we requested that the agency provide a status report of its actions to implement each of the recommendations  For each recommendation that CDC indicated it had implemented as of August 3, 2018, we examined supporting documents, observed or tested the associated security control or procedure, and/or interviewed the responsible agency officials to assess the effectiveness of the actions taken to implement the recommendation or otherwise resolve the underlying control deficiency  Based on this assessment and CDC status reports, we defined the status of each recommendation into the following 3 categories: closed-implemented—CDC had implemented the recommendation; open-partially implemented—CDC had made progress toward, but had not completed, implementing the recommendation; and open-not implemented—CDC had not provided evidence that it had acted to implement the recommendation  We conducted this performance audit from December 2016 to December 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The National Institute of Standards and Technology’s cybersecurity framework consists of five core functions: identify, protect, detect, respond, and recover  Within the five functions are 23 categories and 108 subcategories, as described in the table  In addition to the individuals named above, Gary Austin, Jennifer R  Franks, Jeffrey Knott, and Chris Warweg (assistant directors); Chibuikem Ajulu-Okeke, Angela Bell, Sa’ar Dagani, Nancy Glover, Chaz Hubbard, George Kovachick, Sean Mays, Kevin Metcalf, Brandon Sanders, Michael Stevens, Daniel Swartz, and Angela Watson made key contributions to this report  Edward Alexander, Jr  and Duc Ngo (assistant directors); David Blanding, and Christopher Businsky also provided assistance 

ground_truth: Medicaid has grown by over 50 percent over the last decade, with about $370 billion in federal spending in fiscal year 2017. CMS is responsible for assuring that expenditures—reported quarterly by states—are consistent with Medicaid requirements and matched with the correct amount of federal funds. CMS's review of reported expenditures has become increasingly complex due to variation in states' Medicaid programs and an increasing number of different matching rates. GAO was asked to examine CMS's oversight of state-reported Medicaid expenditures. In this report, GAO examined how CMS assures that (1) expenditures are supported and consistent with requirements; and (2) the correct federal matching rates were applied to expenditures subject to a higher match. GAO also examined the financial impact of resolved errors. GAO reviewed documentation for the most recently completed quarterly reviews by 3 of CMS's 10 regional offices for six states that varied by Medicaid program expenditures and design. GAO also reviewed policies, procedures, and data on resolved errors; and interviewed CMS and state officials. GAO assessed CMS's oversight processes against federal standards for internal control. The Centers for Medicare & Medicaid Services (CMS), which oversees Medicaid, has various review processes in place to assure that expenditures reported by states are supported and consistent with Medicaid requirements. The agency also has processes to review that the correct federal matching rates were applied to expenditures receiving a higher than standard federal matching rate, which can include certain types of services and populations. These processes collectively have had a considerable federal financial benefit, with CMS resolving errors that reduced federal spending by over $5.1 billion in fiscal years 2014 through 2017. However, GAO identified weaknesses in how CMS targets its resources to address risks when reviewing whether expenditures are supported and consistent with requirements. CMS devotes similar levels of staff resources to review expenditures despite differing levels of risk across states. For example, the number of staff reviewing California's expenditures—which represent 15 percent of federal Medicaid spending—is similar to the number reviewing Arkansas' expenditures, which represents 1 percent of federal Medicaid spending. CMS cancelled in-depth financial management reviews in 17 out of 51 instances over the last 5 years. These reviews target expenditures considered by CMS to be at risk of not meeting program requirements. CMS told GAO that resource constraints contributed to both weaknesses. However, the agency has not completed a comprehensive assessment of risk to (1) determine whether oversight resources are adequate and (2) focus on the most significant areas of risk. Absent such an assessment, CMS is missing an opportunity to identify errors in reported expenditures that could result in substantial savings to the Medicaid program. GAO also found limitations in CMS's processes for reviewing expenditures that receive a higher federal matching rate. Internal guidance for examining variances in these expenditures was unclear, and not all reviewers in the three CMS regional offices GAO reviewed were investigating significant variances in quarter-to-quarter expenditures. Review procedures for expenditures for individuals newly eligible for Medicaid under the Patient Protection and Affordable Care Act were not tailored to different risk levels among states. For example, in its reviews of a sample of claims for this population, CMS reviewed claims for the same number of enrollees—30—in California as for Arkansas, even though California had 10 times the number of newly eligible enrollees as Arkansas. Without clear internal guidance and better targeting of risks in its review procedures for expenditures receiving higher matching rates, CMS may overpay states. GAO is making three recommendations, including that CMS improve its risk-based targeting of oversight efforts and resources, and clarify related internal guidance. The Department of Health and Human Services concurred with these recommendations.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: In order to receive federal matching funds, states report expenditures quarterly to CMS on the CMS-64  States are required to report their expenditures to CMS within 30 days of the end of each quarter, but may adjust their past reporting for up to 2 years after the expenditure was made, referred to as the 2-year filing limit  Adjustments can reflect resolved disputes or reclassifications of expenditures  Expenditures reported after the 2-year filing limit are generally not eligible for a federal match, with certain exceptions  The CMS-64 is a series of forms that capture expenditure data for different aspects of states’ Medicaid programs, such as different types of services, populations, and different federal matching rates  (See table 1 for examples of the expenditure types captured by the CMS-64 ) States report their expenditures quarterly on the CMS-64 at an aggregate level— such as a state’s total expenditures for such categories of services as inpatient hospital services—and these reported expenditures are not linked to individual enrollees or services  States’ reporting may vary depending on the features of their Medicaid program  Some examples of this variation include the following: States that expanded eligibility under PPACA would need to report expenditures not only by the type of services (e g , inpatient hospital services), but also by populations receiving different federal matching rates, such as expansion enrollees  States with waivers—that is, where the state received approval from HHS to waive certain Medicaid requirements in order to test and evaluate new approaches for delivering and financing care under a demonstration—would need to report those expenditures associated with these waivers on additional forms  CMS is responsible for assuring that expenditures reported by states are supported and allowable, meaning that the state actually made and recorded the expenditure and that the expenditure is consistent with Medicaid requirements  CMS regional offices perform the ongoing oversight, with enhanced oversight procedures in the 20 states with the highest federal Medicaid expenditures  (See fig 1) CMS is required to review the expenditures reported by states each quarter  (See fig  2 ) Regional office reviewers have 50 days to review the expenditures and compute the federal share of states’ Medicaid expenditures  As part of the quarterly review, regional office reviewers also check that expenditures receive the correct matching rate  In general, the amount of federal funds that states receive for Medicaid services is determined annually by a statutory formula—the Federal Medical Assistance Percentage (FMAP), which results in a specific federal matching rate for each state  However, there are a number of exceptions where higher federal matching rates can apply for certain types of beneficiaries, services, or administrative costs  See table 2 for examples of higher matching rates that apply for expenditures for certain types of enrollees, services, or administrative costs  When CMS identifies questionable expenditures or errors through its reviews, there are several ways that they can be resolved, as summarized below  Deferral of federal funds  CMS can defer federal matching funds if, during the quarterly review, the regional office reviewer needs additional information to determine whether a particular expenditure is allowable  The reviewer may recommend that CMS defer the expenditure until the state provides additional support or corrects the reporting  State reducing reported expenditures  If the state agrees that the questionable expenditure is an error, the state can submit an adjusted report during the quarterly review or make an adjustment in a subsequent quarter  These adjustments prevent federal payments for those expenditures  Disallowance of expenditure  If CMS determines an expenditure is not allowable, CMS can issue a disallowance, and the state returns federal funds through reductions in future federal allocations  States may appeal disallowances  CMS uses a variety of processes to assure that state-reported expenditures are supported during quarterly reviews and performs focused financial management reviews on expenditures considered at risk of not complying with Medicaid requirements  Although we found that CMS was identifying errors and compliance issues using both review methods, we also found weaknesses in how CMS targets its oversight resources to address risks  CMS uses quarterly reviews to assess whether expenditures are supported by the state’s accounting systems; are in accordance with CMS approved methodologies, plans, and spending caps; and whether there are significant unexplained variances—changes in expenditures— from one quarter to the next (referred to as a variance analysis)  CMS review procedures include validation measures that check to ensure that expenditures were reported within the 2-year limit, which is a check done on all types of expenditures  Another validation measure compares expenditures to various approval documents  For example, when a state has a waiver in place, expenditures are reviewed against waiver agreements that authorize payment for specified services or populations  Other examples include comparing supplemental payment expenditures to caps set for those expenditures  (See table 3 ) Our examination of the quarterly reviews indicated that the reviews involved significant coordination with other CMS staff and the state  In addition to reviewing state documentation, officials from two regional offices told us that they consult other regional office staff who oversee the approval of new expenditures to ensure that expenditures reflect approved program features  For example, officials in region 9 told us that in reviewing managed care expenditures, they consult with their colleagues who review the state’s payment methodologies for capitated payments  In reviewing information technology development expenditures—which are subject to a higher federal matching rate— reviewers for all six selected states examined advanced planning documents, which requires coordination with staff who approve those documents to ensure that the state was receiving the correct matching rates and staying within the approved amounts  With regard to coordination with states, we found that regional reviewers for all six reviews contacted states to follow-up on issues identified during the review  Officials also described being in regular contact with states to stay abreast of program, system, and staffing changes to inform their reviews  For example, according to regional officials, Arkansas experienced some significant and unexpected staffing challenges in 2016 that resulted in delays in the state reporting expenditures and returning federal overpayments, and the reviewer worked closely with state staff to track the state’s progress  We found evidence that reviewers identified errors during their quarterly reviews  In the six quarterly reviews we examined, regional offices identified errors in three of the six states  For example, region 3 reviewers found errors in Maryland’s expenditure reporting—including claims for the wrong matching rate for two enrollees who were not eligible for PPACA’s Medicaid expansion and reporting provider incentive payments on the wrong line—and worked with the state to correct those errors  Additionally, region 9 reviewers found errors in California’s reporting of expenditures  For example, they found that the state reported waiver expenditures for the incorrect time period, which has implications for CMS’s ability to monitor and enforce spending limits for the waiver  Reviewers worked with the state to correct those errors  To supplement the quarterly reviews, CMS generally directs regional offices to conduct a focused financial management review (FMR) each year on an area of high risk within the region, typically within one state  According to regional officials, CMS uses these reviews to investigate expenditures in greater depth and detail than is reasonable within the timeframes of a quarterly review  For example, reviewers can examine individual claims for services from providers or the methodologies developed for certain payment types  Regional reviewers also use these reviews to investigate errors that could not have been detected by the quarterly review  For example, regional office 6 officials told us that they uncovered inappropriate financing arrangements when they used an FMR to examine how Texas financed the state share of its supplemental payments to hospitals in one of its counties  To do so, the regional office reviewed payments from the state to the provider, project plans, and interviewed providers—steps that are not part of the quarterly review process  Rather, in the quarterly review, the reviewer only checks that state-reported payments are supported by state accounting records and are within applicable caps; thus, inappropriate financing of the state share would not have been detected through the quarterly review  In fiscal years 2014 through 2017, CMS used FMRs to review various expenditures considered to be at risk for not complying with Medicaid requirements  Specifically, as outlined in annual work plans, regional offices planned to conduct 31 FMRs and estimated that the total amount of federal funds at risk in expenditure areas covered by their planned reviews was $12 billion  (See app  I ) Planned FMRs targeted a wide range of topics, with the reviews most frequently targeting expenditures for the Medicaid expansion population  (See table 4 ) We found that CMS frequently identified compliance issues through FMRs  As of March 2018, CMS reported that reviewers had identified compliance issues with financial impact in 11 of the 31 planned FMRs, though most of those findings were still under review  More findings from the planned FMRs are likely as some of the reviews were still ongoing  We reviewed the draft results for 5 FMRs  Among these, CMS found that four states were reporting expenditures that were not allowable  For example, as noted earlier, a 2014 FMR on supplemental payments in Texas revealed inappropriate funding arrangements, and CMS issued a disallowance for approximately $27 million  In some cases, FMRs did not have apparent financial findings, but identified significant internal control weaknesses in the state and recommended specific corrective actions— such as better aligning eligibility and expenditures systems to better detect and correct irregularities—that would provide greater assurances that federal funds are appropriately spent  Both the quarterly reviews and the FMRs occur in conjunction with other ongoing CMS financial oversight activities  For example in addition to reviewing expenditures, regional office reviewers assess how states estimate their costs, set payment rates for managed care and home and community based services, and allocate costs among different Medicaid administrative activities under their cost allocation plans  CMS officials told us that issues relating to state compliance with Medicaid requirements for expenditures could be identified during these other oversight activities and could inform follow-up during the quarterly reviews or be the subject of a FMR  Officials also told us that since FMRs were instituted, the agency has built in more front-end procedures for preventing problems with the accuracy and allowability of reported expenditures  As examples, they cited their work on managed care rate reviews, among other things  We identified two weaknesses in how CMS is allocating resources for overseeing state-reported expenditures that limited the agency’s ability to target risk in its efforts to assure that these expenditures are supported and consistent with Medicaid requirements  First, we found that CMS has allocated similar staff resources to states with differing levels of risk  For example, the staff resources dedicated to reviewing California’s expenditures—ranking first nationally in expenditures and constituting 15 percent of all federal Medicaid expenditures—are comparable to significantly smaller states in other regions, despite California’s history of reporting challenges and its inability to provide electronic records, which requires on-site review  (See fig  3 ) CMS has allocated 2 2 staff to review California’s expenditures in contrast to one person to review Arkansas’ expenditures, which constitute 1 percent of federal Medicaid expenditures, and Arkansas does not have a similar history of complex reporting challenges  We also found that California’s reviewers have set a higher threshold for investigating variances in reported expenditures than in the five other selected states  Specifically, reviewers investigated variances in California of plus or minus 10 percent if the variances represented more than 2 percent of medical expenditures, or $450 million in the quarter we reviewed  The state experienced an approximately 24 percent increase in its prescription drug expenditures—roughly $200 million—during that quarter, but the variance was deemed not significant  In contrast, for two of our five other selected states, we found that reviewers generally investigated variances of plus or minus 10 percent regardless of the dollar amount of the variance and in the remaining three states they had significantly lower dollar thresholds than used for California  Second, CMS reported cancelling the FMR requirement for regional offices in 17 out of 51 instances in the last 5 years when faced with resource constraints  In some cases, CMS excused individual regional offices from conducting planned FMRs due to staff shortage as the agency did for regions 3 and 7 in 2014; region 8 in 2016; and regions 3, 7, 8, and 9 in 2018  In 2015, according to CMS officials, all 10 regions were excused from conducting an FMR, because the regional offices needed their staff to focus on implementing new procedures for validating expenditures for the Medicaid expansion population  In addition to cancelling FMRs, CMS was delayed in finalizing FMRs  Among the eight FMRs that were conducted in fiscal year 2014, three have been issued as final reports, CMS decided no report was needed on a fourth, and the four remaining FMRs from 2014 were still under review as of March 2018, delaying important feedback to states on their vulnerabilities  According to CMS officials, resource constraints have contributed to both of these weaknesses  Our analysis of staffing data indicated that, from fiscal years 2014 to 2018, the number of full time equivalent staff dedicated to financial oversight activities declined by approximately 19 percent across all 10 regions  These staff are responsible not only for completing the quarterly reviews and FMRs, but also other financial oversight activities, including resolving audit findings and other on-going oversight activities noted previously  During this period, federal Medicaid expenditures are estimated to have increased by approximately 31 percent, and the reporting of expenditures has grown more complex  In addition to the decline in dedicated staff, officials told us they faced challenges in filling vacancies either because of hiring restrictions or challenges in recruiting qualified candidates  Officials described instances where regional offices shared resources with other offices to address critical gaps in resources  For example region 9 was able to obtain part-time assistance from a region 6 reviewer to help review California’s expenditures  However, CMS officials told us that they had not permanently reallocated resources between regional offices, because all regional offices are under-resourced given their various oversight responsibilities as of May 2018  With regard to cancelling FMRs, CMS officials noted that other oversight responsibilities, including the quarterly reviews, are required under statute or regulation and thus have a higher priority than FMRs  Compounding its resource allocation challenges, CMS has not conducted a comprehensive, national assessment of risk to determine whether resources for financial oversight activities are (1) adequate and (2) allocated—both across regional offices and oversight tools—to focus on the greatest areas of risk  Agency officials told us that they have not conducted a formal risk assessment, because they are assessing risk on an on-going basis, allocating resources within each region accordingly and sharing resources across regions to the extent possible  However, this approach does not make clear whether the level of resources dedicated to financial oversight nationally is adequate given the risk  Federal internal control standards for risk assessment require agencies to identify and analyze risks related to achieving the defined objectives (i e , assuring that state-reported expenditures are in accordance with Medicaid rules), and respond to risks based on the significance of the risk  Without completing a comprehensive, national assessment of risk and determining whether staff resources dedicated to financial oversight are adequate and allocated commensurate with risk, CMS is missing an opportunity to improve its ability to identify errors in reported expenditures that could result in hundreds of millions of dollars in potential savings to the Medicaid program  CMS reviewers in the selected regional offices we reviewed did not consistently perform variance analyses—which compare changes in expenditures from the quarter under review to the previous quarter—of higher matched expenditures during quarterly reviews  Further, the sampling procedures used to examine Medicaid expansion expenditures did not account for varying risks across states  CMS has multiple procedures in place to review expenditures that receive a higher federal matching rate  As with other expenditures, reviewers are required to complete a variance analysis, comparing reported expenditures in the quarter under review to those reported in the prior quarter and investigating variances above a certain threshold  However, we found that our three selected regional offices were not consistently conducting these analyses across several different types of expenditures with higher matching rates  While CMS’s internal guidance required that regional offices conduct variance analyses on expenditures with higher matching rates, we found that for the quarter we investigated (generally the 1st quarter of fiscal year 2017), our selected regional offices did not consistently do so for three types of expenditures that we reviewed: IHS, family planning, and certain women with breast or cervical cancer  Two of the three regional offices (regions 3 and 9) did not conduct or did not document these required variance analyses, and the remaining regional office (region 6) conducted the analyses but deviated from standard procedures outlined in CMS guidance, as summarized below  CMS region 3  Reviewers did not conduct variance analyses for either Maryland or Pennsylvania  Regional office staff with whom we spoke said that as part of the quarterly review they conduct the standard variance analysis on category of service lines of the CMS-64  Expenditures for IHS, family planning and services for certain women with breast or cervical cancer are not separately identified at that level  Although CMS reviewers said they thought the standard analysis was sufficient, net changes within a broad service category may obscure major changes within these higher matched expenditures  For example, examining changes in total inpatient hospital expenditures would not necessarily reveal a significant variance limited to inpatient expenditures in IHS facilities that receive a higher federal match  CMS region 9  Reviewers told us that they examined higher matched expenditures for California; however, no variance analyses of IHS, family planning, or breast or cervical cancer services were included in the work papers provided to us  In addition, they told us that they do not conduct a variance analysis on IHS, family planning, and services for certain women with breast or cervical cancer for Nevada, noting that expenditures in these areas tend to be quite small  CMS region 6  Reviewers conducted a variance analysis of these higher matched expenditures for Arkansas and Texas and provided us documentation; however, the documentation showed some deviation from the required steps specified in CMS’s guidance  For example, for Texas, spending on two of the three categories was beyond the threshold for significance, but the reviewer did not document any follow-up with the state  Although expenditures for IHS, family planning, and certain women with breast or cervical cancer constituted a small share of total federal spending on Medicaid services—roughly 1 percent—combined spending on these categories was approximately $1 billion in the first quarter of fiscal year 2017  Our analysis indicated that variances in spending for these three services ranged widely across our six states, and in four of the states, some of their expenditures were above the thresholds for significance  (See fig  4 ) For example, in regional office 3, Maryland experienced a significant variance in its family planning expenditures—an increase of approximately $8 million dollars or 7,700 percent from the previous quarter—but there was no indication in the documentation provided that the regional office identified or investigated that variance  Similar to the variance analyses for other higher matched expenditure types, we found that the selected regional offices did not consistently conduct variance analyses on expenditures reported for the Medicaid expansion population  First, although five of our six states opted to expand Medicaid under PPACA, two of the five states (Maryland and Pennsylvania) were not subjected to a variance analysis for their expansion populations, a segment that accounted for nearly $7 billion in Medicaid expenditures in fiscal year 2016  Among the remaining three states, CMS regional office staff conducted a variance analysis, but in two of them, the reviewers did not document whether they investigated significant variances, leaving it unclear whether this required step was taken  Specifically, for two of the three remaining states—Arkansas and Nevada—reviewers did not document which variances were deemed significant or that any such variances were discussed with state officials  The guidance specified in CMS’s quarterly review guide is not always clear or consistent  For example: For IHS, family planning, and certain women with breast or cervical cancer, the guidance is explicit that the analysis is required, but the automated variance report used by reviewers for the step does not include these expenditures  For Medicaid expansion expenditures, the review guide is not explicit about whether a variance analysis is required, but CMS has an automated variance report available for these expenditures, which suggests that such an analysis was expected  The guidance suggests that a variance analysis should be conducted for expansion enrollees; however, it does not specify whether the analysis should be conducted in conjunction with—or take the place of—more in-depth examinations  According to federal internal controls standards for information and communication, agencies should communicate the information necessary for staff to achieve the agency’s objectives  CMS’s guidance on conducting variance analyses for types of expenditures with higher federal matching rates has not been sufficiently clear to assure that such analyses are being consistently conducted  By not consistently conducting such checks, errors may be going undetected and CMS may be providing federal funds at a higher matching rate than is allowable  CMS has additional procedures in place to review service expenditures reported for the Medicaid expansion population, a category of expenditures that received a 95 percent federal match in 2017  Specifically, in addition to a variance analysis, CMS guidance specifies that each regional office reviewer is to review claims for a sample  The guide directs the reviewer to obtain a full list of all expansion enrollees from the state and to select 30 to 40 for further review  Next, the reviewer is to obtain supporting documentation from the state listing the eligibility factors for the sampled enrollees, such as age, pregnancy status, Medicare enrollment, and income  The reviewer is to select a single claim for each enrollee and verify that the corresponding expenditures were reported under the correct federal matching rate category—i e , that the sample claim for each individual was accounted for in the relevant section of the CMS-64  The review guide specifies that the sample review be conducted each quarter unless the state has had four consecutive quarters with three or fewer errors, in which case, the sampling must be performed only annually  We found that regional offices were identifying errors in their sampling reviews  For example, region 3 reviewers found that Pennsylvania had incorrectly categorized an individual in the sample as a Medicaid expansion enrollee, with the selected expenditures initially reported as eligible for the higher matching rate  According to CMS central office officials, the sampling methodology has helped identify systemic issues with state expenditure systems in some states and resulted in corrections, adjustments, and in one case, a disallowance  Under current procedures, among our five selected states that expanded Medicaid under PPACA, all five were determined to have had four consecutive clean quarters according to agency officials; that is, the state had three or fewer errors in each quarter  Nationally, all but one of the 33 states that have implemented Medicaid expansion under PPACA had four consecutive clean quarters as of March 2018, according to CMS officials  We found, however, that CMS’s procedures for sampling reviews had a key weakness in that they did not account for varying risks across states, as illustrated in the following examples  We found that sample size does not account for significant differences in program size  For example, both California and Arkansas have expanded Medicaid under PPACA, and regional office staff told us they reviewed claims for 30 expansion enrollees in each of the two states, despite the fact that California has over 10 times as many expansion enrollees as Arkansas  Region 9 officials told us that for California they had initially sampled 100 enrollees during the first quarter they were required to conduct this analysis, but the review was time consuming given staff resources, and they were advised by CMS’s central office to limit their sample to 30 individuals  CMS officials told us that the sampling procedures are resource intensive and that the sample size they decided upon was what they thought they had the resources to complete  Additionally, the sample size does not account for previously identified risks in a state’s program  Specifically, as we noted in a 2015 report, CMS’s sampling review of expansion expenditures was not linked to or informed by reviews of eligibility determinations conducted by CMS, some of which identified high levels of eligibility determination errors  According to CMS officials, the expenditure review is primarily intended to ensure that states are correctly assigning expenditures for the expanded eligibility groups as initially determined, not whether the eligibility determination is correct  Federal standards for internal control related to risk assessment require that agencies identify, analyze, and respond to risks  However, because CMS’s sampling methodology does not account for risk factors like program size and high levels of eligibility determination errors, the agency’s review of expansion population expenditures may be missing opportunities to detect systemic issues with improperly matched expenditures  Quarterly variance analyses and sampling of Medicaid expansion enrollees can be supplemented by financial management reviews  For fiscal year 2016, CMS recommended regional offices conduct FMRs on expenditure claims for expansion enrollees  As of March 2018, however, regional offices had completed an FMR on Medicaid expansion expenditures in only one state, with no findings, and were in the process of completing FMRs for five other states  According to CMS officials, no additional reviews in this area were planned for fiscal year 2018  Financial Impact of Expenditure Reviews Compared with Program Integrity Recoveries The impact of CMS’s expenditure review activities is greater than the impact from other program integrity efforts  For example, in fiscal year 2015, CMS resolved errors through expenditure reviews that saved over $1 4 billion in federal funds  In the same year, CMS reported that efforts by states and the federal government to identify improper payments to providers—for example, services that were billed by a provider but were not received by a beneficiary—resulted in recoveries that totaled $852 9 million, in both state and federal funds  In fiscal years 2014 through 2017, CMS’s regional offices resolved expenditure errors that reduced federal spending by over $5 1 billion, with at least $1 billion in errors resolved in each of three of those four years  Errors were resolved through states agreeing to reduce their reported expenditures, which prevented federal payments to the state for those expenditures; and through CMS issuing disallowances, under which states are required to return federal funds  Although CMS resolved over $1 billion in expenditure errors in each year of fiscal years 2014 through 2016, CMS resolved less than $600 million in fiscal year 2017  CMS officials explained that this change likely reflects delays in clearance of disallowances due to the transition between presidential administrations  (See fig  3 ) In addition to these resolved errors, as of the end of 2017, CMS had $4 47 billion in outstanding deferrals of federal funds, where CMS was delaying federal funds until additional information was provided  Expenditures flagged for deferrals may or may not represent errors  All 10 CMS regional offices resolved errors from fiscal years 2014 through 2017, though the magnitude varied across regions  (See table 5 ) Among the 10 regional offices, 9 reported that they had resolved errors through states agreeing to reduce reported expenditures  Additionally, 9 regional offices issued a total of 49 disallowances across 16 states, with the majority of the disallowances occurring in regional offices 2 and 3  Finally, all 10 regional offices had taken deferrals for questionable expenditures, with 22 states having outstanding “active” deferrals that had not been resolved as of the fourth quarter of fiscal year 2017, which ranged in amount from $178 to $444 million  CMS officials told us that the range of resolved errors and deferred funds across regional offices may reflect differences in the proportion of high-expenditure states  For example, regional office 4 oversees four states ranking in the top 20 in terms of Medicaid expenditures, while regional office 8 does not oversee any top- 20 states  The variation may also reflect large actions taken in specific states  For example, the majority of the disallowed funds in regional office 2 from fiscal years 2014 to 2017 were due to a single disallowance of $1 26 billion in one state  The financial significance of individual errors resolved by CMS’s regional offices varied significantly  We found that regional offices resolved errors that ranged from reporting errors that had no federal financial impact— such as expenditures that were allowable, but were reported on the incorrect line—to hundreds of millions of dollars in expenditures that were found to be unallowable under Medicaid requirements  Over the fiscal years we reviewed, more than half of the disallowances CMS issued were less than $15 million; however, in four states CMS issued disallowances of over $100 million, including a disallowance of over $1 billion in New York  (See fig  5 ) In some cases, actions taken by CMS to resolve errors were the culmination of years of work  For example, over several years the California Medicaid program reported a large volume of expenditures for which it did not yet have sufficient supporting documentation  The regional office officials told us that the state reported these expenditures in order to comply with the 2-year filing limit, and had reported these as “placeholder claims,” with the intention of providing additional support at a later time  Over the course of at least 6 years, CMS deferred hundreds of millions of dollars in federal funds related to these placeholder claims  Of the active deferrals as of the end of fiscal year 2017, most of the total amount of deferred funds was taken for expenditures in California, which represented $3 4 billion of the $4 5 billion in total active deferrals  According to CMS officials, in 2015, CMS prohibited California from reporting additional placeholder claims  Region 9 officials told us that they continue to work with the state to clear the deferrals related to this issue  They were able to resolve 9 related deferrals in fiscal year 2017; however, another over 60 deferrals were still unresolved  The growth of federal Medicaid expenditures, estimated at about $370 billion in fiscal year 2017, makes it critically important to assure expenditures are consistent with Medicaid requirements  CMS has a variety of processes in place to review state-reported expenditures, and those reviews have resulted in CMS resolving errors that have saved the federal government a considerable amount of money; over $5 billion in the last 4 years  However, the increasing complexity of expenditure reporting is occurring as resources to review these expenditures are decreasing, hindering CMS’s ability to target risk and potentially allowing for hundreds of millions of federal dollars in errors to go undetected  In the absence of a comprehensive risk assessment, which CMS has not conducted, CMS may be missing opportunities to better target resources to higher risk expenditures and increase the savings from these oversight activities  The variety of different matching rates has contributed to the increased complexity of CMS’s expenditure reviews  Although CMS has review procedures in place to assure that the correct matching rate is applied for services and populations receiving a higher federal matching rate, unclear guidance has contributed to inconsistency in the extent to which these reviews are conducted  In addition, we found weaknesses in the sampling methodology CMS requires its regional offices to use to help ensure that expenditures for Medicaid expansion enrollees—expenditures that receive a higher matching rate and that represented almost 20 percent of total federal Medicaid spending in 2016—are consistent with Medicaid requirements  In particular, the methodology does not account for risk factors like program size or vulnerabilities in state eligibility-determination processes and systems  As a result of the inconsistency in reviews and a sampling methodology that does not consider program risk, errors may be going undetected, resulting in CMS providing federal funds at higher federal matching rates than is allowable  In addition, CMS could be missing opportunities to identify any systemic issues that may contribute to such errors  We are making the following three recommendations to CMS: 1  The Administrator of CMS should complete a comprehensive, national risk assessment and take steps, as needed, to assure that resources to oversee expenditures reported by states are adequate and allocated based on areas of highest risk  (Recommendation 1) 2  The Administrator of CMS should clarify in internal guidance when a variance analysis on expenditures with higher match rates is required  (Recommendation 2) 3  The Administrator of CMS should revise the sampling methodology for reviewing expenditures for the Medicaid expansion population to better target reviews to areas of high risk  (Recommendation 3) We provided a draft of this report to HHS for review and comment  HHS concurred with all three recommendations, noting that it takes seriously its responsibilities to protect taxpayer funds by conducting thorough oversight of states’ claims for federal Medicaid expenditures  Regarding our first recommendation—that CMS complete a comprehensive, national risk assessment and take steps to assure that resources are adequate and allocated based on risk—HHS noted that CMS will complete such an assessment, and, based on this review, will determine the appropriate allocation of resources based on expenditures, program risk, and historical financial issues  CMS will also identify opportunities to increase resources  Regarding our second recommendation—clarifying internal guidance on when a variance analysis on higher matched expenditures is required—HHS noted that CMS will issue such internal guidance  Regarding our third recommendation—that CMS revise the sampling methodology for reviewing expenditures for the Medicaid expansion population to better target reviews to areas of high risk—HHS noted CMS is considering ways to revise its methodology  HHS’s comments are reproduced in appendix II  As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies of this report to the Secretary of Health and Human Services, appropriate congressional committees, and other interested parties  The correspondence is also available at no charge on the GAO website at http://www gao gov  If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or yocomc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix III  Public psychiatric residential treatment facilities Medicare Part B premium buy-ins Outpatient hospital reimbursement for mental health services Review of comprehensive psychiatric emergency program rates Provider taxes implemented to avoid program reductions Health homes data and expenditures reporting Provider incentive payments for health information technology 1115 demonstration provider incentive payments Public psychiatric residential treatment facilities Managed care organizations’ provider payments Provider incentive payments for health information technology Federally qualified health center reimbursement payments Eligibility and enrollment maintenance and operations Managed care organizations’ reporting of drug rebates 3 CMS cancelled these 2014 FMRs due to a staffing shortage  Region 8 was excused from the requirement to conduct an FMR in 2016 due to staffing constraints  In addition to the contact named above, Susan Barnidge (Assistant Director), Jasleen Modi (Analyst-in-Charge), Caroline Hale, Perry Parsons, and Sierra Gaffney made key contributions to this report  Also contributing were Giselle Hicks, Drew Long, and Jennifer Whitworth 


Report 2:  At a minimum, this effort should involve the agencies that our survey suggests may offer such insights  (Recommendation 2) We provided a draft of this report to the Director of the Office of Management and Budget for review and comment  We also provided a draft of the report to the heads of each of the 24 federal agencies covered by our survey  OMB had no comments, and informed us that it would assess our recommendations and consider how best to respond  We are sending copies of this report to congressional requesters, the Director of the Office of Management and Budget, the heads of each of the 24 agencies, and other interested parties  This report will also be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mcneilt@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report  Key contributors to this report are listed in appendix IV  (USDA) (Goverment-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Commerce) (Goverment-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Goverment-wide) (DOD) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Education) (Goverment-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (Energy) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (HHS) (Goverment-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (DHS) (Goverment-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (HUD) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Interior) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (DOJ) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (DOL) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (State) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (DOT) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Treasury) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (VA) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Goverment-wide) (USAID) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (EPA) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (GSA) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (NASA) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (NSF) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (NRC) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (OPM) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (SBA) (Government-wide) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” (Government-wide) (SSA) 10  The individual I report to 11  Employees that report to me 0 Percent of managers reporting “Great” or “Very Great” This report responds to a request that we analyze agency-level results from our 2017 survey of federal managers at the 24 agencies covered by the Chief Financial Officers (CFO) Act of 1990, as amended, to determine the extent agencies are using performance information  This report assesses the extent to which: 1  the reported use of performance information and related leading practices at 24 agencies has changed compared to our prior survey in 2013; 2  being subject to data-driven reviews related to managers’ reported use of performance information and leading practices; and 3  the Executive Branch has taken actions to enhance agencies’ use of performance information in various decision-making activities  From November 2016 through March 2017, we administered our online survey to a stratified random sample of 4,395 individuals from a population of 153,779 mid- and upper-level civilian managers and supervisors at the 24 CFO Act agencies  The management levels covered general schedule (GS) or equivalent schedules at levels comparable to GS-13 through GS-15, and career Senior Executive Service (SES) or equivalent  We obtained the sample from the Office of Personnel Management’s Enterprise Human Resources Integration database as of September 30, 2015—the most recent fiscal year data available at the time  The sample was stratified by agency and whether the manager or supervisor was a member of the SES  To help determine the reliability and accuracy of the database elements used to draw our sample of federal managers for the 2017 survey, we checked the data for reasonableness and the presence of any obvious or potential errors in accuracy and completeness and reviewed our past analyses of the reliability of this database  We concluded in our September 2017 report that the data used to draw our sample were sufficiently reliable for the purpose of the survey  For the 2017 survey, we received usable questionnaires from about 67 percent of the eligible sample  The weighted response rate at each agency generally ranged from 57 percent to 82 percent, except the Department of Justice, which had a weighted response rate of 36 percent  The overall survey results are generalizable to the population of managers government-wide and at each individual agency  To assess the potential bias from agencies with lower response rates, we conducted a nonresponse bias analysis using information from the survey and sampling frame as available  The analysis confirmed discrepancies in the tendency to respond to the survey related to agency and SES status  The analysis also revealed some differences in response propensity by age and GS level; however, the direction and magnitude of the differences on these factors were not consistent across agencies or strata  Our data may be subject to bias from unmeasured sources for which we cannot control  Results, and in particular estimates from agencies with low response rates such as the Department of Justice, should be interpreted with caution  However, the survey’s results are comparable to five previous surveys we conducted in 1997, 2000, 2003, 2007, and 2013  To address the first objective, we used data from our 2017 survey to update agency scores on our use of performance information index  This index, which was last updated using data from our 2013 survey, averages managers’ responses on 11 questions related to the use of performance information for various management activities and decision making  Using 2017 survey data, we conducted statistical analyses to ensure these 11 questions were still positively correlated  That analysis confirmed that no negative correlations existed and therefore no changes to the index were needed  Figure 11 shows the questions that comprise the index  After calculating agency index scores for 2017, we compared them to previous results from 2007 and 2013, and to the government-wide average for 2017, to identify any statistically significant differences  We focus on statistically significant results because these indicate that observed relationships between variables and differences between groups are likely to be valid, after accounting for the effects of sampling and other sources of survey error  For each of the 11 questions that comprise the index, we identified individual agency results, excluding missing and no basis to judge responses, and determined when they were statistically significantly different from (1) the agency’s results on the same question in 2013, or (2) the government-wide average results on the question in 2017  In this report, we analyzed and summarized the results of our 2017 survey of federal managers  Due to the limited scope of the engagement, we did not conduct additional audit work to determine what may have caused statistically significant changes between our 2017 and past survey results  To further address this objective we completed several statistical analyses that allowed us to assess the association between the index and 22 survey questions that we determined relate to leading practices we previously found promote the use of performance information  See figure 12 for the 22 specific questions related to these five practices that we included in the analysis  When we individually tested these 22 survey questions (bivariate regression), we found that each was statistically significantly and positively related to the index in 2017  This means that each question, when tested in isolation from other factors, was associated with higher scores on the index  However, when all 22 questions were tested together (multivariate regression), we found that 5 questions continued to be positively and significantly associated with the index in 2017, after controlling for other factors  To conduct this multivariate analysis, we began with a base model that treated differences in managers’ views of agency performance management use as a function of the agency where they worked  We found, however, that a model based on agency alone had little predictive power (R-squared of 0 04)  We next examined whether managers’ responses to these questions reflecting practices that promote the use of performance information related to their perceptions of agency use of performance information, independent of agency  The results of this analysis are presented in table 1 below  Each coefficient reflects the increase in our index associated with a one-unit increase in the value of a particular survey question  Our final multivariate regression model had an R-squared of 0 67, suggesting that the variables in this model explain approximately 67 percent of the variation in the use index  We also tested this model controlling for whether a respondent was a member of the SES and found similar results  As shown above in table 1, five questions related to three of the leading practices that promote agencies’ use of performance information were statistically significant in 2017  These results suggest that, when controlling for other factors, certain specific efforts to increase agency use of performance information—such as providing information on the validity of performance data—may have a higher return and lead to higher index scores  With respect to aligning agency-wide goals, objectives, and measures, we found that each increase in terms of the extent to which individuals felt that managers aligned performance measures with agencywide goals and objectives was associated with a 0 08 increase in their score on the use index  In terms of improving the usefulness of performance information, we found that having information on the validity of performance data for decision making was the strongest predictor in our model (0 18)  As measured here, taking steps to ensure the performance information is useful and appropriate was associated with almost as large a change in a managers’ index score (0 16)  In terms of developing agency capacity to use performance information, we found that having sufficient analytical tools to collect, analyze, and use performance information (0 07), and providing or paying for training that would help link their programs to achievement of agency strategic goals (0 10), were also statistically significantly related to a manager’s reported use of performance information  When we combined these results with what we previously found through a similar analysis of 2013 survey results in September 2014, we identified 10 questions that have had a statistically significant association with higher index scores  This reinforces the importance of the five leading practices to promote the use of performance information  For each of these questions, which are outlined in figure 13 below, we determined when agency results were statistically significantly different from 2013 results or the 2017 government-wide average  For the second objective, we examined, based on the extent they responded their programs had been subject to agency data-driven reviews, differences in managers’ use index scores and responses on questions related to practices that promote the use of performance information  We grouped managers based on the extent they reported their programs had been subject to these reviews, from “no extent” through “very great extent ” We then calculated the average index scores for the managers in each of those five categories  We also examined differences in how managers responded to the 10 questions reflecting practices that can promote the use of performance information based on the extent they reported their programs had been subject to data-driven reviews  We grouped managers into three categories based on the extent they reported their programs had been subject to these reviews (no-small extent, moderate extent, great-very great extent)  We then compared how these groups responded to the ten questions  For the third objective, we reviewed our past work that assessed Executive Branch activities to enhance the use of performance information; various resources (i e , guidance, guides, and playbooks) developed by the Office of Management and Budget (OMB) and the Performance Improvement Council (PIC) that could support agencies’ use of performance information; and the President’s Management Agenda, and related materials with information on cross-agency efforts to improve the use of data in federal decision making  Lastly, for the third objective we also interviewed OMB and PIC staff about any actions they have taken, or planned to take, to further support the use of performance information across the federal government  We conducted this performance audit from October 2017 to September 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  In addition to the above contact, Benjamin T  Licht (Assistant Director) and Adam Miles (Analyst-in-Charge) supervised this review and the development of the resulting report  Arpita Chattopadhyay, Caitlin Cusati, Meredith Moles, Dae Park, Amanda Prichard, Steven Putansu, Alan Rozzi, Shane Spencer, and Khristi Wilkins also made key contributions  Robert Robinson developed the graphics for this report  Alexandra Edwards, Jeff DeMarco, Mark Kehoe, Ulyana Panchishin, and Daniel Webb verified the information presented in this report  Results of the Periodic Surveys on Organizational Performance and Management Issues Managing for Results: Further Progress Made in Implementing the GPRA Modernization Act, but Additional Actions Needed to Address Pressing Governance Challenges  GAO-17-775  Washington, D C : September 29, 2017  Supplemental Material for GAO-17-775: 2017 Survey of Federal Managers on Organizational Performance and Management Issues  GAO-17-776SP  Washington, D C : September 29, 2017  Program Evaluation: Annual Agency-wide Plans Could Enhance Leadership Support for Program Evaluations  GAO-17-743  Washington, D C : September 29, 2017  Managing for Results: Agencies’ Trends in the Use of Performance Information to Make Decisions  GAO-14-747  Washington, D C : September 26, 2014  Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges  GAO-13-518  Washington, D C : June 26, 2013  Managing for Results: 2013 Federal Managers Survey on Organizational Performance and Management Issues, an E-supplement to GAO-13-518  GAO-13-519SP  Washington, D C : June 26, 2013  Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making  GAO-13-570  Washington, D C : June 26, 2013  Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results  GAO-08-1026T  Washington, D C : July 24, 2008  Government Performance: 2007 Federal Managers Survey on Performance and Management Issues, an E-supplement to GAO-08-1026T  GAO-08-1036SP  Washington, D C : July 24, 2008  Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results  GAO-04-38  Washington, D C : March 10, 2004  Managing for Results: Federal Managers’ Views on Key Management Issues Vary Widely Across Agencies  GAO-01-592  Washington, D C : May 25, 2001  Managing for Results: Federal Managers’ Views Show Need for Ensuring Top Leadership Skills GAO-01-127  Washington, D C : October 20, 2000  The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven  GAO/GGD-97-109  Washington, D C : June 2, 1997 

ground_truth: To reform the federal government and make it more efficient and effective, agencies need to use data about program performance. The benefit of collecting performance information is only fully realized when it is used by managers to make decisions aimed at improving results. GAO was asked to review agencies' use of performance information. This report assesses, among other things, the extent to which: (1) 24 agencies' reported use of performance information and related leading practices has changed since 2013 and (2) the Executive Branch has taken actions to enhance the use of performance information. To address the first objective, GAO analyzed results from its 2017 survey of federal managers, and compared them to 2013 results. The survey covered a stratified random sample of 4,395 managers from the 24 Chief Financial Officers Act agencies. The survey had a 67 percent response rate and results can be generalized to the population of managers government-wide and at each agency. For the second objective, GAO reviewed agency documents and interviewed staff from OMB and the PIC. Agencies' reported use of performance information to make decisions, and leading practices that can promote such use, generally has not improved since GAO's last survey of federal managers in 2013. However, GAO's survey results continue to point to certain practices that could help agencies improve managers' use of performance information. For example, as shown in the table below, GAO's survey found that managers whose programs were subject to data-driven reviews (regular reviews used to assess progress on select agency goals) to a greater extent reported statistically significantly greater use of performance information to make decisions. The Executive Branch has begun taking steps to improve the use of performance information within agencies and across the government. For example, In the President's Management Agenda and government-wide reform plan, released in March and June 2018 respectively, the administration acknowledged the need to do more, and announced a goal, among other actions, to improve the use of data in federal decision making. However, the Office of Management and Budget (OMB) and others responsible for this goal have yet to fully develop action plans to hold agencies accountable for achieving it. The Performance Improvement Council (PIC), which is chaired by OMB, has undertaken efforts to improve the use of performance information by, for example, creating a working group on agency performance reviews. But it has not yet taken a systematic approach to identify and share proven practices that led to, or challenges that may be hampering, increased use of performance information by managers. GAO's survey results identified agencies that may have insights into such practices and challenges. More fully developing action plans for the new goal, and identifying and sharing proven practices and challenges, could help ensure the Executive Branch takes further steps to improve the use of performance information by managers within agencies and across the federal government. To improve the use of performance information within agencies and across the federal government, GAO recommends that OMB work with (1) fellow goal leaders to more fully develop action plans for the new goal to improve the use of data and (2) the PIC to prioritize efforts to identify and share proven practices and challenges. OMB had no comments on this report.