prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.

Report 1: GPRAMA significantly enhances GPRA, the centerpiece of a statutory framework that Congress put in place during the 1990s to help resolve longstanding performance and management problems in the federal government and provide greater accountability for results  Congress passed GPRAMA in 2010 to address a number of persistent federal performance challenges, including focusing attention on crosscutting issues and enhancing the use and usefulness of performance information  OMB and agencies are to establish various government-wide and agency-specific performance goals, in line with GPRAMA requirements or OMB guidance  These include the following: Cross-agency priority (CAP) goals: CAP goals are crosscutting and include outcome-oriented goals covering a limited number of policy areas as well as goals for management improvements needed across the government  OMB is to coordinate with agencies to establish CAP goals at least every 4 years  OMB is also required to coordinate with agencies to develop annual federal government performance plans to, among other things, define the level of performance to be achieved toward the CAP goals  Strategic objectives: A strategic objective is the outcome or impact the agency is intending to achieve through its various programs and initiatives  Agencies establish strategic objectives in their strategic plans and may update the objectives during the annual update of performance plans  Agency priority goals (APG): At the agency level, every 2 years, GPRAMA requires that the heads of certain agencies, in consultation with OMB, identify a subset of agency performance goals as APGs  These goals are to reflect the agencies’ highest priorities  They should be informed by the CAP goals as well as consultations with relevant congressional committees and other interested parties  In a schedule established by GPRAMA, OMB and agencies are to develop and publish new CAP goals, APGs, and strategic plans (with updated strategic objectives) in February 2018  GPRAMA and related OMB guidance require agencies to regularly assess their progress in achieving goals and objectives through performance reviews  Data-driven reviews: Agency leaders and managers are to use regular meetings, at least quarterly, to review data and drive progress toward key performance goals and other management-improvement priorities  For each APG, GPRAMA requires agency leaders to conduct reviews at least quarterly to assess progress toward the goal, determine the risk of the goal not being met, and develop strategies to improve performance  Similarly, the Director of OMB, with relevant parties, is to review progress toward each CAP goal  Strategic reviews: OMB guidance directs agency leaders to annually assess progress toward achieving each strategic objective using a broad range of evidence  GPRAMA establishes certain senior leadership positions and a council, as described below  Chief Operating Officer (COO): The deputy agency head, or equivalent, is designated COO, with overall responsibility for improving agency management and performance  Performance Improvement Officer (PIO): Agency heads are to designate a senior executive within the agency as the PIO  The PIO reports directly to the COO and assists the agency head and COO with various performance management activities  Goal leaders: Goal leaders are responsible for developing strategies to achieve goals, managing execution, and regularly reviewing performance  GPRAMA requires goal leaders for CAP goals and agency performance goals, including APGs  OMB guidance directs agencies to designate goal leaders for strategic objectives  Performance Improvement Council (PIC): The PIC is charged with assisting OMB to improve the performance of the federal government and achieve the CAP goals  The PIC is chaired by the Deputy Director for Management at OMB and includes agency PIOs from each of the 24 CFO Act agencies as well as other PIOs and individuals designated by the chair  Among its responsibilities, the PIC is to work to resolve government-wide or crosscutting performance issues, and facilitate the exchange among agencies of practices that have led to performance improvements within specific programs, agencies, or across agencies  GPRAMA includes several provisions related to providing the public and Congress with information, as described below  Performance gov: GPRAMA calls for a single, government-wide performance website to communicate government-wide and agency performance information  Among other things, the website— implemented by OMB as Performance gov—is to include (1) quarterly progress updates on CAP goals and APGs; (2) an inventory of all federal programs; and (3) agency strategic plans, annual performance plans, and annual performance reports  Reporting burden: GPRAMA establishes a process to reexamine the usefulness of certain existing congressional reporting requirements  Specifically, GPRAMA requires an annual review (including congressional consultation), based on OMB guidance, of agencies’ reporting requirements to Congress  Additionally, OMB is to include in the budget a list of plans and reports determined to be outdated or duplicative and may submit legislation to eliminate or consolidate such plans or reports  In early 2017, the administration announced several efforts that are intended to improve government performance  The 2018 Budget Blueprint states that the President’s Management Agenda will seek to improve the federal government’s effectiveness by using evidence-based approaches, balancing flexibility with accountability to better achieve results, improving mission support functions, and developing and monitoring critical performance measures  In addition, OMB issued several memoranda detailing the administration’s plans to improve government performance by reorganizing the government, reducing the federal workforce, and reducing federal agency burden  A number of these efforts, which are to leverage GPRAMA and our past work, have the potential to further progress in addressing key governance challenges  As part of reorganization efforts, OMB and agencies are developing government-wide and agency reform plans, respectively, that are to leverage various GPRAMA provisions  For example, an April 2017 memorandum states that OMB intends to monitor implementation of the reform plans using CAP goals, APGs, annual strategic reviews, and Performance gov  The government-wide plan also is to include crosscutting reform proposals, such as merging agencies or programs that have similar missions  To that end, the memorandum states agencies should consider our reports, including our work on fragmentation, overlap, and duplication, as well as inspectors general reports  Many of the meaningful results that the federal government seeks to achieve, such as those related to ensuring public health, providing homeland security, and promoting economic development, require the coordinated efforts of more than one federal agency, level of government, or sector  For more than 2 decades, we have reported on agencies’ missed opportunities for improved collaboration through the effective implementation of GPRA and, more recently, GPRAMA  Our reports also have demonstrated that collaboration across agencies is critical to address issues of fragmentation, overlap, and duplication as well as many of the areas on our High-Risk List  Fragmentation, Overlap, and Duplication: Since 2011, our annual reports have identified 133 crosscutting areas that require the coordinated effort of more than one federal organization, level of government, or sector  For instance, for the area of federal grant awards, we found in January 2017 that the National Park Service (NPS), Fish and Wildlife Service, Food and Nutrition Service, and Centers for Disease Control and Prevention (CDC) had not established guidance and formal processes to ensure their grant-management staff review applications for potential duplication and overlap among grants in their agencies before awarding  We recommended that these agencies do so, and they agreed  As of August 2017, these agencies had taken several actions to address the recommendation  For example, the Department of the Interior (Interior) provided documentation showing that the Fish and Wildlife Service now requires discretionary grant applicants to provide a statement that addresses whether there is any overlap or duplication of proposed projects or activities to be funded by the grant  Fish and Wildlife also updated its guidance to grant awarding offices instructing them to perform a potential overlap and duplication review of all selected applicants prior to award  Our Action Tracker provides details on the status of actions from our annual reports  Within the 133 crosscutting areas, since 2011 we have identified 315 targeted actions where opportunities exist to better manage fragmentation, overlap, and duplication, including 29 new actions in our most recent report issued in April 2017  We found that the executive branch and Congress addressed 145 (46 percent) of the 315 actions  For example, in November 2014, we recommended that the U S  Coast Guard and Consumer Product Safety Commission establish a formal approach to coordination (such as a memorandum of understanding) to facilitate information sharing; better leverage their resources; and address challenges, including those related to fragmentation and overlap that we identified  In response to this recommendation, the two agencies signed a formal policy document to govern their coordination in May 2015  This policy document outlined procedures for determining jurisdictional authority for recreational boat-associated equipment and marine safety items  Specifically, the procedures clarified that upon receiving notice of a possible defect, the agency receiving such notice shall determine whether the item properly falls within its jurisdiction, and if not, initiate discussions to determine the appropriate jurisdiction  These new procedures should help the agencies share information and leverage each other’s resources so they can better ensure that recreational boat-associated equipment and marine safety items are fully regulated  However, more work is needed on the remaining 170 actions (54 percent) that have not been fully addressed  For example, in July 2016, we reported that four federal agencies—the Departments of Defense, Education, Health and Human Services, and Justice—manage at least 10 efforts to collect data on sexual violence, which differ in target population, terminology, measurements, and methodology  We found that data collection efforts use 23 different terms to describe sexual violence  Data collection efforts also differed in how they categorized particular acts of sexual violence, the context in which data were collected, data sources, units of measurement, and time frames  We recommended that OMB convene an interagency forum to better manage fragmentation of efforts to collect sexual violence data  In commenting on that report, OMB stated it would consider implementing the action in the future but did not believe it was the most effective use of resources at that time, in part because the agencies were not far enough along in their research  In response, we stated that given the number of federal data collection efforts, the range of differences across them, and the potential for causing confusion, it would be beneficial for agencies to discuss these differences and determine whether they are, in fact, necessary  As of July 2017, OMB had not provided an update on the status of this recommendation  High-Risk List: Since the early 1990s, our high-risk program has focused attention on government operations with greater vulnerabilities to fraud, waste, abuse, and mismanagement or that are in need of transformation to address economy, efficiency, or effectiveness challenges  As of February 2017, there were 34 high-risk areas covering a wide range of issues including human capital management, modernizing the U S  financial regulatory system, and ensuring the security of federal information systems and cyber critical infrastructure  Many of these high- risk areas require a coordinated response from more than one branch of government, agency, or sector  In the time between our 2015 and 2017 High-Risk Updates, many of these high-risk areas on our list demonstrated solid progress  During that period, 15 high-risk areas fully met at least one of the five criteria required for removal from the High-Risk List  In many cases, progress was possible through the joint efforts of Congress and leadership and staff in agencies  For example, Congress passed over a dozen laws following our 2015 High-Risk Update to help address high-risk issues  In addition, in 2017, we removed one high-risk area on managing terrorism-related information, because significant progress had been made to strengthen how intelligence on terrorism, homeland security, and law enforcement is shared among federal, state, local, tribal, international, and private sector partners  Despite this progress, continued oversight and attention is also warranted given the issue’s direct relevance to homeland security as well as the constant evolution of terrorist threats and changing technology  Our February 2017 High-Risk Update also highlighted a number of long- standing high-risk areas that require additional attention  We also added three new crosscutting areas to incorporate the management of federal programs that serve tribes and their members, the government’s environmental liabilities, and the 2020 decennial census  Based on our body of work on federal programs that serve tribes and their members, we concluded that federal agencies had (1) ineffectively administered Indian education and health care programs and (2) inefficiently fulfilled their responsibilities for managing the development of Indian energy resources  For example, we identified numerous challenges facing Interior’s Bureau of Indian Education (BIE) and Bureau of Indian Affairs, and the Department of Health and Human Services’ (HHS) Indian Health Service (IHS), in administering education and health care services  We concluded that these challenges put the health and safety of American Indians served by these programs at risk  In May 2017, we issued two additional reports on accountability for school construction and safety at schools funded by BIE  Although these agencies have taken some actions to address recommendations we made related to Indian programs, about 50 recommendations have yet to be fully resolved  We are monitoring federal efforts to address the unresolved recommendations  We also are reviewing IHS’s workforce, and tribal nations’ management and use of their energy resources  Many of the crosscutting areas highlighted by our annual reports on fragmentation, overlap, and duplication and designated as high-risk would benefit from enhanced collaboration among the federal agencies involved in them  GPRAMA establishes a framework aimed at taking a more crosscutting and integrated approach to focusing on results and improving government performance  Our survey results and past work demonstrate that agencies continue to face difficulties when working together on crosscutting issues, but also that implementing certain GPRAMA requirements can have a positive effect on collaboration  An item related to coordination in our survey of federal managers is statistically significantly lower in 2017, relative to our previous survey in 2013 and our initial survey in 1997  In 2017, an estimated 43 percent of managers agreed that they use information obtained from performance measurement to a great or very great extent when coordinating program efforts with internal or external organizations (compared to an estimated 50 percent in 2013 and an estimated 57 percent in 1997)  Moreover, our past work has found that agencies face a variety of challenges when working across organizational boundaries to deliver programs and improve performance  For example, our work has found that interagency groups have, at times, encountered difficulty clarifying roles and responsibilities or developing shared outcomes and performance measures  In contrast, our past work demonstrates that implementing GPRAMA provisions can improve collaboration  For example, in May 2016, we found that OMB and the PIC updated the governance structure for CAP goals to include both agency-level and Executive Office of the President goal leaders and held regular, senior-level reviews on CAP goal progress  Moreover, CAP goal teams told us that the CAP goal designation increased leadership attention and improved interagency collaboration on their crosscutting issues  Furthermore, our prior work has found that priority goals and related data-driven reviews have also been used to help manage crosscutting issues and enhance collaboration  Various GPRAMA requirements are aimed at improving agencies’ coordination of efforts to address crosscutting issues  As with our 2013 survey, our 2017 survey continues to show that CAP goals, APGs, and related data-driven reviews—also called quarterly performance reviews (QPR)—are associated with reported higher levels of collaboration with internal and external stakeholders  For example, our 2017 survey data indicate that about half of federal managers (an estimated 54 percent) reported they were somewhat or very familiar with CAP goals  Among these individuals, those who viewed their programs as contributing to CAP goals to a great or very great extent (36 percent) were more likely to report collaborating outside their program to a great or very great extent to help achieve CAP goals (62 percent), as shown in figure 2  Our analysis shows a similar pattern exists for APGs and QPRs  Our past work also has highlighted ways in which OMB and agencies could better implement GPRAMA’s crosscutting provisions—many of which have been addressed  A continued focus on fully and effectively implementing these provisions will be important as OMB and agencies establish new CAP goals and APGs, and assess progress toward them through related QPRs  Cross-agency priority (CAP) goals: In May 2012 and June 2013, we found that OMB had not always identified relevant agencies and program activities as contributors to the initial set of CAP goals  OMB took actions in response to our recommendations to include relevant contributors  Our most recent review, in May 2016, found that all relevant contributors had been identified for a subsequent set of CAP goals  In that report, we also found that OMB and the PIC had improved implementation of the CAP goals, in part, by helping agencies build their capacity to contribute to implementing the goals  Appendix II summarizes our past recommendations related to GPRAMA and the actions agencies have taken to address them  Agency priority goals (APGs): In April 2013, we found that agencies did not fully explain the relationship between their APGs and crosscutting efforts  Identify contributors: Similar to OMB’s responsibilities with the CAP goals, agencies are to identify the various organizations and programs that contribute to each of their performance goals, including APGs  We found that agencies identified internal contributors for their APGs, but did not list external contributors in some cases  We recommended that the Director of OMB ensure that agencies adhere to OMB’s guidance for website updates by providing complete information about the organizations, program activities, regulations, tax expenditures, policies, and other activities—both within and external to the agency— that contribute to each APG  In response, in April 2015, OMB asked agencies to identify organizations, program activities, regulations, policies, tax expenditures, and other activities contributing to their 2014-2015 APGs  Based on an analysis of the final quarterly updates for those APGs, published in December 2015, we found that agencies made progress in identifying external organizations and programs for their APGs  Describe how agency goals contribute to CAP goals: Agencies generally did not identify how their APGs contributed to CAP goals  We recommended that OMB direct agencies to describe in their performance plans how the agency’s performance goals—including APGs—contribute to any of the CAP goals as required by GPRAMA  In response, in July 2013, OMB updated its guidance directing agencies to include a list of the CAP goals to which the agency contributes and explain the agency’s contribution to them in their strategic plans, performance plans, and performance reports  Data-driven reviews: For their data-driven reviews of agency priority goals, agencies are to include, as appropriate, relevant personnel within and outside the agency who contribute to the accomplishment of each goal  However, in February 2013, we found that most Performance Improvement Officers (PIO) we surveyed (16 of 24) indicated that there was little to no involvement in these reviews from external officials who contribute to achieving agency goals  We recommended that OMB and the PIC help agencies extend their QPRs to include, as relevant, representatives from outside organizations that contribute to achieving their APGs  OMB staff told us that they generally concurred with the recommendation, but believed it would not always be appropriate to regularly include external representatives in agencies’ data-driven reviews, which they considered to be internal management meetings  In a subsequent review, we found in July 2015 that PIOs at 21 of the 22 agencies we surveyed said that their data-driven reviews had a positive effect on collaboration among officials from different offices or programs within the agency  Despite the positive effects, most agency PIOs (17) indicated that there continued to be little to no involvement in the reviews from external officials who contribute to achieving agency goals  In May 2016, OMB and PIC staff reported that, in response to our earlier recommendation, they were working with agencies to identify examples where agencies included representatives from outside organizations in data-driven reviews, and to identify promising practices based on those experiences  PIC staff told us they would disseminate any promising practices identified through the PIC Internal Reviews Working Group and other venues  In August 2017, OMB staff told us they plan to hold a summit with agencies later in the year to discuss implementing various performance management requirements, which could include agencies highlighting experiences and promising practices related to involving external officials in their data-driven reviews  We continue to believe data- driven reviews should include any relevant contributors from outside organizations and will continue to monitor progress  Despite the important role priority goals and related reviews can play in addressing crosscutting issues and enhancing collaboration, OMB recently removed the priority status of the current sets of priority goals  According to OMB staff, removing the priority designation from CAP goals and APGs returned them to regular performance goals, which are not subject to quarterly data-driven reviews or updates on the results of those reviews on Performance gov  In a June 2017 memorandum, OMB stated that CAP goals and APGs are intended to focus efforts toward achieving the priorities of current political leadership, and therefore reporting on the priority goals of the previous administration on Performance gov was discontinued for the remainder of the period covered by the goals (through September 30, 2017, the end of fiscal year 2017)  The memorandum further noted that agencies and teams working on those goals should continue working on the current goals where they align with the priorities of the current administration  Moreover, the memorandum states that agencies have flexibility in structuring their data-driven reviews, but they should continue such reviews focused on agency priorities  When asked about these actions, OMB staff told us that they believed they were working in line with the intentions of GPRAMA, which realigned the timing of goal setting with presidential terms, to better take into account changes in priorities  This is the first presidential transition since GPRAMA was enacted, and OMB staff told us they thought the act was unclear on how to handle priority goals during the changes in administrations and priorities  They stated that it was not practical to continue reporting on the priority goals of the prior administration as agencies worked to develop new strategic plans and priority goals for publication in February 2018  Hence, they told us OMB ended the current round of CAP goals and directed agencies to remove the priority designation from the APGs, returning them to regular performance goals  OMB staff further told us that although the guidance was published in a June 2017 memorandum, these decisions had been made and previously communicated to agencies during the transition in administrations  Therefore, reporting on the fiscal year 2014-2017 CAP goals, fiscal year 2016-2017 APGs, and related reviews stopped much earlier in the year, well before goal cycles were planned to be completed on September 30, 2017  OMB staff further stated that although the goals no longer had priority designations, work towards them largely continued in 2017  For example, one of the prior administration’s CAP goals was to modernize the federal permitting and review process for major infrastructure projects  OMB staff told us that they and agencies have continued many of the activities intended to achieve that goal, but they are no longer subject to quarterly data-driven reviews or updates on the results of these reviews on Performance gov  Moreover, they expect most of this work will continue towards a new and refocused CAP goal on infrastructure permitting modernization  OMB staff reaffirmed to us their intentions to resume implementation of CAP goals, APGs, and related data-driven reviews when the new planning and reporting cycle begins in February 2018  This is in line with stated plans to leverage various GPRAMA provisions to track progress of proposed government-wide and agency-specific reforms, as outlined in OMB’s April 2017 memorandum on the reform plans  In addition, OMB’s July 2017 update to its guidance for implementing GPRAMA similarly focuses on continued implementation of the act  Additional aspects of GPRAMA implementation could similarly help improve the management of crosscutting issues  Strategic reviews: OMB’s 2012 guidance implementing GPRAMA established a process in which agencies, beginning in 2014, were to conduct leadership-driven, annual reviews of their progress toward achieving each strategic objective established in their strategic plans  As we found in July 2015, effectively implementing strategic reviews could help identify opportunities to reduce, eliminate, or better manage instances of fragmentation, overlap, and duplication  Under OMB’s guidance, agencies are to identify the various organizations, program activities, regulations, tax expenditures, policies, and other activities that contribute to each objective, both within and outside the agency  Where progress in achieving an objective is lagging, the reviews are intended to identify strategies for improvement, such as strengthening collaboration to better address crosscutting challenges, or using evidence to identify and implement more effective program designs  If successfully implemented in a way that is open, inclusive, and transparent—to Congress, delivery partners, and a full range of stakeholders—this approach could help decision makers assess the relative contributions of various programs to a given objective  Successful strategic reviews could also help decision makers identify and assess the interplay of public policy tools that are being used to ensure that those tools are effective and mutually reinforcing, and that results are being efficiently achieved  In July 2017, OMB released guidance which updated the status of the 2017 strategic reviews  Because agencies are currently developing new strategic goals and objectives, OMB stated that agencies may forego the reporting and categorization requirements for any current strategic objectives that an agency determines will be substantively different or no longer aligned with the current administration’s policy, legislative, regulatory, or budgetary priorities  In addition, OMB stated that while there will be no formal meetings between OMB and the agencies to discuss findings and related progress from the 2017 strategic reviews, it expects that agencies will continue to conduct strategic reviews or assess progress made toward strategic goals and objectives aligned with administration policy  Furthermore, OMB stated that during this transition year, updates of progress on agency strategic objectives will only be published in the agency’s annual performance report and will not be reported to Performance gov  Full reporting through Performance gov is to resume after new agency strategic plans are published in February 2018  Agencies are to include a progress update for strategic objectives as part of their progress update in their fiscal year 2017 annual performance reports  Agencies also must address next steps for performance improvement as part of their fiscal year 2019 annual performance plans  Program inventories: GPRAMA requires OMB to publish a list of all federal programs, along with related budget and performance information, on a central government-wide website  Such a list could help decision makers and the public fully understand what the federal government does, how it does it, and how well it is doing  An inventory of federal programs could also be a critical tool to help decision makers better identify and manage fragmentation, overlap, and duplication across the federal government  Agencies developed initial program inventories in May 2013, but since then have not updated or more fully implemented these inventories  In October 2014, we found several issues limited the completeness, comparability, and usefulness of the May 2013 program inventories  OMB and agencies did not take a systematic approach to developing comprehensive inventories  For example, OMB’s guidance in Circular No  A-11 presented five possible approaches agencies could take to define their programs and noted that agencies could use one or more of those approaches in doing so  We found that because the agencies used inconsistent approaches to define their programs, the comparability of programs was limited within agencies as well as government-wide  In addition, we found that the inventories had limited usefulness for decision making, as they did not consistently provide the program and related budget and performance information required by GPRAMA  Moreover, we found that agencies did not solicit feedback on their inventories from external stakeholders—which can include Congress, state and local governments, third party service providers, and the public  Doing so would have provided OMB and agencies an opportunity to ensure they were presenting useful information for stakeholder decision making  We concluded that the ability to tag and sort information about programs through a more dynamic, web-based presentation could make the inventory more useful  In October 2014, we made several recommendations to OMB to update relevant guidance to help develop a more coherent picture of all federal programs and to better ensure relevant information is useful for decision makers  For example, we recommended that OMB revise its guidance to direct agencies to consult with relevant congressional committees and stakeholders on their approach to defining and identifying programs when developing or updating their inventories  OMB staff generally agreed with these recommendations, but have not yet taken any actions to implement them  OMB’s guidance for the program inventory has largely remained unchanged since 2014, when OMB postponed further development of the program inventory and eliminated portions of the guidance  For example, the guidance no longer describes, or provides directions for agencies to meet, GPRAMA’s requirements for presenting related budget or performance information for each program  OMB decided to postpone implementing a planned May 2014 update to the program inventory in order to coordinate with the implementation of the public spending reporting required by the Digital Accountability and Transparency Act of 2014 (DATA Act)  OMB subsequently stated that it would not begin implementing the program inventory until after the DATA Act was implemented in May 2017, despite requirements for regular updates to the program inventory to reflect current budget and performance information  The DATA Act is now being implemented, but OMB has postponed resuming the development of the program inventory  In July 2017, OMB staff told us that they are now considering how to align GPRAMA’s program inventory provisions with future implementation of the Program Management Improvement Accountability Act (PMIAA)  This was reflected in OMB’s July 2017 update to its guidance, which states that OMB is working with agencies to determine the right strategy to merge the implementation of the DATA Act and PMIAA with GPRAMA’s program inventory requirements to the extent possible to avoid duplicating efforts  For example, PMIAA requires OMB to coordinate with agency Program Management Improvement Officers to conduct portfolio reviews of agency programs to assess the quality and effectiveness of program management  GPRAMA requires OMB to issue guidance for implementing the program inventory requirements, among other things  Moreover, federal internal control standards state that organizations should clearly define what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement  As described above, OMB’s current guidance for the program inventory lacks some of those details—such as describing and providing direction to meet GPRAMA’s requirements for budget and performance information—in part because OMB is working with agencies to determine a strategy for implementation  Ensuring all GPRAMA requirements are covered and taking action on our past recommendations would help OMB improve its guidance to more fully implement the program inventory and improve its usefulness  To that end, in a report issued earlier this month, we identified a series of iterative steps that OMB could use in directing agencies to develop a useful inventory, as described in figure 3  A useful inventory would consist of all programs identified, information about each program, and the organizational structure of the programs  Our work showed that the principles and practices of information architecture—a discipline focused on organizing and structuring information—offer an approach for developing such an inventory to support a variety of uses, including increased transparency for federal programs  Such a systematic approach to planning, organizing, and developing the inventory that centers on maximizing the use and usefulness of information could help OMB ensure the inventory is implemented in line with GPRAMA requirements and meets the needs of decision makers and the public, among others  OMB’s guidance also lacks specific time frames, with associated milestones for resuming implementation of the program inventory requirements  As part of PMIAA’s requirements, OMB is to issue standards, policies, and guidelines for program and project management for agencies by December 2017  OMB staff told us that, within a year after that, they expect to issue further guidance on moving forward with resuming the program inventory  However, that general time frame was not reflected in the July 2017 update to OMB’s guidance  Providing specific time frames and associated milestones would bring the program inventory guidance in line with other portions of OMB’s guidance for implementing GPRAMA requirements, which contains a timeline of various performance planning and reporting requirements, including specific dates for meeting those requirements and related descriptions of required actions  For example, OMB’s July 2017 guidance identifies over 30 actions agencies should take between June 2017 and December 2018 to implement various GPRAMA provision  More specific time frames and milestones related to the program inventory requirements would help agencies prepare for resumed implementation by allowing them to know what actions they would be expected to take and by when  Moreover, publicly disclosing planned implementation time frames and associated milestones also would help ensure that external stakeholders are prepared to engage with agencies as they develop and update their program inventories  Effectively implementing various GPRAMA tools could help inform assessments of the performance of tax expenditures, which are reductions in tax liabilities that result from preferential provisions (figure 4)  In fiscal year 2016, tax expenditures represented an estimated $1 4 trillion in forgone revenue, an amount greater than total discretionary spending that year  Despite the magnitude of these investments, our work has also shown that little has been done to determine how well specific tax expenditures work to achieve their stated purposes and how their benefits and costs compare to those of spending programs with similar goals  GPRAMA requires OMB to identify tax expenditures that contribute to the CAP goals  In addition, OMB guidance directs agencies to identify tax expenditures that contribute to their strategic objectives and APGs  However, our past work reviewing GPRAMA implementation found that OMB and agencies rarely identified tax expenditures as contributors to these goals  Fully implementing our recommendation to identify how tax expenditures contribute to various goals could help the federal government establish a process for evaluating the performance of tax expenditures  To that end, in May 2017, we provided the Director of OMB with three priority recommendations that require attention: Develop framework for reviewing performance: In June 1994, and again in September 2005, we recommended that OMB develop a framework for reviewing tax expenditure performance  We explained that the framework should (1) outline leadership responsibilities and coordination among agencies with related responsibilities, (2) set a review schedule, (3) identify review methods and ways to address the lack of credible tax expenditure performance information, and (4) identify resources needed for tax expenditure reviews  Since their initial efforts in 1997 and 1999 to outline a framework for evaluating tax expenditures and preliminary performance measures, OMB and the Department of the Treasury (Treasury) have ceased to make progress and retreated from setting a schedule for evaluating tax expenditures  Inventory tax expenditures: In October 2014, we found that OMB had not included tax expenditures in the federal program inventory, and therefore was missing an opportunity to increase the transparency of tax expenditures and the outcomes to which they contribute  We recommended that OMB should designate tax expenditures as a program type in relevant guidance, and develop, in coordination with the Secretary of the Treasury, a tax expenditure inventory that identifies each tax expenditure and provides a description of how the tax expenditure is defined, its purpose, and related budget and performance information  OMB staff said they neither agreed nor disagreed with these recommended actions  As noted earlier, OMB has not resumed updates to the program inventory  Therefore, OMB had not taken any actions in response to this recommendation, according to OMB staff as of July 2017  Identify contributions to agency goals: In July 2016, we found that agencies had made limited progress identifying tax expenditures’ contribution to agency goals, as directed by OMB guidance  As of January 2016, 7 of the 24 CFO Act agencies identified tax expenditures as contributing to their missions or goals  The 11 tax expenditure they identified—out of the 169 tax expenditures included in the President’s Budget for Fiscal Year 2017—represented approximately $31 9 billion of the $1 2 trillion in estimated forgone revenues for fiscal year 2015  (See figure 5 ) To help address this issue, we recommended that OMB, in collaboration with the Department of the Treasury, work with agencies to identify which tax expenditures contribute to their agency goals, as appropriate  In particular, we recommended that they identify which specific tax expenditures contribute to specific strategic objectives and APGs  In July 2017, OMB staff said they had taken no actions to address the recommendation  Our July 2016 report also identified options for policymakers to further incorporate tax expenditures into federal budgeting processes, several of which options align with the recommendations discussed above  These options could help achieve various benefits, but we also reported that policymakers would need to consider challenges and tradeoffs in deciding whether or how to implement them  For example, one option was to require that all tax expenditures, or some subset of them, expire after a finite period  This option could result in greater oversight, requiring policymakers to explicitly decide whether to extend more or all tax expenditures  One consideration with this option is that it could lead to frequent changes in the tax code, such as from extended or expired tax expenditures, which can create uncertainty and make tax planning more difficult  Our previous work has shown that using performance information in decision making is essential to improving results  Performance information can be used across a range of management activities, such as setting priorities, allocating resources, or identifying problems to be addressed  However, our work continues to show that agencies can better use performance information in decision making, as shown in the example in the text box below  Department of Justice (DOJ) Could Better Analyze Performance Information to Reduce Backlog in Immigration Courts In June 2017, we found that the case backlog—cases pending from previous years that remain open at the start of a new fiscal year—at DOJ’s Executive Office for Immigration Review (EOIR) courts more than doubled from fiscal years 2006 through 2015  Stakeholders identified various factors that potentially contributed to the backlog, including continuances—temporary case adjournments until a different day or time  Our analysis of continuance records showed that the use of continuances increased by 23 percent from fiscal years 2006 through 2015  We found that EOIR collects continuance data but does not systematically assess them  Systematically analyzing the use of continuances could provide EOIR officials with valuable information about challenges the immigration courts may be experiencing, such as with operational issues like courtroom technology malfunctions, or areas that may merit additional guidance for immigration judges  Further, using this information to potentially address operational challenges could help that office meet its goals for completing cases in a timely manner  We recommended that the Director of EOIR systematically analyze immigration court continuance data to identify and address any operational challenges faced by courts or areas for additional guidance or training  EIOR agreed with this recommendation  EOIR stated that it supports conducting additional analysis of immigration court continuance data and recognizes that additional guidance or training regarding continuances may be beneficial to ensure that immigration judges use continuances appropriately in support of EOIR’s mission to adjudicate immigration cases in a careful and timely manner  We will monitor EOIR’s progress in taking these actions  Our 2017 survey of federal managers shows little change in their reported use of performance information  Using a set of survey questions, we previously developed an index that reflects the extent to which managers reported that their agencies used performance information for various management activities and decision making  The index suggests that government-wide use of performance information did not change significantly between 2013 and 2017, and it is statistically significantly lower relative to our 2007 survey, when we created the index  Figure 6 shows the questions included in the index and the government-wide results  In regard to individual survey items, in 2017 federal managers reported no changes or decreases in their use of performance information when compared to our last survey and when those survey items were first introduced  These results are generally consistent with our last few surveys  For example, in 2008 we found that there had been little change in federal managers’ reported use of performance information government-wide from 1997 to our 2007 survey  Citing those results, the Senate Committee on Homeland Security and Governmental Affairs report accompanying the bill that would become GPRAMA stated that agencies were not consistently using performance information to improve their management and results  The report further stated that provisions in GPRAMA are intended to address those findings and increase the use of performance information to improve performance and results  However, five items that were highlighted in our 2008 statement on the 2007 survey results generally show no improvement when compared to the 2017 results, as shown in figure 7  The one exception is for managers’ reported use of performance information to refine program performance measures  While this item was statistically significantly higher in 2013 relative to 2007—an estimated 46 percent to 53 percent—the 2017 result (43 percent) is a statistically significant decrease relative to 2013 and is not statistically different from the 2007 results  Another item, the use of performance information to adopt new program approaches or change work processes, also was statistically significantly lower in 2017 (47 percent) when compared to 2007 and 2013 (53 and 54 percent, respectively)  This is of particular concern as agencies are developing their reform plans  Moreover, when compared to our 1997 survey, the 2017 results show four of the five items are statistically significantly lower, and the remaining item—allocating resources—has not changed  Similarly, we found there was no improvement in 2017 for more recent survey items on other uses of performance information compared to the years in which they were introduced, as shown in figure 8  Although one item, on the use of performance information to develop program strategy, was statistically significantly higher in 2013 relative to 2007 (an estimated 58 and 51 percent, respectively), the 2017 result (53 percent) does not represent a statistically significant change from either of those years  Another item, on the use of performance information to streamline programs to reduce duplicative activities, is statistically significantly lower relative to 2013, when it was introduced (from 44 to 33 percent in 2017)  This is especially concerning because streamlining and reducing duplication are to be key parts of agencies’ reform plans  There is one area in the survey where we saw improvement: an estimated 46 percent of managers agreed to a great or very great extent that employees who report to them pay attention to their agency’s use of performance information in management decision making  That is statistically significantly higher relative to 2013 (40 percent), as well as when compared to when the item was introduced in 2007 (37 percent)  For a new and related item in the 2017 survey that asked managers the amount of attention their employees pay to the use of performance information in decision making when compared to 3 years ago, we found an estimated 48 percent reported that employees pay about the same 33 percent reported that employees pay somewhat or a great deal more attention  In September 2005, we identified five practices that agencies can apply to enhance the use of performance information in their decision making and improve results: demonstrating management commitment; communicating performance information frequently and efficiently; improving the usefulness of performance information, such as by ensuring the accessibility of the information; developing the capacity to use performance information; and aligning agency-wide goals, objectives, and measures  Many of the requirements put in place by GPRAMA reinforce the importance of these practices  Our 2017 survey of federal managers includes a number of items related to these practices  However, the 2017 results suggest that managers have not effectively adopted them  In the following sections, we examine several of the practices to enhance the use of performance information and their related survey items further  In doing so, we also highlight a subset of six survey items related to these practices that, while separate from those in our use of performance information index, we found in September 2014 to have a statistically significant and positive relationship with it  The commitment of agency leaders to results-oriented management is critical to increased use of performance information for policy and program decisions  GPRAMA requires top leadership involvement in performance management, including leading data-driven performance reviews  However, we have previously reported that improvements are needed to strengthen leadership’s commitment to use performance information, as discussed in the text box below  Department of Defense Should Strengthen Leadership Responsibilities for Using Performance Information In January 2005, we designated the Department of Defense’s (DOD) approach to business transformation as high-risk because DOD had not taken the necessary steps to achieve and sustain business reform on a broad, strategic, department-wide, and integrated basis  In the February 2017 update to our High-Risk List, we found that DOD had taken some positive steps to improve its business transformation efforts continuing to hold business function leaders accountable for diagnosing performance problems and identifying strategies for improvement, and leading regular DOD performance reviews regarding transformation goals and associated metrics and ensuring that business function leaders attend these reviews to facilitate problem solving  In July 2017, DOD officials told us that the department’s performance reviews have been put on hold until after the new Agency Strategic Plan is issued  We will review DOD’s updated Agency Strategic Plan when it is issued (expected in February 2018, as required by GPRAMA) to see if it addresses continuing to hold business function leaders accountable for diagnosing performance problems and identifying strategies for improvement  We will continue to monitor the status of these actions  GAO, High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others, GAO-17-317 (Washington, D C : Feb  15, 2017)  Results from our 2017 survey show no statistically significant difference relative to 2013 in managers’ perceptions of leaders’ and supervisors’ attention and commitment to the use of performance information  (See figure 9 ) Three items are statistically significantly different from the years when they were introduced  Two items increased between 1997 and 2017: changes by management to my program(s) are based on results-oriented information (from an estimated 16 to 25 percent), and the individual I report to periodically reviews with me the outcomes of my program(s) (from 42 to 54 percent)  For the third item, top leadership demonstrates a strong commitment to using performance information to guide decision making, results decreased from 49 percent in 2007 to 42 percent in 2017  New items in the 2017 survey show some improvement in management commitment to the use of performance information in decision making  An estimated 36 percent of federal managers reported that, when compared to 3 years ago, the individual they report to pays somewhat or a great deal more attention to the use of performance information in decision making, while 46 percent said they pay about the same amount of attention  Additionally, an estimated 21 percent of federal managers said that, when compared to 3 years ago, the head of their agency pays somewhat or a great deal more attention to the use of performance information in decision making, while 33 percent said they pay about the same amount of attention  Communicating performance information frequently and effectively throughout an agency can help to achieve the agency’s goals  GPRAMA includes requirements for communicating performance information, such as reporting progress updates for APGs at least quarterly  However, our prior work has found that some agencies could continue to improve in the communication of performance information, as illustrated by the example in the text box below  Department of Education (Education) Could Better Share Effective Practices across States in Grant Program Education awards 21st Century Community Learning Centers grants to states, which in turn competitively award funds to local organizations that use them to offer academic enrichment and other activities to improve students’ academic and behavioral outcomes  In April 2017, we found that states are experiencing substantial difficulty in sustaining their programs after 21st Century funding ends  We further found that Education was missing opportunities in its monitoring efforts to collect information on states’ strategies and practices for program sustainability—information that could be useful for sharing promising practices across states  We recommended that Education use the information it collects from its monitoring visits and ongoing interactions with states to share effective practices across states for sustaining their 21st Century programs once program funding ends  Education neither agreed nor disagreed with the recommendation but outlined steps it is taking to address it  We will continue to monitor progress on the implementation of this recommendation  There is no difference for two survey items on federal managers communicating performance information relative to 2013 or since those items were introduced in 2007  In 2017, we estimate that 44 percent of federal managers agreed to a great or very great extent that agency managers at their level effectively communicate performance information on a routine basis  In addition, 34 percent agreed to a great or very great extent that managers at their level use performance information to share effective program approaches with others  Our 2017 survey data also indicate that agencies may not be effectively communicating to their employees about contributions to CAP goals or progress toward achieving APGs  Of the estimated 54 percent of federal managers who indicated they were familiar with CAP goals, 23 percent reported that their agency has communicated to its employees on those goals to a great or very great extent  Of the 74 percent of federal managers who indicated familiarity with APGs, 44 percent reported that their agency has communicated on progress toward achieving those goals to great or very great extent  Our prior work has shown that agencies should consider users’ differing needs—for accessibility, accuracy, completeness, consistency, ease of use, timeliness, and validity, among others things—to ensure that performance information will be both useful and used  GPRAMA introduced several requirements that could help to address aspects of usefulness, such as requiring agencies to disclose more information about the accuracy and validity of their performance data and actions to address limitations to the data  However, agencies face challenges in ensuring their performance information is useful, with one instance from our past work described in the text box below  The Environmental Protection Agency (EPA) Could Improve Usefulness of Information in Planned Grantee Portal EPA monitors performance reports and program-specific data from grantees to ensure that grants achieve environmental and other program results  However, in July 2016, we found that EPA’s 2014 internal analysis of its grants management business processes identified improvements that, if implemented into EPA’s planned web-based portal, could improve the accessibility and usefulness of information in grantee performance reports for EPA, grantees, and other users  We recommended, among other actions, that EPA incorporate expanded search capability features, such as keyword searches, into its proposed web- based portal for collecting and accessing performance reports to improve their accessibility  EPA agreed with our recommendation but stated that it is a long- term initiative, subject to the agency’s budget process and replacement of its existing grants management system  As of May 2017, EPA officials said that they have not begun work on the web-based portal project, which is subject to the availability of funds  Federal managers generally responded similarly in 2017 on a variety of survey items related to usefulness, relative to earlier surveys  On a broadly worded item, less than half of managers agreed to a great or very great extent that agency managers at their level take steps to ensure that performance information is useful and appropriate  At an estimated 43 percent in 2017, this represents no statistically significant change compared to our last surveys in 2013 or 2007, when the item was introduced  Responses to four survey items indicate no changes in hindrances related to the usefulness of performance information  There is no statistically significant change in managers reporting hindrances compared to 1997 or 2013, as shown in figure 10  In addition, there was a statistically significant increase when compared to 2013 on only one of six items about managers’ views on the usefulness of performance information, as shown in figure 11  As the figure shows, approximately one-third to half of managers agreed to a great or very great extent on each item related to the usefulness of performance information  Although less than half of managers reported having sufficient information on validity of performance data used to make decisions, this represents a statistically significant increase to an estimated 42 percent in 2017 compared to 36 percent in 2013, and from 28 percent in 2000, when this item was introduced  This is a notable improvement because our September 2014 report found that the strongest driver of the use of performance information was whether federal managers had confidence in its validity  Our analysis suggests that easy access to performance information is related to the effective communication of performance information  Of the estimated 49 percent of federal managers in 2017 who agreed to a great or very great extent that performance information is easily accessible to managers at their level, 63 percent also agreed that agency managers at their level effectively communicate performance information on a routine basis to a great or very great extent  Conversely, of the 20 percent that agreed to a small or no extent that performance information is easily accessible to managers at their level, 12 percent also agreed that agency managers at their level effectively communicate performance information on a routine basis to a great or very great extent  Our prior work has shown that building capacity—including analytical tools and staff expertise—is critical to using performance information in a meaningful manner  GPRAMA lays out specific requirements that reinforce the importance of staff capacity to use performance information  GPRAMA directed the Office of Personnel Management (OPM) to take certain actions to support agency hiring and training of performance management staff  Specifically, by January 2012, OPM was to identify skills and competencies needed by government personnel for setting goals, evaluating programs, and analyzing and using performance information for improving government efficiency and effectiveness  By January 2013, OPM was to incorporate these skills and competencies into relevant position classifications and to work with each agency to incorporate the identified skills into employee training  In April 2013, we found that OPM had completed its work on the first two responsibilities and taken steps to work with agencies to incorporate performance management staff competencies into training  However, OPM did not assess competency gaps among agency performance management staff to inform its work  Without this information, OPM, working with the PIC, was not well-positioned to focus on the most- needed resources and help other agencies use them  We recommended that the Director of OPM, in coordination with the PIC and the Chief Learning Officer Council, work with agencies to take the following three actions: 1  Identify competency areas needing improvement within agencies  2  Identify agency training that focuses on needed performance management competencies  3  Share information about available agency training on competency areas needing improvement  In July 2017, PIC staff stated they have not focused on identifying competency areas because the competencies do not resonate strongly with the performance community  Instead, staff said they identified a need for introductory training on performance management, which they have developed and piloted  They said that they are not sure when they will implement the training, since the PIC is reviewing priorities with its new executive director  We continue to believe that identifying the competency areas would be useful, and will monitor the PIC’s efforts to identify and share training  The need for performance management training is further highlighted by our survey results  Our 2017 survey shows no statistically significant change in managers’ responses about the availability of training on various performance management activities relative to 2013, including the use of performance information to make decisions  However, the response to each of the six questions related to specific training is statistically significantly higher relative to the year in which it was introduced, as shown in figure 12  Similarly, in 2017 there was no statistically significant change on four survey items related to agencies’ analysis and evaluation tools and staff’s skills and competencies when compared to 2013 or when these items were introduced  We estimate that in 2017 29 percent of managers agreed to a great or very great extent that their agencies were investing in resources to improve the agencies’ capacity to use performance information; 28 percent of managers agreed to a great or very great extent that their agencies were investing the resources needed to ensure that performance data are of sufficient quality; 33 percent of managers reported that they agreed to a great or very great extent that their agencies have sufficient analytical tools for managers at their levels to collect, analyze, and use performance information; and 33 percent of managers reported that they agree to a great or very great extent that the programs they are involved with have sufficient staff with the knowledge and skills needed to analyze performance information  Performance reviews can serve as a strategy to bring leadership and other responsible parties together to review performance information and identify important opportunities to drive performance improvements  Our prior work has examined how different types of performance reviews—strategic reviews, data-driven reviews, and retrospective regulatory reviews—can contribute to agencies assessing progress toward desired results  Strategic reviews: As previously mentioned, in implementing GPRAMA, OMB established a review process in which agencies are to annually assess their progress in achieving each strategic objective in their strategic plans, known as strategic reviews  Given the long-term and complex nature of many outcomes, the strategic review should be informed by a variety of evidence regarding the implementation of strategies and their effectiveness in achieving outcomes  OMB’s guidance states that the strategic review process should consider multiple perspectives and sources of evidence to understand the progress made on each strategic objective  It further states that the results of these reviews should inform many of the decision-making processes at the agency, as well as decision making by the agency’s stakeholders, in areas such as long-term strategy, budget formulation, and risk management  In 2017, agencies are completing their fourth round of these reviews  Our prior work has identified ways in which agencies can effectively conduct these reviews and leverage the results that come from them  In July 2015, we identified seven practices federal agencies can employ to facilitate effective strategic reviews  (See sidebar ) In addition, earlier this month we reported on selected agencies’ experiences in implementing these reviews  Specifically, we found that (1) strategic reviews helped direct leadership attention to progress on strategic objectives, (2) agencies used existing management and performance processes to conduct the reviews, and (3) agencies refined their reviews by capturing lessons learned  Data-driven reviews: GPRAMA requires agencies to review progress toward APGs at least once a quarter  The Senate Committee on Homeland Security and Governmental Affairs report accompanying the bill that would become GPRAMA stated that this approach is aimed at increasing the use of performance information to improve performance and results  In February 2013, we identified nine leading practices to promote successful data-driven performance reviews in the federal government  (See sidebar ) In July 2015, we found that most of the 24 CFO Act agencies were conducting their reviews in line with GPRAMA requirements and our leading practices  Moreover, agencies reported that their data-driven performance reviews had positive effects on progress toward agency goals, collaboration between agency officials, the ability to hold officials accountable for progress, and efforts to improve the efficiency of operations  Our 2017 survey shows that federal managers remain largely unfamiliar with their agency’s data-driven performance reviews, also known as quarterly performance reviews (QPRs)  An estimated 35 percent of managers reported familiarity with their agency’s QPRs  Survey results show that a greater percentage of Senior Executive Service (SES) managers than non-SES managers reported that they were familiar with QPRs  Approximately 50 percent of SES managers reported being somewhat or very familiar with QPRs; 34 percent of non-SES reported the same  However, for the estimated 35 percent of managers who reported familiarity with QPRs, the more they viewed their programs being subject to a QPR, the more likely they were to report their agency’s QPRs were driving results and conducted in line with our leading practices  Figure 13 shows several illustrative examples of these survey items  For example, of the estimated 48 percent of federal managers who reported their programs being subject to QPRs to a great or very great extent, 83 percent also reported their agencies use QPRs to identify problems or opportunities associated with agency performance goals  Conversely, for the 24 percent of managers who reported their programs were subject to QPRs to a small or no extent, 22 percent also reported the reviews were used for these purposes to a great or very great extent  Being subject to a QPR is also positively related to viewing QPRs as having led to similar meetings at lower levels  An estimated 62 percent of federal managers who reported being subject to QPRs to a great or very great extent also reported their agencies have similar meetings at lower levels to a great or very great extent  An estimated 16 percent of federal managers subject to QPRs to a small or no extent reported the same  Despite the reported benefits of and results achieved through QPRs, as found by our past work and survey data, these reviews are not necessarily widespread  GPRAMA requires agencies to conduct QPRs for APGs, which represent a small subset of goals—generally 2 to 8 priority goals at each designated agency, with approximately 100 total government-wide  Moreover, these required reviews are at the department (or major independent agency) level  These reasons may explain why most managers reported they were not familiar with the reviews  As was described previously, our 2017 survey data show that the reported use of performance information in decision making generally has not improved and in some cases is lower than it was 20 years ago  Survey data also show that managers generally have not reported increases in their employment of practices that further promote the use of performance information in decision making  This suggests that agencies could increase the use of performance information in decision making and the likelihood of achieving desired results by going beyond the specific GPRAMA requirements and expanding their use of data-driven performance reviews—in line with leading practices—to more broadly cover other agency-wide performance goals, as well as goals at lower levels within the agency  For example, such reviews at the program level could help inform the previously mentioned portfolio reviews required by the Program Management Improvement Accountability Act (PMIAA)  We have already suggested expanding reviews to other performance goals  Our management agenda for the presidential and congressional transition includes a key action to expand the use of data-driven performance reviews to assess progress toward meeting agency performance goals  Our prior work has stated that although GPRAMA’s requirements apply at the agency-wide level, they can also serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects  In addition, federal internal control standards call for the design of appropriate control activities, such as top-level reviews of actual performance and reviews by management at the functional or activity level  The standards also recommend that management design control activities at the appropriate levels in the organizational structure  The July 2017 update to OMB’s guidance states that agency leaders, including various chief officer positions, are to conduct frequent data- driven reviews to drive improvements on various management functions  For example, the agency Chief Human Capital Officer is to conduct quarterly data-driven reviews (known as HRStat) to monitor the progress of human capital goals and measures contained in the human capital operating plan  Beyond these management areas, OMB’s guidance also states that agencies may expand quarterly progress reviews beyond APGs to include other goals and priorities  However, OMB’s guidance does not identify practices for agencies to expand the use of these reviews to other goals, such as other agency-wide performance goals or those at lower levels within the agency  As mentioned previously, one of the responsibilities of the Performance Improvement Council (PIC) is to facilitate the exchange among agencies of practices that have led to performance improvements within specific programs, agencies, or across agencies  By working with the PIC to identify and share among agencies practices to expand the use of data- driven reviews, OMB could help agencies increase the use of performance information in decision making and achieve results  Retrospective regulatory reviews: In retrospective reviews, agencies evaluate how existing regulations are working in practice and whether they are achieving expected outcomes  GPRAMA requires agencies to identify and assess how their various program activities and other activities, including regulations, contribute to APGs  However, in April 2014, we found that agencies reported mixed experiences linking retrospective analyses to APGs  We recommended that OMB strengthen these reviews by issuing guidance for agencies to take actions to ensure that contributions made by regulations toward achieving APGs are properly considered, and improve how retrospective regulatory reviews can be used to help inform assessments of progress toward these APGs  OMB staff agreed with this recommendation and stated that the agency was working on strategies to help facilitate agencies’ ability to use retrospective reviews to inform APGs  To that end, in April 2017, OMB issued guidance to agencies that, among other things, emphasized the importance of performance measures related to evaluating and improving the net benefits of their respective regulatory programs  OMB included explicit references to section 6 of Executive Order 13563, which directed agencies’ efforts to conduct retrospective regulatory reviews  Specifically, the updated guidance encourages agencies to establish and report “meaningful performance indicators and goals for the purpose of evaluating and improving the net benefits of their respective regulatory programs ” The guidance further states that agencies’ efforts to improve such net benefits may be conducted as part of developing agency strategic and performance plans and priority goals  In July 2017, OMB confirmed that the updated guidance was issued, in part, to address our April 2014 recommendation  For several years, OMB has encouraged agencies to expand their use of evidence—performance measures, program evaluation results, and other relevant data analytics and research studies—in budget, management, and policy decisions with the goal of improving government effectiveness  In particular, OMB has encouraged agencies to strengthen their program evaluations—systematic studies that use research methods to address specific questions about program performance  Evaluation is closely related to performance measurement and reporting  Evaluations can be designed to better isolate the causal impact of programs from other external economic or environmental conditions in order to assess a program’s effectiveness  Thus, an evaluation study can provide a valuable supplement to ongoing performance reporting by measuring results that are too difficult or expensive to assess annually, explaining the reasons why performance goals were not met, or assessing whether one approach is more effective than another  Despite the valuable insights and information that program evaluations can provide, we continue to find that most federal managers lack access to or awareness of such studies  Our 2017 survey shows that an estimated 40 percent of managers reported that an evaluation had been completed within the past 5 years of any program, operation, or project in which they were involved—comparable to the results in our 2013 survey, when questions about program evaluations were added  In recent years, OMB has encouraged agencies to explore evidence-based tools to strengthen agency and grantee evaluation capacity, consider the effectiveness of their programs, and foster innovation rooted in research and rigorous evaluation  During the past 2 years, we examined several of those tools, as described below  Pay for success: Also known as social impact bonds, pay for success is a contracting mechanism under which investors provide the capital the government uses to provide a social service  The government specifies performance outcomes in pay for success contracts and generally includes a requirement that a program’s impact be independently evaluated  The evaluators also are to regularly review performance data, while those managing and investing in a project focus on performance and accountability, as shown in the figure 14  In September 2015, we found that the federal government’s involvement in pay for success had been limited  In addition, a formal mechanism for federal agencies to collaborate on pay for success did not exist  We concluded that, given the evolving nature of pay for success, a mechanism for federal agencies to collaborate would increase access to leading practices  We therefore recommended that OMB establish a formal means for federal agencies to collaborate on pay for success  OMB concurred and, in February 2016, announced that it had developed the Pay for Success Interagency Learning Network with representatives from 10 federal agencies to share lessons, hone policy, and strengthen implementation  Tiered evidence grants: Tiered evidence grants seek to incorporate evidence of effectiveness into grant making  Federal agencies establish tiers of grant funding based on the level of evidence grantees provide on their approaches to deliver social, educational, health, or other services  (See figure 15 ) Smaller awards are used to test new and innovative approaches, while larger awards are used to scale up approaches that have strong evidence of effectiveness  This creates incentives for grantees to use approaches supported by evidence and helps them build the capacity to conduct evaluations  In September 2016, we found that interagency collaboration had helped federal agencies that administer tiered evidence grants address challenges and share lessons learned  At that time, such collaborative efforts relied on informal networks  We recommended that OMB establish a formal means for agencies to collaborate on tiered evidence grants  OMB had no comment on the recommendation  In July 2017, OMB staff told us that they had established an interagency working group and other mechanisms to facilitate collaboration and disseminate information on tiered evidence grants  Performance partnerships: Performance partnerships allow federal agencies to provide grant recipients flexibility in how they use funding across two or more programs along with additional flexibilities  In exchange, the recipient commits to improve and assess progress toward agreed-upon outcomes  Figure 16 provides an overview of the performance partnership model  In April 2017, we examined two performance partnership initiatives authorized by Congress: the Environmental Protection Agency’s Performance Partnership Grants and the Performance Partnership Pilots for Disconnected Youth, which allows funding from multiple programs across multiple agencies to be combined into pilot programs serving disconnected youth  For the Performance Partnership Pilots for Disconnected Youth, we found that the agencies involved in the initiative had not fully identified the key financial and staff resources each agency would need to contribute over the lifetime of the initiative in line with leading practices for interagency collaboration  This was because agencies primarily had been focused on meeting near-term needs to support design and implementation  We also found that agencies had not developed criteria to help determine whether, how, and when to implement the flexibilities tested by the pilots in a broader context  (This is known as scalability ) Officials involved in the pilots told us it was too early in pilot implementation to determine such criteria  However, by not identifying these criteria while designing the pilots, they were risking not collecting needed data during pilot implementation  We recommended that OMB coordinate with federal agencies to identify (1) agency resource contributions needed for the lifetime of the pilots and (2) criteria and related data for assessing scalability  OMB neither agreed nor disagreed with these recommendations  We continue to monitor progress on these recommendations  In 2003, we identified nine key practices for effective performance management that collectively create a “line of sight” between individual performance and organizational success  (See sidebar on next page ) Our recent work and the results of our 2017 survey of federal managers highlight areas where agencies have made progress but could take additional action to better reflect several of these practices, thereby better instilling results-oriented cultures  Align individual performance expectations with organizational goals: Our 2003 report found that high-performing organizations use their performance management systems to help individuals see the connection between their daily activities and organizational goals  The executive branch has taken several steps to link individual and organizational results  For example, in October 2000, OPM issued guidance to link SES performance expectations with GPRA-required goals  In January 2012, OPM and OMB released a government-wide SES performance appraisal system that provided agencies with a standard framework to manage the performance of SES members  However, our work continues to identify areas for improvement  Goal leaders and deputy goal leaders are responsible for achieving APGs, but our July 2014 review found that the performance plans for a sample of goal and deputy goal leaders generally did not link their individual performance and the broader goal  We recommended that OMB ensure that those plans demonstrate a clear connection with APGs  OMB staff generally agreed with our recommendation  In July 2017, OMB staff stated that components of both OMB and OPM guidance support accountability for agency priority goals  Despite this, we continue to believe that ensuring an explicit connection in performance plans to APGs will improve accountability, and that additional action is needed to do so  In May 2016, we found that the Federal Emergency Management Agency (FEMA) had not aligned Federal Disaster Recovery Coordinators’ performance expectations with its organizational goals for implementing the National Disaster Recovery Framework  We concluded that without this linkage, FEMA could not evaluate how effectively the coordinators performed in implementing the framework  We recommended that FEMA align performance expectations consistent with leading practices  The Department of Homeland Security concurred with our recommendation  In July 2017, FEMA stated that it is preparing the Field Leader Manual, which will define the core competencies and duties of coordinators  We will continue to monitor FEMA’s actions to implement this recommendation  Our 2017 survey also shows that this linkage could be improved for other federal employees  An estimated 58 percent of federal managers reported using performance information to a great or very great extent in setting expectations for employees they manage or supervise  The 2017 responses do not represent a statistically significant change when compared to our last survey in 2013 (62 percent) or to 1997 (61 percent), the year this survey item was introduced  Address organizational priorities: Our prior work showed that, by requiring and tracking follow-up actions on performance gaps, high- performing organizations underscore the importance of holding individuals accountable for making progress on their priorities  Our past and 2017 surveys have identified differences in responses between SES and non-SES managers reporting being held accountable for results  For example, in 2017, our survey results indicate that there was a statistically significant difference between SES and non-SES managers reporting to a great or very great extent that they were held accountable for results of the programs for which they are responsible  However, our 2017 survey shows no change compared to our last survey in either SES or non-SES managers reporting they were held accountable for results  There are statistically significant increases when compared to 1997, when these survey items were introduced  For example, an estimated 79 percent of SES managers and 64 percent of non-SES managers reported being held accountable to a great or very great extent for results of the programs for which they are responsible in 2017  This does not represent a statistically significant change from our 2013 survey (80 percent and 67 percent, respectively), but it is statistically significantly higher than the 62 percent of SES managers and 54 percent of non-SES managers in 1997  (See figure 17 ) Similarly, as shown in figure 18, an estimated 71 percent of SES managers reported being held accountable to a great or very great extent for accomplishing agency strategic goals in 2017  This represents no statistical change since 2013 (73 percent), but it is a statistically significant increase compared to when this item was introduced in 2003 (61 percent)  Additionally, as figure 18 shows, a gap between being held accountable for strategic goals and having the decision-making authority needed to help accomplish those goals has nearly closed, due to an increase in the latter survey item  The estimated 69 percent of SES managers who reported having such authority to a great or very great extent in 2017 is a statistically significant increase relative to both 2013 (61 percent) and 1997 (51 percent)  As noted earlier, GPRAMA requires goal leaders for CAP goals and APGs  Our past work has generally found that they are in place  GPRAMA also requires agencies to identify an agency official responsible for resolving major management challenges, which can help ensure accountability  (See sidebar ) However, in June 2016 we found that 17 of the 24 CFO Act agencies had not identified an agency official responsible for resolving each of their challenges, partly because OMB guidance was not clear that major management challenges should be identified in agency performance plans  We recommended that the 17 agencies identify such officials in their performance plans, and that OMB clarify its guidance  OMB revised its guidance accordingly in July 2016, and, as of July 2017, 7 of the 17 agencies had identified officials responsible for resolving major management challenges  Link pay to individual and organizational performance: High- performing organizations seek to create pay, incentive, and reward systems that clearly link employee knowledge, skills, and contributions to organizational results  Our work has found that agencies have made progress in this area  For example, in July 2013, we found that the Securities and Exchange Commission (SEC) lacked mechanisms to monitor how supervisors used its performance management system to recognize and reward performance  To help enhance the credibility of SEC’s performance management system, we recommended that it create mechanisms to monitor how supervisors use the performance management system  In a subsequent (December 2016) report, we found that, in response to our recommendation, SEC began monitoring how supervisors provide feedback, recognize and reward staff, and address poor performance  However, federal managers generally reported no change on three items related to recognizing and rewarding employee performance since our last survey in 2013 (figure 19)  One of those items—managers agreeing to a great or very great extent that employees in their agency receive positive recognition for helping the agency to accomplish its strategic goals—had a statistically significant increase between 1997 and 2017 (from an estimated 26 percent to 46 percent)  Make meaningful distinctions in performance: Effective performance management requires the organization’s leadership to meaningfully distinguish between acceptable and outstanding performance of individuals and to appropriately reward those who perform at the highest level  For example, in January 2015, we found disparities in performance ratings for SES among agencies  Across the 24 CFO Act agencies, the percent of SES rated at the highest level ranged from about 22 percent to 95 percent in fiscal year 2013  To help address these disparities, we recommended that the Director of OPM consider the need to refine the performance certifications guidelines addressing distinctions in performance  To address this recommendation, OPM informed us, in June 2015, that it had convened a cross-agency working group that developed a standard template for agencies to complete and post on a website to more transparently justify their SES ratings distributions  In May 2016, we found that about 74 percent of non-SES employees under a five-level appraisal system—the most commonly used system— were rated in the top two of five performance categories in 2013  We explored this issue further in our December 2016 review of human capital challenges at the Veterans Health Administration (VHA), which illustrates the importance of making meaningful distinctions in performance for non- SES employees  We found that in fiscal year 2014, about 73 percent of VHA employees were rated in the top two of five performance categories  This may have been due, in part, to a policy that did not require standards to be defined for each level of performance  We recommended that VHA ensure that meaningful distinctions are being made in employee performance ratings by reviewing and revising performance management policies consistent with leading practices, among other actions  The Department of Veterans Affairs partially concurred with our recommendation  In May 2017, the department stated that it had begun piloting a new performance management process and would analyze results at the end of fiscal year 2017  One key aspect of connecting daily operations to results is aligning program performance measures to agency-wide goals and objectives  However, in 2017, an estimated 50 percent of federal managers agreed to a great or very great extent that managers at their level took steps to create such an alignment  There has been no statistically significant change since this item was introduced in 2007  In addition, GPRAMA calls for agencies to develop a balanced set of performance measures, which reinforces the need for agencies to have a variety of measures across program areas  Our 2017 survey shows that managers have not reported any difference in the availability of performance measures for their programs when compared to the 2013 results  However, the 2017 result (an estimated 87 percent) represents a statistically significant increase when compared to 1997 (76 percent)  When asked about the availability of certain types of performance measures, three of the five types (outcome, output, and efficiency) were statistically significantly higher in 2017 when compared to our initial 1997 survey  However, when comparing 2017 results to those in 2013, two of the five types (output and quality) showed a statistically significant decrease, and the other types did not change  These are illustrated in figure 20  Beyond the survey results, our work has found that some agencies had not developed or used outcome measures, but have taken steps to do so  Agencies have been responsible for measuring program outcomes since GPRA was enacted in 1993  The text box below describes two illustrative examples from our past work  Examples of Agencies That Did Not Develop or Use Outcome Measures Patient access to electronic health information: In March 2017, we found that the Department of Health and Human Services (HHS) had invested over $35 billion since 2009 to enhance patient access to electronic health information, among other things  HHS had not developed outcome measures to gauge the effectiveness of these efforts, which meant the department did not have information to determine whether the efforts were contributing to its overall goals  We recommended that HHS develop relevant outcome measures and HHS concurred  Safety interventions: According to the Federal Motor Carrier Safety Administration (FMCSA), between 2011 and 2015, over 4,000 people died in crashes involving motor carriers each year  GAO, Motor Carriers: Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions, GAO-17-49 (Washington, D C : Oct  27, 2016)  Further OMB actions could also help agencies make progress in measuring the performance of different program types  In our June 2013 report on initial GPRAMA implementation, we found that agencies experienced common issues in measuring the performance of various types of programs, such as contracts and grants  We recommended that OMB work with the PIC to develop a detailed approach to examine those difficulties  Although they took some actions, OMB and the PIC have not yet developed a comprehensive and detailed approach to address these issues  We concluded that, without such an approach, it would be difficult for the PIC and agencies to fully understand these measurement issues and develop a crosscutting approach to help address them  In August 2017, OMB staff stated that efforts related to the future implementation of the Program Management Improvement Accountability Act (PMIAA) could help address this recommendation  As highlighted in table 1, our work continues to show why it is important for OMB and the PIC to take actions to more fully address our recommendation  Congress has passed legislation to increase the transparency and accessibility of federal performance and financial data  For example, GPRAMA modernized agency reporting requirements to ensure that they make timely, relevant data available to inform decision making by Congress and agency officials as well as improve transparency for the public  Results of our 2017 survey, however, show the need for improvements in the public availability of agency performance information  An estimated 17 percent of managers reported that their agency’s performance information is easily accessible to the public to a great or very great extent, the same percentage as in 2013  Moreover, of the 87 percent of managers that reported there are performance measures for the programs they are involved in, 25 percent reported that they use information obtained from performance measurement when informing the public about how programs are performing to a great or very great extent  This is not statistically different from the 30 percent estimated in 2013  The DATA Act, enacted in 2014, built on previous transparency legislation by expanding what federal agencies are required to report regarding their spending  The act significantly increases the types of data that must be reported, requires government-wide data standards, and regular reviews of data quality to help improve the transparency and accountability of federal spending data  OMB provides websites and guidance to make agency performance and financial information available to the public; however, our prior work has identified a number of areas related to Performance gov and the DATA Act where OMB action is needed to improve the transparency and accessibility of this information  Performance gov: Since 2013, our work has identified a number of issues with Performance gov, the website intended to serve as a central source of information on the federal government’s goals and performance  Over time, we have recommended that OMB take a number of specific actions to improve the website  For example, in June 2013, we found that the website offered an inconsistent user experience and presented accessibility and navigation challenges  To clarify the purpose of the website and enhance its usability, we recommended that OMB take steps to systematically collect customer input  In August 2016, we reported that OMB was not meeting all of the reporting requirements for Performance gov, and did not have a plan to develop and improve the website  We recommended that OMB ensure that information presented on Performance gov consistently complies with reporting requirements and develop a plan for the website that includes, among other things, a customer outreach plan  OMB agreed with these recommendations and, in July 2017, OMB staff informed us that they will be partnering with a vendor to redesign Performance gov to improve the accessibility of information on the website  To inform this redesign, OMB staff said that they will consider our previous recommendations and plan to engage a wide group of stakeholders, including Congress, agency staff, and interested members of the public and outside organizations  OMB staff anticipated releasing updated agency reporting guidance in the fall of 2017 and the redesigned website in February 2018  Under GPRAMA, OMB is required to make available, through Performance gov, quarterly updates on progress toward CAP goals and APGs  As described earlier, in June 2017 OMB announced that reporting to Performance gov has been discontinued through the end of fiscal year 2017 as agencies develop new priority goals  However, Performance gov does not state that it will not be updated, nor does it provide the location of the final progress updates for these goals  OMB’s guidance states that agencies should report the results of progress on their previous APGs in their annual performance reports for fiscal year 2017  Moreover, OMB staff told us that the existing updates on Performance gov for CAP goals, last updated in December 2016, represent the final updates on those goals, although they are not labeled as such on the website  As a result, those interested in progress updates and reported results for the previous priority goals may not know where they will be able to find this information, limiting the transparency and accessibility of those results for decision makers and the public  DATA Act: The DATA Act requires federal agencies to disclose their spending and link this to program activities so that policymakers and the public can more effectively track federal spending  The act has the potential to improve the accuracy and transparency of federal spending information and increase its usefulness for government decision making and oversight  Since the DATA Act became law, OMB and Treasury have taken significant steps to make more complete and accurate federal spending data available  These have included standardizing data element definitions to make it easier to compare different federal agencies’ financial information, and issuing guidance to help agencies submit required data  In May 2017, federal agencies started to report data under the standardized definitions developed under the act  We have made a number of recommendations to address challenges that could affect the consistency and quality of the data  Addressing these recommendations could help ensure that financial data are provided to the public in a transparent and useful manner  For example, in January 2016, we found some standardized data element definitions were imprecise or ambiguous, which could result in inconsistent or potentially misleading reporting  We recommended that OMB provide agencies with additional guidance to address potential issues with the clarity, consistency, and quality of reported data  OMB released guidance in May and November 2016, but in April 2017 we found that additional guidance was needed to help agencies implement certain data definitions to produce data that would be consistent and comparable across agencies  We are in the process of examining the quality of the data that was submitted by agencies in May 2017 and was made available to the public on an early version of the USAspending gov website  We expect to issue the results of this work in fall 2017  Our past work also identified a number of actions agencies need to take to make performance information more transparent  Increasing the accessibility of this information could enhance oversight and accountability of agency performance and results  CAP goals: In May 2016, we found that while selected CAP goal teams were working to develop performance measures to track progress, they were not consistently reporting on their efforts to develop these measures  We recommended that OMB report on Performance gov the actions that CAP goal teams are taking to develop performance measures and quarterly targets to help ensure that measures are aligned with major activities, and ensure that it is possible to track teams’ progress toward establishing measures  While OMB agreed with this recommendation, it did not address it before reporting on the CAP goals was discontinued, as discussed earlier  Customer service standards: As we described earlier, in 2017, an estimated 48 percent of federal managers that indicated they have performance measures for the programs they are involved in also agreed to a great or very great extent that they have customer service performance measures  There has been no statistically significant change relative to our last survey in 2013, or the initial survey in 1997  Relatedly, in October 2014, we reviewed customer service standards at five federal agencies  Customer service standards inform customers about what they have a right to expect when they request services, and the standards should include goals for the quality and timeliness of a service an agency provides to its customers  They should also be easily available to the public so that customers know what to expect, when to expect it, and from whom  In our review of standards at five agencies, however, we found that only Customs and Border Protection had standards that were easily available to the public  We recommended the other four agencies—the United States Forest Service, Federal Student Aid, the National Park Service (NPS), and the Veterans Benefits Administration (VBA)—make their standards more easily accessible to the public  As of July 2017, only VBA had done so  Major management challenges: In June 2016, we found that 14 of the 24 CFO Act agencies did not describe their major management challenges in their performance plans, as required by GPRAMA  Furthermore, 22 of the 24 agencies reviewed did not report complete performance information for each of their major management challenges, including performance goals, milestones, indicators, and planned actions that they have developed to address such challenges  As a result, it was not always transparent what these agencies considered to be their major management challenges or how they planned to resolve these challenges  We recommended that the 22 agencies describe their major management challenges in their agency performance plans and include goals, measures, milestones, and information on planned actions and responsible officials  As of August 2017, 8 agencies—the U S  Agency for International Development, Small Business Administration, Nuclear Regulatory Commission, OPM, National Aeronautics and Space Administration (NASA), and the Departments of Education, State, and Veterans Affairs—had fully implemented our recommendations; the other 14 agencies had not  Quality of performance information: In September 2015, we found that six selected agencies reported limited information on the actions they are taking to ensure the quality of their performance information for selected APGs, as required by GPRAMA  We recommended that all six of the agencies work with OMB to fully report this information  In response, the Department of Homeland Security and NASA described how they ensure reliable performance information is reported to external audiences  As of June 2017, the Departments of Agriculture, Defense, the Interior, and Labor had not yet taken actions to address this recommendation by providing more specific explanations of how they ensure reliable performance information is reported for their APGs  Unnecessary reports: GPRAMA requires that OMB guide an annual review of agencies’ plans and reports for Congress and include in the President’s budget a list of those plans and reports determined to be outdated or duplicative  However, in July 2017, we found that OMB did not implement the report review process on an annual basis, as required  We also found that OMB published the list of agency plans and reports on Performance gov, rather than in the President’s annual budget, where they may be more visible and useful to congressional decision makers and others  Therefore, we recommended that OMB instruct agencies to identify outdated or duplicative reports on an annual basis and submit or reference the list of identified plans and reports with the President’s annual budget  OMB agreed with these recommendations  In July 2017, OMB stated it would include a list of report modification proposals in the President’s fiscal year 2019 budget as required by GPRAMA  For all of the unimplemented recommendations described above, we will continue to monitor agencies’ actions  In addition to providing access to performance and financial information, federal agencies can directly engage and collaborate with citizens, nonprofits, academic institutions, and other levels of government using open innovation strategies  Open innovation involves using various tools and approaches to harness the ideas, expertise, and resources of those outside an organization to address an issue or achieve specific goals  In October 2016, we found that in recent years agencies had frequently used five open innovation strategies—singularly or in combination—to collaborate with citizens and encourage their participation in agency initiatives  (See figure 21 ) Our October 2016 report found that agencies can use these strategies for a variety of purposes  To develop new ideas, solutions to specific problems, or new products: For example, from April 2015 to November 2016, the Department of Energy held a prize competition to create more efficient devices that would double the energy captured from ocean waves  According to the competition’s website, the winning team achieved a five-fold improvement  To enhance collaboration and agency capacity by leveraging external resources, knowledge, and expertise: For example, every 2 years since 2009, the Federal Highway Administration has regularly engaged stakeholders to identify and implement innovative ideas that have measurably improved the execution of highway construction projects  To collect the perspectives and preferences of a broad group of citizens and external stakeholders: For example, the Food and Drug Administration used in-person and online dialogue to engage outside stakeholders in the development of an online platform designed to make key datasets easily accessible to the public  Subsequently, in June 2017, we found that OMB, the Office of Science and Technology Policy (OSTP), and the General Services Administration (GSA) developed resources to support the use of open innovation strategies by federal agencies  These resources included guidance, staff to assist agencies in implementing initiatives, and websites to improve access to relevant information  For example, GSA developed a step-by-step implementation guide, program management team, and website to help agency staff carry out prize competitions and challenges  Agencies have also developed their own resources, including guidance, staff positions, and websites, to reach specific audiences and to provide tailored support for open innovation strategies they use frequently  For example, NASA’s Solve website provides a central location for the public to find the agency’s challenges and citizen science projects, as well as links to relevant resources  We also evaluated key government-wide guidance for the five strategies listed above to determine the extent to which the guidance reflects leading practices for effectively implementing open innovation initiatives  We identified these practices in our October 2016 report  We found that the guidance for each strategy reflected these practices to differing extents, as shown in figure 22  We made 22 recommendations to GSA, OMB, and OSTP to enhance the guidance  GSA and OMB generally agreed with these recommendations and OSTP neither agreed nor disagreed  We will monitor their progress toward implementing these recommendations  GPRAMA provides important tools that can help decision makers better achieve results and address the federal government’s significant and long-standing governance challenges  Although OMB and agencies have made progress in improving implementation of the act over the years, our work has highlighted numerous opportunities for further improvements  In 2017, OMB removed the priority designation of CAP goals and APGs  For those goals, this action stopped related data-driven reviews and quarterly updates of progress on Performance gov until new priority goals are published next year  What OMB considers to be the final results of CAP goals for fiscal years 2014 to 2017 already are on Performance gov (although not labeled as such)  In addition, agencies may report on their former APGs in their annual fiscal year 2017 performance reports  However, Performance gov does not state that it will not be updated or provide the location of the final progress updates for these goals, limiting transparency and its value to the public  OMB has stated its plans to restart implementation of those provisions in February 2018, with the start of a new goal cycle  We believe it is critical for OMB to do so, given the important role those tools play in addressing key governance challenges and the results we have seen in better managing crosscutting areas and driving performance improvements across the government  In addition, OMB has postponed implementation of the federal program inventory  To date, the inventory has only been developed once, in 2013, despite requirements for regular updates to reflect current budget and performance information  OMB has given a variety of reasons for the delays over the past 4 years—most recently, to determine the right strategy to merge implementation of the DATA Act and PMIAA with GPRAMA’s program inventory requirements  Although OMB staff told us that they expect to issue guidance by the end of 2018 to resume implementation of the program inventory requirements, they have not provided more specific time frames and milestones related to the program inventory requirements  Doing so would help agencies prepare for resumed implementation  Moreover, publicly disclosing planned implementation time frames and associated milestones would help ensure that interested stakeholders, such as federal decision makers and the public, are prepared to engage with agencies as they develop and update their program inventories, which in turn could help ensure the inventories meet stakeholders’ needs  A well-developed inventory would provide key program, budget, and performance information in one place to help federal decision makers better understand the federal investment and results in given policy areas, and better identify and manage fragmentation, overlap, and duplication  Information architecture offers one approach to developing an inventory  As OMB determines a strategy for implementing the program inventory and develops its guidance, considering such a systematic approach to planning, organizing, and developing the inventory that centers on maximizing the use and usefulness of information could help it ensure the inventory meets GPRAMA requirements as well as the needs of decision makers and the public  Moreover, such an approach could also help OMB implement our past recommendations related to the program inventory, which are intended to ensure the inventory provides more complete information and is useful to various stakeholders  Our survey of federal managers continues to generally show no improvement in their reported use of performance information in decision making, nor in the employment of practices that can enhance such use  One area where our survey data and past work show promise is through the use of regular, leadership-driven reviews of performance data at agencies, especially when conducted in line with related leading practices  However, GPRAMA only requires these data-driven reviews for APGs, which represent a small subset of goals, both within individual agencies as well as across the government  This is probably why most federal managers were not familiar with the reviews  Identifying and sharing practices for expanding the use of those reviews—such as for additional agency-wide performance goals and at lower levels within agencies—could significantly enhance the use of performance information and drive to better and greater results  We are making the following four recommendations to OMB: The Director of OMB should update Performance gov to explain that quarterly reporting on the fiscal year 2014 through 2017 CAP goals and fiscal year 2016 and 2017 APGs was suspended, and provide the location of final progress updates for these goals  (Recommendation 1) The Director of OMB should revise and publicly issue OMB guidance— through an update to its Circular No  A-11, a memorandum, or other means—to provide time frames and associated milestones for implementing the federal program inventory  (Recommendation 2) The Director of OMB should consider—as OMB determines its strategy for resumed implementation of the federal program inventory—using a systematic approach, such as the information architecture framework, to help ensure that GPRAMA requirements and our past recommendations for the inventory are addressed  (Recommendation 3) The Director of OMB should work with the Performance Improvement Council to identify and share among agencies practices for expanding the use of data-driven performance reviews beyond APGs, such as for other performance goals and at lower levels within agencies, that have led to performance improvements  (Recommendation 4) We provided a draft of this report to the Director of the Office of Management and Budget for review and comment  In comments provided orally and via email, OMB staff agreed with the recommendations in this report  OMB staff also asked us to (1) consider revising the draft title of the report, to better reflect progress in GPRAMA implementation, and (2) clarify our recommendations on issuing guidance for implementing the federal program inventory and expanding the use of data-driven performance reviews, by describing possible actions that could be taken to implement them  We agreed and made revisions accordingly  We are sending copies of this report to interested congressional committees, the Director of the Office of Management and Budget, and other interested parties  This report will also be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report  Key contributors to this report are listed in appendix III  The GPRA Modernization Act (GPRAMA) includes a statutory provision for us to periodically evaluate implementation of the act  Since 2012, we have issued over 30 products in response to this provision; this is the third summary report  This report assesses how implementation of GPRAMA has affected the federal government’s progress in resolving key governance challenges in (1) addressing crosscutting issues, (2) ensuring performance information is useful and used in decision making, (3) aligning daily operations with results, and (4) building a more transparent and open government  We reviewed relevant statutory requirements, related Office of Management and Budget (OMB) guidance, and our recent work related to GPRAMA implementation and the four key governance challenges included in our reporting objectives  Specifically, since our last summary report in September 2015, we examined various aspects of GPRAMA implementation in 12 products that covered 35 agencies, including the 24 agencies covered under the Chief Financial Officers (CFO) Act of 1990, as amended (identified in table 2)  We interviewed OMB and Performance Improvement Council staff to obtain (1) their perspectives on GPRAMA implementation and progress on the four governance challenges, and (2) updates on the status of our past recommendations  We also received updates from other agencies on the status of our past recommendations to them related to GPRAMA implementation  To supplement this review, we administered our periodic survey of federal managers on organizational performance and management issues from November 2016 through March 2017  This survey is comparable to five previous surveys we conducted in 1997, 2000, 2003, 2007, and 2013  We selected a stratified random sample of 4,395 people from a population of approximately 153,779 mid-level and upper-level civilian managers and supervisors working in the 24 executive branch agencies covered by the CFO Act, as shown in table 2  We obtained the sample from the Office of Personnel Management’s (OPM) Enterprise Human Resources Integration (EHRI) database as of September 30, 2015, which was the most recent fiscal year data available at the time  We used file designators indicating performance of managerial and supervisory functions  In reporting survey data, we use the term “government-wide” and the phrases “across the government” or “overall” to refer to the 24 CFO Act executive branch agencies  We use the terms “federal managers” and “managers” to collectively refer to both managers and supervisors  We designed the questionnaire to obtain the observations and perceptions of respondents on various aspects of results-oriented management topics  These topics include the presence and use of performance measures, any hindrances to measuring performance and using performance information, agency climate, and program evaluation use  To assess implementation of GPRAMA, the questionnaire included questions to collect respondents’ views on various provisions of GPRAMA, such as cross-agency priority goals, agency priority goals, and related quarterly performance reviews  Similar to the five previous surveys, the sample was stratified by agency and by whether the manager or supervisor was a member of the Senior Executive Service (SES)  The management levels covered general schedule (GS) or equivalent schedules at levels comparable to GS-13 through GS-15 and career SES or equivalent  Stratifying the sample in this way ensured that the population from which we sampled covered at least 90 percent of all mid- to upper-level managers and supervisors at the departments and agencies we surveyed  Most of the items on the questionnaire were closed-ended, meaning that depending on the particular item, respondents could choose one or more response categories or rate the strength of their perception on a 5-point extent scale ranging from “no extent” to “very great extent ” On most items, respondents also had an option of choosing the response category “no basis to judge/not applicable ” A few items had other options, such as “yes,” “no,” or “do not know,” or a 3-point familiarity scale (“not familiar,” “somewhat familiar,” and “very familiar”)  We asked many of the items on the questionnaire in our earlier surveys, though we introduced a number of new items in 2013, including the sections about GPRAMA and program evaluations  For 2017, we added a new question on use of performance information (question 12) and a new question on program evaluation (question 24)  Before administering the survey, questions were reviewed by our staff, including subject matter experts, a survey specialist, and a research methodologist  We also conducted pretests of the new questions with federal managers in several of the 24 CFO Act agencies  We changed the wording of subquestions or added clarifying examples based on pretester feedback  To administer the survey, we e-mailed managers in the sample to notify them of the survey’s availability on our website and we included instructions on how to access and complete the survey  To follow up with managers in the sample who did not respond to the initial notice, we emailed or called multiple times to encourage survey participation or provide technical assistance, as appropriate  Similar to our last survey, we worked with OPM to obtain the names of the managers and supervisors in our sample, except for those within selected subcomponents whose names were withheld from the EHRI database  Since Foreign Service officials from the Department of State (State) are not in the EHRI database, we drew a sample for that group with the assistance from State  We worked with officials at the Department of Homeland Security (DHS) and the Department of the Treasury (Treasury) to gain access to these individuals to maintain continuity of the population of managers surveyed from previous years  The Department of Justice (DOJ) was concerned about providing identifying information (e g , names, e-mail addresses, and phone numbers) of federal agents to us, so we administered the current survey to DOJ managers in our sample through DOJ officials  To identify the sample of managers whose names were withheld from the EHRI database, we provided DOJ with the last four digits of Social Security numbers, the subcomponent, duty location, and pay grade information  To ensure that DOJ managers received the same survey administration process as the rest of the managers in our sample to the extent possible, we provided DOJ with text for the survey activation and reminder e-mails similar to ones we emailed to managers at other agencies  DOJ administered the survey to these managers and emailed them one reminder to complete the survey  To help determine the reliability and accuracy of the EHRI data elements used to draw our sample of federal managers, we checked the data for reasonableness and the presence of any obvious or potential errors in accuracy and completeness and reviewed past analyses of the reliability of this database  For example, we identified cases where the managers’ names were withheld and contacted OPM to discuss this issue  We also checked the names of the managers in our selected sample provided by OPM with the applicable agency contacts to verify these managers were still employed with the agency  We noted discrepancies when they occurred and excluded them from our population of interest, as applicable  On the basis of these procedures, we believe the data we used from the EHRI database are sufficiently reliable for the purpose of the survey  Of the 4,395 managers selected for the 2017 survey, we found that 388 of the sampled managers had retired, separated, or otherwise left the agency or had some other reason that excluded them from the population of interest  These exclusions included managers that the agency could not locate, and therefore we were unable to request that they participate in the survey  We received usable questionnaires from 2,726 sample respondents, for a weighted response rate of about 67 percent of the remaining eligible sample  The weighted response rate across 23 of the 24 agencies ranged from 57 percent to 82 percent, while DOJ had a weighted response rate of 36 percent  See the supplemental material for each agency’s response rate  We conducted a nonresponse bias analysis using information from the survey and sampling frame as available  The analysis confirmed discrepancies in the tendency to respond to the survey related to agency and SES status  The analysis also revealed some differences in response propensity by age and GS level; however, the direction and magnitude of the differences on these factors were not consistent across agencies or strata  Our data may be subject to bias from unmeasured sources for which we cannot control  Results, and in particular estimates from agencies with low response rates such as DOJ, should be interpreted with caution because these estimates are associated with a higher level of uncertainty  The overall survey results are generalizable to the government-wide population of managers as described above  The responses of each eligible sample member who provided a usable questionnaire were weighted in the analyses to statistically account for all members of the population  All results are subject to some uncertainty or sampling error as well as nonsampling error, including the potential for nonresponse bias as noted above  Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn  The magnitude of sampling error will vary across the particular surveys, groups, or items being compared because we (1) used complex survey designs that differed in the underlying sample sizes, usable sample respondents, and associated variances of estimates, and (2) conducted different types of statistical analyses  For example, the 2000 and 2007 surveys were designed to produce agency-level estimates and had effective sample sizes of 2,510 and 2,943, respectively  However, the 1997 and 2003 surveys were designed to obtain government-wide estimates only, and their sample sizes were 905 and 503, respectively  Consequently, in some instances, a difference of a certain magnitude may be statistically significant  In other instances, depending on the nature of the comparison being made, a difference of equal or even greater magnitude may not achieve statistical significance  Because each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval  This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn  The percentage estimates presented in this report based on our sample for the 2017 survey have 95 percent confidence intervals within plus or minus 5 5 percentage points of the estimate itself, unless otherwise noted  We also note in this report when we are 95 percent confident that changes from 1997 or 2013 relative to 2017 are statistically significant  Online supplemental material shows the questions asked on the survey along with the percentage estimates and associated 95 percent confidence intervals for each item for each agency and government-wide  In a few instances, we report estimates with larger margins of error because we deemed them reliable representations of given findings due to the statistical significance of larger differences between comparison groups  In all cases, we report the applicable margins of error  In addition to sampling errors, the practical difficulties of conducting any survey may also introduce other types of errors, commonly referred to as nonsampling errors  For example, difficulties in how a particular question is interpreted, in the sources of information available to respondents, or in how the data were entered into a database or analyzed can introduce unwanted variability into the survey results  With this survey, we took a number of steps to minimize these nonsampling errors  For example, our staff with subject matter expertise designed the questionnaire in collaboration with our survey specialists  As noted earlier, the new questions added to the survey were pretested to ensure they were relevant and clearly stated  When the data were analyzed, a second independent analyst on our staff verified the analysis programs to ensure the accuracy of the code and the appropriateness of the methods used for the computer-generated analysis  Since this was a web-based survey, respondents entered their answers directly into the electronic questionnaire, thereby eliminating the need to have the data keyed into a database, thus avoiding a source of data entry error  To supplement descriptive analysis of the survey questions, we generated an index to gauge government-wide use of performance information  The index, which was identical to one we reported in 2014, averaged manager’s responses to 11 questions deemed to relate to the concept of performance information use  The index runs from 1 (corresponding to an average value of “to no extent”) to 5 (corresponding to an average value of “to a very great extent”)  We used Cronbach’s alpha to assess the internal consistency of the scale  Our government- wide index score weights each agency’s contribution equally, and provides a relative measure of the use of performance information over time rather than an absolute indicator of the government-wide level of use of performance information  We conducted this performance audit from January 2016 to September 2017 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The Office of Management and Budget (OMB) and agencies have taken some actions to address our recommendations related to implementation of the GPRA Modernization Act of 2010 (GPRAMA); however, the majority of recommendations remain open  Since GPRAMA was enacted in January 2011, we have made 100 recommendations in 18 reports to OMB and agencies aimed at improving the act’s implementation (table 3)  Of those 100, OMB and the agencies have implemented 42 recommendations  Fifty-eight recommendations require additional action  Nearly half (47) of our recommendations are directed to OMB  For the 23 recommendations that OMB has implemented, many represent revisions to guidance to better reflect GPRAMA’s requirements or to enhance implementation  Many of the 24 recommendations to OMB that are not implemented deal with long-standing or complex challenges, on which OMB has taken limited action to date  Of those, we have designated 3 as priorities for OMB to address  Agencies have also taken some action on our recommendations, implementing 19 of the 53 recommendations we have made  The following tables present each of the 100 recommendations along with a summary of actions taken to address it  Tables 4 and 5 provide information about our recommendations to OMB that are implemented and not implemented, respectively  Tables 6 and 7 provide information about our recommendations to other agencies that are implemented and not implemented, respectively  In addition to the above contact, Benjamin T  Licht (Assistant Director) and Shannon Finnegan (Assistant Director) supervised this review and the development of the resulting report  Leah Q  Nash (Assistant Director), Elizabeth Fan (Analyst-in-Charge), and Adam Miles (Analyst-in- Charge) supervised the development and administration of the Federal Managers Survey and the resulting supplemental material  Peter Beck, Valerie Caracelli, Karin Fangman, Steven Flint, Robert Gebhart, Ricky Harrison Jr , John Hussey, Jill Lacey, Won Lee, Krista Loose, Meredith Moles, Anna Maria Ortiz, Steven Putansu, Alan Rozzi, Cindy Saunders, Stephanie Shipman, Shane Spencer, Andrew J  Stephens, and Brian Wanlass also made key contributions  Ann Czapiewski and Donna Miller developed the graphics for this report  John Ahern, Divya Bali, Jeff DeMarco, Alexandra Edwards, Ellen Grady, Jyoti Gupta, Erinn L  Sauer, and Katherine Wulff verified the information presented in this report  Managing for Results: Implementation of GPRA Modernization Act Has Yielded Mixed Progress in Addressing Pressing Governance Challenges  GAO-15-819  Washington, D C : September 30, 2015  Managing For Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges  GAO-13-518  Washington, D C : June 26, 2013  Supplemental Material for GAO-17-775: 2017 Survey of Federal Managers on Organizational Performance and Management Issues  GAO-17-776SP  Washington, D C : September 29, 2017  Program Evaluation: Annual Agency-wide Plans Could Enhance Leadership Support for Program Evaluations  GAO-17-743  Washington, D C : September 29, 2017  Managing for Results: Agencies’ Trends in the Use of Performance Information to Make Decisions  GAO-14-747  Washington, D C : September 26, 2014  Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges  GAO-13-518  Washington, D C : June 26, 2013  Managing for Results: 2013 Federal Managers Survey on Organizational Performance and Management Issues, an E-supplement to GAO-13-518  GAO-13-519SP  Washington, D C : June 26, 2013  Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making  GAO-13-570  Washington, D C : June 26, 2013  Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results  GAO-08-1026T  Washington, D C : July 24, 2008  Government Performance: 2007 Federal Managers Survey on Performance and Management Issues, an E-supplement to GAO-08-1026T  GAO-08-1036SP  Washington, D C : July 24, 2008  Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results  GAO-04-38  Washington, D C : March 10, 2004  Managing for Results: Federal Managers’ Views on Key Management Issues Vary Widely Across Agencies  GAO-01-592  Washington, D C : May 25, 2001  Managing for Results: Federal Managers’ Views Show Need for Ensuring Top Leadership Skills  GAO-01-127  Washington, D C : October 20, 2000  The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven  GAO/GGD-97-109  Washington, D C : June 2, 1997  Federal Programs: Information Architecture Offers a Potential Approach for Inventory Development  GAO-17-739  Washington, D C : September 28, 2017  Managing for Results: Selected Agencies’ Experiences in Implementing Strategic Reviews  GAO-17-740R  Washington, D C : September 7, 2017  Federal Reports: OMB and Agencies Should More Fully Implement the Process to Streamline Reporting Requirements  GAO-17-616  Washington, D C : July 14, 2017  Open Innovation: Executive Branch Developed Resources to Support Implementation, but Guidance Could Better Reflect Leading Practices  GAO-17-507  Washington, D C : June 8, 2017  Performance Partnerships: Agencies Need to Better Identify Resource Contributions to Sustain Disconnected Youth Pilot Programs and Data to Assess Pilot Results  GAO-17-208  Washington, D C : April 18, 2017  Open Innovation: Practices to Engage Citizens and Effectively Implement Federal Initiatives  GAO-17-14  Washington, D C : October 13, 2016  Tiered Evidence Grants: Opportunities Exist to Share Lessons from Early Implementation and Inform Future Federal Efforts  GAO-16-818  Washington, D C : September 21, 2016  Performance gov: Long-Term Strategy Needed to Improve Website Usability  GAO-16-693  Washington, D C : August 30, 2016  Tax Expenditures: Opportunities Exist to Use Budgeting and Agency Performance Processes to Increase Oversight  GAO-16-622  Washington, D C : July 7, 2016  Managing for Results: Agencies Need to Fully Identify and Report Major Management Challenges and Actions to Resolve them in their Agency Performance Plans  GAO-16-510  Washington, D C : June 15, 2016  Managing for Results: OMB Improved Implementation of Cross-Agency Priority Goals, But Could Be More Transparent About Measuring Progress  GAO-16-509  Washington, D C : May 20, 2016  Managing for Results: Greater Transparency Needed in Public Reporting on the Quality of Performance Information for Selected Agencies’ Priority Goals  GAO-15-788  Washington, D C : September 10, 2015  Pay for Success: Collaboration among Federal Agencies Would Be Helpful as Governments Explore New Financing Mechanisms  GAO-15-646  Washington, D C : September 9, 2015  Managing for Results: Practices for Effective Agency Strategic Reviews  GAO-15-602  Washington, D C : July 29, 2015  Managing for Results: Agencies Report Positive Effects of Data-Driven Reviews on Performance but Some Should Strengthen Practices  GAO-15-579  Washington, D C : July 7, 2015  Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity  GAO-15-25  Washington, D C : November 13, 2014  Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories  GAO-15-83  Washington, D C : October 31, 2014  Managing for Results: Selected Agencies Need to Take Additional Efforts to Improve Customer Service  GAO-15-84  Washington, D C : October 24, 2014  Managing for Results: Enhanced Goal Leader Accountability and Collaboration Could Further Improve Agency Performance  GAO-14-639  Washington, D C : July 22, 2014  Managing for Results: OMB Should Strengthen Reviews of Cross-Agency Goals  GAO-14-526  Washington, D C : June 10, 2014  Managing for Results: Implementation Approaches Used to Enhance Collaboration in Interagency Groups  GAO-14-220  Washington, D C : February 14, 2014  Managing for Results: Leading Practices Should Guide the Continued Development of Performance gov  GAO-13-517  Washington, D C : June 6, 2013  Managing for Results: Agencies Should More Fully Develop Priority Goals under the GPRA Modernization Act  GAO-13-174  Washington, D C : April 19, 2013  Managing for Results: Agencies Have Elevated Performance Management Roles, but Additional Training Is Needed  GAO-13-356  Washington, D C : April 16, 2013  Managing for Results: Data-Driven Performance Reviews Show Promise But Agencies Should Explore How to Involve Other Relevant Agencies  GAO-13-228  Washington, D C : February 27, 2013  Managing for Results: A Guide for Using the GPRA Modernization Act to Help Inform Congressional Decision Making  GAO-12-621SP  Washington, D C : June 15, 2012  Managing for Results: GAO’s Work Related to the Interim Crosscutting Priority Goals under the GPRA Modernization Act  GAO-12-620R  Washington, D C : May 31, 2012  Managing for Results: Opportunities for Congress to Address Government Performance Issues  GAO-12-215R  Washington, D C : December 9, 2011 


Report 2: Cybersecurity incidents continue to impact federal entities and the information they maintain  According to OMB’s 2018 annual FISMA report to Congress, agencies reported 35,277 information security incidents to DHS’s U S  Computer Emergency Readiness Team (US-CERT) in fiscal year 2017  As shown in figure 1, these incidents involved threat vectors, such as web-based attacks, phishing attacks, and the loss or theft of computer equipment, among others  These incidents and others like them can pose a serious challenge to economic, national, and personal privacy and security  The following examples highlight the impact of such incidents: In March 2018, the Department of Justice reported that it had indicted nine Iranians for conducting a massive cybersecurity theft campaign on behalf of the Islamic Revolutionary Guard Corps  According to the department, the Iranians allegedly stole more than 31 terabytes of documents and data from more than 140 American universities, 30 U S  companies, and 5 federal government agencies, among other entities  In March 2018, a joint alert from DHS and the Federal Bureau of Investigation stated that, since at least March 2016, Russian government actors had targeted U S  government entities and critical infrastructure sectors, including the energy, nuclear, water, aviation, and critical manufacturing sectors  In June 2015, the Office of Personnel Management reported that an intrusion into its systems had affected the personnel records of about 4 2 million current and former federal employees  Then, in July 2015, the agency reported that a separate but related incident had compromised its systems and the files related to background investigations for at least 21 5 million individuals  The federal approach and strategy for securing information systems is prescribed by federal law and policy  FISMA sets requirements for effectively securing federal systems and information  In addition, the Federal Cybersecurity Enhancement Act of 2015 requires protecting federal networks through the use of federal intrusion prevention and detection capabilities  Further, Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure, directs agencies to manage cybersecurity risks to the federal enterprise by, among other things, using the NIST Framework for Improving Critical Infrastructure Cybersecurity (cybersecurity framework)  FISMA was enacted to improve federal cybersecurity and clarify government-wide responsibilities  The law is intended to provide for improved oversight of federal agencies’ information security programs  Specifically, the law provides a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets  The law is also intended to ensure the effective oversight of information security risks, including those throughout civilian, national security, and law enforcement agencies  FISMA assigns OMB and DHS oversight roles in ensuring federal agencies’ compliance with the law  Among other things, FISMA requires OMB to develop and oversee the implementation of policies, principles, standards, and guidelines on information security in federal agencies, except with regard to national security systems  The law also assigns OMB the responsibility of requiring agencies to identify and provide information security protections commensurate with assessments of risk to their information and information systems  The law further requires DHS to administer the implementation of agency information security policies and practices for non-national security information systems, in consultation with OMB, by developing, issuing, and overseeing implementation of binding operational directives; monitoring agency implementation of information security policies and practices; and convening meetings with senior agency officials to help ensure their effective implementation of information security policies and practices, among other things  FISMA assigned to NIST the responsibility for developing standards and guidelines that include minimum information security requirements  To this end, NIST has issued several publications to provide guidance for agencies in implementing an information security program  For example, NIST Special Publication (SP) 800-53 provides guidance to agencies on the selection and implementation of information security and privacy controls for systems  FISMA also assigns to the head of each executive branch agency, responsibility for providing information security protections commensurate with the risk and magnitude of harm resulting from unauthorized access, use, disclosure, disruption, modification, or destruction of information systems used or operated by an agency or by a contractor of an agency or other organization on behalf of an agency  The law also delegates to the agency chief information officer (CIO), or comparable official, the authority to ensure compliance with FISMA requirements  The CIO is responsible for designating a senior agency information security officer whose primary duty is information security  In addition, the law requires agencies to develop, document, and implement an agency-wide information security program to secure federal information systems  Specifically, these information security programs are to provide risk-based protections for the information and information systems that support the operations and assets of the agency  Further, FISMA requires agencies to comply with DHS binding operational directives, OMB policies and procedures, and NIST federal information processing standards  FISMA also has reporting requirements for OMB and federal agencies  Specifically, OMB is to report annually, in consultation with DHS, on the effectiveness of agency information security policies and practices, including a summary of major agency information security incidents and an assessment of agency compliance with NIST standards  Further, the law requires agencies to report annually to OMB, DHS, certain congressional committees, and the Comptroller General of the United States on the adequacy and effectiveness of their information security policies, procedures, and practices, as well as their compliance with FISMA  The Federal Cybersecurity Enhancement Act of 2015, among other things, sets forth authority for enhancing federal intrusion prevention and detection capabilities among federal entities  The act contains several provisions for DHS and OMB  Specifically, the act requires that DHS deploy, operate, and maintain capabilities to prevent and detect cybersecurity risks in network traffic traveling to or from an agency’s information system  DHS is to make these capabilities available for use by any agency  In addition, the act requires DHS to improve intrusion detection and prevention capabilities, as appropriate, by regularly deploying new technologies and modifying existing technologies  The act also requires OMB and DHS, in consultation with appropriate agencies, to review and update government-wide policies and programs to ensure appropriate prioritization and use of network security monitoring tools within agency networks, and to brief appropriate congressional committees  In May 2017, the President signed Executive Order 13800, which sets policy for managing cybersecurity risk as an executive branch enterprise  Specifically, it outlines actions to enhance cybersecurity across federal agencies and critical infrastructure to improve the nation’s cyber posture and capabilities against cybersecurity threats  To this end, the order states that the President will hold executive agency heads accountable for managing agency-wide cybersecurity risk and directs each executive agency to use the NIST cybersecurity framework to manage those risks  The cybersecurity framework, which provides guidance for cybersecurity activities, is based on five core security functions: Identify: Develop an organizational understanding to manage cybersecurity risk to systems, people, assets, data, and capabilities  Protect: Develop and implement appropriate safeguards to ensure delivery of critical services  Detect: Develop and implement appropriate activities to identify the occurrence of a cybersecurity event  Respond: Develop and implement appropriate activities to take action regarding a detected cybersecurity incident  Recover: Develop and implement appropriate activities to maintain plans for resilience and to restore capabilities or services that were impaired due to a cybersecurity incident  According to NIST, these five functions should be performed concurrently and continuously to address cybersecurity risk  In addition, when considered together, they provide a high-level, strategic view of the life cycle of an organization’s management of cybersecurity risk  Within the five functions are 23 categories and 108 subcategories that include controls for achieving the intent of each function  Appendix II provides a description of the cybersecurity framework categories and subcategories of controls  In February 2013, we reported that the government had issued a variety of strategy-related documents that addressed priorities for enhancing cybersecurity within the federal government, as well as for encouraging improvements in the cybersecurity of critical infrastructure within the private sector  However, we noted that no overarching cybersecurity strategy had been developed that articulated priority actions, assigned responsibilities for performing them, and set time frames for their completion  Accordingly, we recommended that the White House Cybersecurity Coordinator in the Executive Office of the President develop an overarching federal cybersecurity strategy that included all key elements of the desirable characteristics of a national strategy  These characteristics would include, among other things, milestones and performance measures for major activities to address stated priorities; cost and resources needed to accomplish stated priorities; and specific roles and responsibilities of federal organizations related to the strategy’s stated priorities  Since that time, the executive branch has made progress toward outlining a federal strategy for confronting cyber threats  For example, in September 2018, we reported that recent executive branch initiatives that identify cybersecurity priorities for the federal government provide a good foundation toward establishing a more comprehensive strategy  Nevertheless, we pointed out that additional efforts were needed to address all of the desirable characteristics of a national strategy that we recommended  Specifically, recently issued executive branch strategy documents did not include key elements of desirable characteristics that can enhance the usefulness of a national strategy as guidance for decision makers in allocating resources, defining policies, and helping to ensure accountability  For example, these strategy documents did not generally include: milestones and performance measures to gauge results; resources needed to carry out the goals and objectives; and clearly defined roles and responsibilities for key agencies, such as DHS, the Department of Defense, and OMB  Ultimately, we determined that a more clearly defined, coordinated, and comprehensive approach to planning and executing an overall strategy would likely lead to significant progress in furthering strategic goals and lessening persistent weaknesses  Subsequent to our September 2018 report, the President issued the National Cyber Strategy on September 20, 2018  The strategy builds upon Executive Order 13800 and describes actions that federal agencies and the administration are to take to, among other things, secure federal information systems  For example, the strategy states that the administration is expected to further enable DHS to secure federal department and agency networks, to include ensuring that DHS has appropriate access to agency information systems for cybersecurity purposes and can take and direct action to safeguard systems  In addition, the strategy states that the administration plans to continue with its existing efforts underway to transition agencies to shared services and infrastructure and that DHS is to have appropriate visibility into those services and infrastructure to improve cybersecurity posture  DHS’s Network Security Deployment (NSD) division manages cybersecurity programs that are intended to improve the cybersecurity posture of the federal government  Among these programs, NCPS provides a capability to detect and prevent potentially malicious network traffic from entering agencies’ networks  In addition, the Continuous Diagnostics and Mitigation (CDM) program provides tools to agencies intended to identify and resolve cyber vulnerabilities on an ongoing basis  Operated by DHS’s US-CERT, NCPS is intended to detect and prevent cyber intrusions into agency networks, analyze network data for trends and anomalous data, and share information with agencies on cyber threats and incidents  Deployed in stages, this system, operationally known as EINSTEIN, has provided increasing capabilities to detect and prevent potential cyberattacks involving the network traffic entering or exiting the networks of participating federal agencies  Table 1 provides an overview of the EINSTEIN deployment stages to date  In January 2016, we reported the projected total life-cycle cost of the program was approximately $5 7 billion through fiscal year 2018  In addition, according to the Federal CIO, Congress appropriated $468 million in fiscal year 2017 and $402 million in fiscal year 2018 for NCPS  In that report, we also noted that NCPS was partially, but not fully, meeting most of its stated system objectives  Although the system’s intrusion detection capabilities provided the ability to detect known patterns of malicious activity on agency networks, it was limited in its capabilities to identify potential threats using anomaly-based detection  We also reported that although DHS had developed metrics for measuring the performance of NCPS, the metrics did not gauge the quality, accuracy, or effectiveness of the system’s intrusion detection and prevention capabilities  The department had also identified needs for future capabilities, but had not defined requirements for the capability to detect threats entering and exiting cloud service providers  Further, DHS had not considered specific vulnerability information for agency information systems in making risk- based decisions about future intrusion prevention capabilities  Accordingly, we made nine recommendations to DHS to, among other things, enhance the NCPS capabilities for meeting its objectives and better define requirements for future capabilities  DHS agreed with each of our nine recommendations and indicated that it would take steps to address them  DHS’s CDM program provides federal agencies with tools and services that have the intended capability to automate network monitoring, correlate and analyze security-related information, and enhance risk- based decision making at agency and government-wide levels  These tools include sensors that perform automated scans or searches for known cyber vulnerabilities, the results of which can feed into a dashboard that, at an agency level, is intended to alert network managers and enable the agency to allocate resources based on the risk  Summary data from each participating agency’s dashboard is expected to be transmitted to the Federal Dashboard where the data can be used to inform decisions about cybersecurity risks across the federal government  There are four phases of CDM implementation: Phase 1—involves deploying products to automate hardware and software asset management, configuration settings, and common vulnerability management capabilities  According to the Cybersecurity Strategy and Implementation Plan, DHS purchased phase 1 tools and integration services for all participating agencies in fiscal year 2015  DHS plans to have all phase 1 tools deployed at participating agencies by the end of the second quarter of fiscal year 2019  Phase 2—intends to address privilege management and infrastructure integrity by allowing agencies to monitor users on their networks and to detect whether users are engaging in unauthorized activity  According to the Cybersecurity Strategy and Implementation Plan, DHS was to provide agencies with additional phase 2 capabilities throughout fiscal year 2016, with the full suite of CDM phase 2 capabilities delivered by the end of that fiscal year  However, according to the OMB FISMA Annual Report to Congress for Fiscal Year 2017, the CDM program began deploying Phase 2 tools and sensors during fiscal year 2017  DHS plans to have all phase 2 tools deployed at participating agencies by the end of fiscal year 2019  Phase 3—includes detection capabilities that are intended to assess agency network activity and identify any anomalies that may indicate a cybersecurity compromise  Full operating capability for phases 1, 2, and 3 is planned to be achieved by the end of fiscal year 2022  Phase 4—intends to provide tools to (1) protect data at rest, in transit, and in use; (2) prevent loss of data; and (3) manage and mitigate data breaches  According to CDM program officials, phase 4 has not been approved and no tools have been selected  An approach for protecting systems against cybersecurity compromise is for federal agencies to build successive layers of defense mechanisms at strategic points in their information technology infrastructures  This approach, commonly referred to as defense in depth, entails implementing a series of protective mechanisms so that if one mechanism fails to detect and prevent an attack, another will provide a backup defense  By utilizing defense in depth, federal agencies can reduce the risk of a successful cyberattack by implementing intrusion detection and prevention capabilities  NIST has developed guidelines for protecting agency information systems using intrusion detection and prevention capabilities  For example, NIST SP 800-53 recommends that agencies strategically deploy capabilities and perform monitoring of their systems to include observation of events occurring on their network and at the external boundary of their network  In addition, NIST SP 800-94 provides agencies with guidance in designing, implementing, configuring, securing, monitoring, and maintaining such capabilities  As part of their defense-in-depth approach and, as recommended by the NIST guidelines, agencies can deploy the following list of capabilities, among others, on their networks to detect and prevent an attack: Protecting email from intrusions: According to OMB, email, by way of phishing attacks, remains one of the most common threat vectors across the government  Methods for protecting email include encryption, false email alerts, and anti-spear-phishing training  Monitoring cloud services: Cloud vendors provide services to agencies, including Software as a Service, Platform as a Service, and Infrastructure as a Service  As agencies increasingly rely on cloud services, monitoring traffic to and from their cloud service providers helps to ensure that agencies detect malicious traffic  Using host-based intrusion prevention: Host-based intrusion prevention systems provide defense at an individual system or device level by protecting against malicious activities  Host-based capabilities include memory-based protection and application whitelisting  Monitoring external and internal traffic: Agencies can monitor external and internal traffic, including: encrypted traffic, traffic between workstations and servers on the network, and direct connections to outside entities such as universities  Monitoring traffic helps to ensure that agencies detect malicious activity  Using security information and event management: A security information and event management capability produces real-time alerts and notifications of significant security events  Security alerts and notifications can provide the agency with better situational awareness regarding possible intrusion activity  According to inspectors general, agency CIOs, and OMB reports on federal information security practices, many agencies were not effectively implementing the federal government’s approach and strategy to securing information systems as of fiscal year 2017  Agencies’ inspectors general determined that most of the 23 civilian CFO Act agencies did not have effective agency-wide information security programs  They also reported that agencies did not have effective information security controls in place, leading to deficiencies in internal control over financial reporting  In addition, the CIOs demonstrated that, during fiscal years 2016 and 2017, most agencies had not met all targets for the cybersecurity CAP goal for improving cybersecurity performance  Further, based on FISMA metrics reported for fiscal year 2017, OMB determined that 13 of the 23 agencies were managing risks to their enterprise, while the other 10 agencies were at risk of ineffectively identifying, protecting, detecting, responding to, and if necessary, recovering from cyber intrusions  Figure 2 summarizes agencies’ efforts to implement the government’s approach and strategy for securing information systems as of fiscal year 2017  Appendix III includes a table that provides an additional overview of the effectiveness of each agency’s implementation of the government’s approach and strategy to securing information systems  Inspectors general determined that more than half of the 23 civilian CFO Act agencies did not have effective agency-wide information security programs as of fiscal year 2017  Further, in agency financial statement audit reports for fiscal year 2017, inspectors general reported that, despite improvements being made in information security practices, most of the civilian CFO Act agencies continued to exhibit deficiencies in information security controls  As a result of these deficiencies, inspectors general reported material weaknesses or significant deficiencies in internal control over financial reporting  FISMA requires inspectors general to determine the effectiveness of their respective agencies’ information security programs  To do so, FISMA reporting instructions direct inspectors general to provide a maturity rating for agency information security policies, procedures, and practices related to the five core security functions established in the NIST cybersecurity framework, as well as for the agency-wide information security program  The ratings used to evaluate the effectiveness of agencies’ information security programs are based on a five-level maturity model, as described in table 2  According to this maturity model, Level 4 (managed and measurable) represents an effective level of security  Therefore, if an inspector general rates the agency’s information security program at Level 4 or Level 5, then that agency is considered to have an effective information security program  For fiscal year 2017, the inspectors general for 6 of the 23 civilian CFO Act agencies reported that their agencies had an effective agency-wide information security program  More specifically, for the 5 core security functions, most inspectors general reported that their agency was at Level 3 (consistently implemented) for the identify, protect, and recover functions, and at Level 2 (defined) for the detect and respond functions, as shown in figure 3  Inspectors general report on the effectiveness of agencies’ information security controls as part of the annual audits of the agencies’ financial statements  The reports resulting from these audits include a description of information security control deficiencies related to the five major control categories defined by the Federal Information System Controls Audit Manual (FISCAM)—access controls, configuration management, segregation of duties, contingency planning, and security management  The reports also identify the inspectors general’s designation of information security as a significant deficiency or material weakness in internal control over financial reporting systems  For fiscal year 2017, inspectors general continued to identify information security control deficiencies in each of the five major control categories across the 23 civilian CFO Act agencies  The number of agencies with deficiencies in the access control and contingency planning information security control categories decreased between fiscal years 2016 and 2017, according to the inspectors general  Nevertheless, the inspectors general reported that agencies continued to exhibit deficiencies in these two control categories  In addition, the number of agencies with deficiencies in the security management and segregation of duties control categories increased from the prior year  The number of agencies reported as having deficiencies in the configuration management control category remained the same  Figure 4 shows the number of agencies that reported deficiencies in each of the information security control categories for fiscal years 2016 and 2017  Overall, inspectors general for the 23 civilian CFO Act agencies reported progress in agencies’ information security practices for fiscal year 2017  Specifically, during that time, 17 inspectors general designated information security as either a significant deficiency (11) or material weakness (6) in internal control over financial reporting systems for their agencies  This is a decrease from the previous fiscal year when 19 inspectors general designated information security as a significant deficiency (12) or material weakness (7)  Reporting instructions contained in the fiscal year 2017 FISMA metrics directed CIOs to assess their agencies’ progress toward achieving outcomes that strengthen federal cybersecurity  To do this, CIOs evaluated their agencies’ performance in reaching targets for specific FISMA reporting metrics  According to the reporting instructions, certain metrics were selected to represent the administration’s cybersecurity CAP goal  These selected metrics allowed CIOs to evaluate their agencies progress in meeting targets for that goal  The cybersecurity CAP goal for fiscal years 2015 through 2017 was to improve cybersecurity performance by having an ongoing awareness of information security, vulnerabilities, and threats impacting the operating information environment; ensuring that only authorized users have access to resources and information; and implementing technologies and processes that reduce the risk of malware  The cybersecurity CAP goal consisted of three priority areas with a total of nine performance indicators  Each of the nine performance indicators had an expected level of performance, or target, for implementation  Table 3 shows the three priority areas and related performance indicators and targets of the cybersecurity CAP goal for fiscal years 2015 through 2017  According to agency CIO assessments for fiscal year 2017, 6 of the 23 agencies met all 9 targets for the cybersecurity CAP goal  More specifically, 8 agencies met all four targets for the information security continuous 16 agencies met the two targets for the identity, credential, and access management priority area; and 17 agencies met all three targets for the anti-phishing and malware defense priority area  In addition, CIOs reported that agencies were making progress in meeting the targets for the nine performance indicators for fiscal years 2016 and 2017, with increases in the number of agencies meeting the targets within each of the three priority areas  However, although the number of agencies that met the targets in individual priority areas showed a net increase, not all agencies maintained their status  For example, the CIO for one agency reported meeting all three targets for the anti-phishing and malware defense priority area in fiscal year 2016, but reported that the agency only met two of the three targets in fiscal year 2017  Figure 5 shows the number of agencies that reported meeting each of the targets within the individual cybersecurity CAP goal priority areas for fiscal years 2016 and 2017  Although the CIOs for only six agencies reported meeting each of the targets associated with all nine performance indicators for the three cybersecurity CAP goal priority areas, the CIOs at an additional eight agencies reported meeting each target for two of the three priority areas  Specifically, one CIO reported that its agency met each of the targets for the (1) information security continuous monitoring and (2) identity, credential, and access management priority areas; another CIO reported that its agency met each of the targets for the (1) information security continuous monitoring and (2) anti-phishing and malware defense priority areas; and the CIOs at six other agencies met each of the targets for the (1) identity, credential, and access management and (2) anti-phishing and malware defense priority areas  In fiscal year 2018, the President’s Management Agenda replaced the three cybersecurity-focused CAP goal priority areas with updated performance indicators, most of which are to be met by 2020: 1  the manage asset security priority area is similar to the information security continuous monitoring priority area from the previous CAP goal and has a focus on understanding the assets and users on agency networks  In addition to hardware asset and software asset management, this priority area includes performance indicators for authorization and mobile device management  2  the limit personnel access priority area focuses on issues of access management  This area includes performance indicators for using automated access management and managing access for privileged network and high-impact system users  The privileged network access management performance indicator is a continuation of the identity, credential, and access management priority area of the previous cybersecurity CAP goal  Therefore, agencies are expected to complete this metric by the end of the fiscal year 2018 FISMA reporting year  3  the protect networks and data priority area, which is similar to the anti-phishing and malware defense priority area from the previous cybersecurity CAP goal, has three new performance indicators: intrusion detection and prevention, exfiltration and enhanced defenses, and data protection  Appendix IV describes the updated cybersecurity-focused CAP priority areas and performance indicators in more detail  In Executive Order 13800, the President directed OMB, in coordination with DHS, to assess and report to the executive branch on the sufficiency and appropriateness of federal agencies’ processes for managing cybersecurity risks  For these risk management assessments, OMB leveraged the FISMA metrics reported by agency CIOs and inspectors general for fiscal year 2017  The metrics addressed domains that correspond with the five core security functions identified in the cybersecurity framework  Table 4 lists these domains and their relationship to the core functions  Based on OMB’s evaluation of these domains, agency risk management processes related to the five core security functions and overall agency enterprise fell into one of the following three rating categories: managing risk: required cybersecurity policies, procedures, and tools are in use and the agency actively manages cybersecurity risks; at risk: some essential policies, processes, and tools are in place to mitigate overall cybersecurity risk, but significant gaps remain; and high risk: key fundamental cybersecurity policies, processes, and tools are either not in place or not deployed sufficiently  For fiscal year 2017, OMB reported that not all agencies were managing risk  When considering each of the five core security functions, OMB reported that most of the 23 agencies were at risk or at high risk with regard to the identify and protect core security functions  Less than half of the 23 agencies were at risk with regard to the detect, respond, and recover core security functions  Overall, OMB determined that 13 agencies were managing risk and that the remaining 10 agencies were at risk of not effectively identifying, protecting, detecting, responding to, and if necessary, recovering from cyber intrusions  Figure 6 shows OMB’s risk management assessment ratings by core security function across the 23 agencies for fiscal year 2017  DHS and OMB, as required by law and policy, have taken various actions to facilitate the agencies’ use of intrusion detection and prevention capabilities to secure federal systems  For example, DHS has developed an intrusion assessment plan, deployed NCPS to offer intrusion detection and prevention capabilities to agencies, and is providing tools and services to agencies to monitor their networks through its CDM program  However, NCPS still had limitations in detecting certain types of traffic and agencies were not sending all appropriate traffic through the system  Further, CDM was behind at meeting planned implementation dates, and agencies have requested additional training and guidance for these services  OMB has taken steps to improve upon agencies’ capabilities, but has not completed a policy and strategy to do so, or fully reported on its assessment of agencies’ capabilities  The Federal Cybersecurity Enhancement Act of 2015 requires DHS, in coordination with OMB, to develop and implement an intrusion assessment plan to proactively detect, identify, and remove intruders in agency information systems on a routine basis  The act also requires that the plan be updated, as necessary  In December 2016, DHS documented its Intrusion Assessment Plan  In the plan, DHS outlined tools, platforms, resources, and ongoing work that the department provides, and that are intended to help agencies detect, identify, and remove intruders on their networks and systems  The intrusion assessment plan also outlines a defense-in-depth strategy, which utilizes multiple layers of cybersecurity and deploys multiple capabilities in combination, to secure agencies’ networks and information systems  For example, the plan calls for DHS to implement NCPS to provide a perimeter defense for the networks of federal civilian executive branch agencies, while the agencies are to deploy their own intrusion detection and prevention capabilities inside their networks  DHS submitted its intrusion assessment plan to OMB in January 2017  The Federal Cybersecurity Enhancement Act of 2015 also requires DHS to deploy, operate, and maintain a capability to detect cybersecurity risks and prevent network traffic associated with such risks from transiting to or from an agency information system  In addition, the act requires that DHS make regular improvements to intrusion detection and prevention capabilities by deploying new technologies and modifying existing technologies  Further, the act requires agencies to use this capability on all information traveling between their information systems and any information system other than an agency information system  DHS developed NCPS, operationally known as EINSTEIN, to provide the capabilities to detect and prevent potentially malicious network traffic from entering agency networks  Consistent with recommendations we made to DHS in January 2016, DHS has taken actions to improve these capabilities and has other actions underway  For example, the department determined that enhancing NCPS’s current intrusion detection approach to include functionality that would detect deviations from normal network behavior baselines would be feasible  In addition, according to DHS officials, the department was operationalizing functionality intended to identify malicious activity in network traffic otherwise missed by signature-based methods  determined that developing enhancements to current intrusion detection capabilities to facilitate the scanning of Internet Protocol Version 6 (IPv6) traffic would be feasible  According to DHS officials, the department has developed plans to fully support IPv6 for several of its NCPS intrusion detection capabilities  Further, the department has developed implementation schedules and begun roll-out of the enhancements  updated the tool it uses to manage and deploy intrusion detection signatures to include a mechanism to clearly link signatures to publicly available, open-source information  developed clearly defined requirements for detecting threats on agency internal networks and at cloud service providers to help better ensure effective support of information security activities  According to DHS officials, the department was also continuing pilot activities with cloud service providers to enhance protections of agency assets  developed processes and procedures for using vulnerability information, such as data from the CDM program as it becomes available, to help ensure the department is using a risk-based approach for the selection/development of future NCPS intrusion prevention capabilities  Nevertheless, NCPS continues to have known limitations in its ability to identify potential threats  For example: NCPS does not have the ability to effectively detect intrusions across multiple types of traffic  Specifically, DHS determined that developing enhancements to current intrusion detection capabilities to facilitate the scanning of traffic related to supervisory control and data acquisition (SCADA) control systems would not be feasible  However, according to DHS officials, the department is exploring capabilities that are intended to provide critical, cross-sector, real-time visibility into critical infrastructure companies that utilize SCADA systems  In addition, DHS determined that the scanning of encrypted traffic would not be feasible  Nevertheless, according to its officials, the department performed research on potential architectural, technical, and policy mitigation strategies that could provide both the protection and situational awareness for encrypted traffic  The department has actions under way to continue its research in this area  DHS does not always explicitly ask agencies for feedback or confirmation of receipt of NCPS-related notification  While the department had drafted a standard operating procedure related to its incident notification process, the policy did not instruct DHS analysts specifically to include a solicitation of feedback from agencies within the notification  Further, US-CERT could not provide any information regarding the timetable for when these procedures would take effect  Metrics for NCPS, as provided by DHS, do not provide information about how well the system is enhancing government information security or the quality, efficiency and accuracy of supporting actions  Without the deployment of comprehensive measures, DHS cannot appropriately articulate the value provided by NCPS  While the department had taken actions to develop new measures, these measures did not provide a qualitative or quantitative assessment of the system’s ability to fulfill the system’s objectives  NSD did not provide guidance to agencies on how to securely route their information to their Internet service providers  Without providing network routing guidance, NSD has no assurance that the traffic it sees constitutes all or only a subset of the traffic the customer agencies intend to send  As shown in table 5, as of October 2018, the department had implemented five of the nine recommendations and was in the process of implementing the remainder  However, until DHS completes implementation of the remaining recommendations, the effectiveness of NCPS’s intrusion detection and prevention capabilities may be hindered  In addition, the 23 civilian CFO Act agencies had implemented NCPS capabilities to varying degrees  In a March 2018 report, OMB reported that 21 (about 91 percent) of the 23 agencies had implemented the first two iterations of the NCPS capabilities  In addition, 15 (about 65 percent) of the 23 agencies had implemented all three NCPS capabilities, as shown in table 6 below  However, agencies did not route all network traffic for all information traveling between their information systems and any information system other than an agency information system through NCPS sensors  For example, officials at 13 of 23 agencies stated that not all of their agency external network traffic flowed through NCPS  To illustrate, officials at one agency estimated that 20 percent of their external network traffic did not flow through the system  In addition, 4 of the agencies in our review previously cited several challenges in routing all of their traffic through NCPS intrusion detection sensors, including capacity limitations of the sensors, agreements with external business partners that use direct network connections, interagency network connections that do not route through Internet gateways, use of encrypted communications mechanisms, and backup network circuits that are not used regularly  NSD officials stated that agencies are responsible for routing their traffic to the intrusion detection sensors, and DHS does not have a role in that aspect of NCPS implementation  As a result, potential cyberattacks may not be detected or prevented for a portion of the external traffic at federal agencies  As noted above, we previously recommended that DHS work with agencies to better ensure the complete routing of information to NCPS sensors  The Federal Cybersecurity Enhancement Act of 2015 requires DHS to include, in the efforts of the department to continuously diagnose and mitigate cybersecurity risks, advanced network security tools to improve the visibility of network activity and to detect and mitigate intrusions and anomalous activity  According to DHS officials, the department is addressing the requirement to improve the visibility of network activity by including advanced network security tools as a part of CDM phase 3  In April 2018, we testified that DHS had previously planned to provide 97 percent of federal agencies with the services they needed for CDM phase 3 in fiscal year 2017  In addition, according to OMB’s annual FISMA report for fiscal year 2017, the CDM program was to continue to incorporate additional capabilities, including phase 3, in fiscal year 2018  However, DHS now expects initial operational capabilities to be in place for phase 3 in fiscal year 2019  The department has awarded contracts of approximately $3 26 billion to support its Dynamic and Evolving Federal Enterprise Network Defense (also known as DEFEND) aspect of the CDM program, which is to include phase 3  DEFEND also is to provide coverage for existing agency deployments

ground_truth: Full implementation of GPRAMA could facilitate efforts to reform the federal government and make it more effective. GPRAMA includes a provision for GAO to review the act's implementation. This report assesses how GPRAMA implementation has affected the federal government's progress in resolving key governance challenges in (1) addressing cross-cutting issues, (2) ensuring performance information is useful and used, (3) aligning daily operations with results, and (4) building a more transparent and open government. To address these objectives, GAO reviewed statutory requirements, OMB guidance, and GAO's recent work related to GPRAMA implementation and the key governance challenges. GAO also interviewed OMB staff and surveyed a stratified random sample of 4,395 federal managers from 24 agencies on various performance and management topics. With a 67 percent response rate, the survey results are generalizable to the government-wide population of managers. The Office of Management and Budget (OMB) and agencies have made some progress in more fully implementing the GPRA Modernization Act (GPRAMA), but GAO's work and 2017 survey of federal managers highlight numerous areas where improvements are needed. Cross-cutting issues: Various GPRAMA provisions are aimed at addressing cross-cutting issues, such as cross-agency and agency priority goals and related data-driven reviews of progress towards those goals. To ensure alignment with the current administration's priorities, OMB's 2017 guidance removed the priority status of those goals, which stopped quarterly data-driven reviews and related public progress reports until new goals are published. OMB plans to resume implementation of these provisions in February 2018. GPRAMA also requires OMB and agencies to implement an inventory of federal programs, which could help decision makers better identify and manage fragmentation, overlap, and duplication. OMB and agencies implemented the inventory once, in May 2013. In October 2014, GAO found several issues limited the usefulness of that inventory. Since then, OMB has postponed updating the inventory, citing among other reasons the passage of subsequent laws. OMB has yet to develop a systematic approach for resuming implementation of the inventory and specific time frames for doing so. A systematic approach to developing the inventory could help ensure it provides useful information for decision makers and the public. Performance information: Survey results show federal managers generally reported no improvements in their use of performance information in decision making for various management activities, or practices that can enhance such use, since GAO's 2013 survey. For example, the use of performance information to streamline programs to reduce duplicative activities (an estimated 33 percent in 2017) is statistically significantly lower relative to 2013 (44 percent). In contrast, managers who were familiar with and whose programs were subject to quarterly data-driven reviews reported that those reviews were used to make progress toward agency priority goals. Identifying and sharing practices to expand the use of such reviews—for other performance goals and at lower levels within agencies—could lead to increased use of performance information. Daily operations: Agencies have made progress in developing results-oriented cultures but need to take additional actions. GAO's past work found that high-performing organizations use performance management systems to help individuals connect their daily activities to organizational goals. In 2017, about half of federal managers reported using performance information when setting expectations with employees (no change from GAO's last survey in 2013). Transparent and open government: GAO's past work identified a number of needed improvements to Performance.gov, the central government-wide website required by GPRAMA. The site is to provide quarterly updates on priority goals in effect through September 2017, but those updates stopped in December 2016. According to OMB, the existing information for cross-agency priority goals is the final update, and agencies should publish final updates on their priority goals in annual performance reports. Performance.gov does not provide users with this information, thereby limiting the transparency and accessibility of those results. In addition to following through on plans to resume implementation of key GPRAMA provisions, GAO recommends that OMB (1) consider a systematic approach to developing the program inventory, (2) revise guidance to provide specific time frames for inventory implementation, (3) identify and share practices for expanding the use of data-driven reviews, and (4) update Performance.gov to explain that reporting on priority goals was suspended and provide the location of final progress updates. OMB staff agreed with these recommendations.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.



Report 2:  According to DHS documentation, the task orders associated with DEFEND are to be issued between the second quarter of fiscal year 2018 and the second quarter of fiscal year 2024  FISMA requires that DHS provide operational and technical assistance to agencies in implementing policies, principles, standards, and guidelines on information security  Toward this end, DHS has available training and guidance related to the implementation of the capabilities of NCPS (i e , EINSTEIN) and CDM  Specifically: According the DHS officials, the department offers training and guidance to agencies on EINSTEIN 1 implementation  For example, DHS established a program in which the Software Engineering Institute will provide training and mentoring to agencies looking to enhance their understanding of, and proficiency with, the EINSTEIN 1 capability (e g , network traffic information)  NCPS program officials stated that agencies can use this service, which is available at no charge to them, on an unlimited basis as long as the requests relate to EINSTEIN 1  According to the officials, training and guidance related to EINSTEIN 2 and EINSTEIN 3 Accelerated is limited because DHS intentionally restricts the amount of data provided to agencies  According to DHS officials, the department also offers training and guidance to assist agencies with the implementation and use of resources associated with the CDM program, including webinars, guides, and computer-based training  The DEFEND contracts that the department awarded also include a mechanism for agencies to procure specialized tailored training, such as on the use of CDM tools  The department also offers customer advisory forums every other month that agencies are invited to attend  According to CDM program officials, the program’s governance, among other topics, is commonly discussed during these forums  Further, the department provides agencies with guidance, such as various governance documents, best practices, and frequently asked questions, through a web portal that is made available by OMB  In addition, US-CERT offers the CDM training program, which is to provide CDM implementation resources  Nevertheless, most agencies told us that they wanted DHS to provide more training and guidance as it relates to their implementation of the capabilities made available by NCPS and CDM  Specifically, Officials from 16 of 23 agencies reported that they wanted to receive additional training on NCPS capabilities  For example, officials at 5 agencies stated that they would like to receive training related to using network traffic information, understanding alerts, or implementing capabilities for cloud services  The officials also stated that they wanted training specific to agency security personnel  Officials from 19 of 23 agencies stated that they wanted to receive additional guidance related to NCPS’s capabilities, but not all of the 19 provided specific details  For example, officials from at least 3 agencies stated that they wanted additional guidance such as, “how to” documents, descriptions of architecture details, or guidance documents that explain NCPS’s capabilities so that agencies can gauge the gap between the security that the system provides and the security being provided by their own agency’s capabilities  Officials from 21 of 23 agencies reported that they wanted to receive additional training on implementing CDM at their agencies  For example, officials from 7 agencies suggested that additional training on the use of the tools would be beneficial  Officials from 22 of 23 agencies stated that they wanted additional guidance as it relates to CDM implementation  For example, officials from one agency stated that they would like examples of best practices and successful deployments  These requests for additional training and guidance demonstrate that agencies are either unaware of the available training and guidance, or that the training may not meet their needs  Until DHS coordinates with agencies to determine if additional training and guidance are needed, agencies may not be able to fully realize the benefits of the capabilities provided by the NCPS and CDM programs  Although OMB took steps to report on agencies’ implementation of intrusion detection and prevention capabilities, it did not report on all required actions  For example, the office did not submit DHS’s intrusion plan to Congress as required by the Federal Cybersecurity Enhancement Act of 2015  In addition, OMB provided various reports to Congress that described agencies’ intrusion detection and prevention capabilities, but the reports did not always include all information required by the act  Further, OMB developed a draft policy and strategy that were intended to improve agency capabilities, but it had not finalized these documents  The Federal Cybersecurity Enhancement Act of 2015 requires OMB to submit the intrusion assessment plan developed by DHS to the appropriate congressional committees no later than 6 months after the date of enactment of the act  The act also required OMB to submit to Congress a description of the implementation of the intrusion assessment plan and the findings of the intrusion assessments conducted pursuant to the intrusion assessment plan no later than 1 year after the date of enactment of the act, and annually thereafter  Although DHS developed and documented an intrusion assessment plan, which described a defense-in-depth approach to security, OMB did not submit the plan to Congress, as called for in the act  Even though DHS submitted the plan to OMB in January 2017, OMB had not submitted it to Congress as of October 2018 (21 months after DHS submitted the plan and 28 months past the due date)  On the other hand, OMB did submit its own reports to Congress which generally described elements of the implementation of DHS’s intrusion assessment plan and intrusion assessment findings  In September 2017, OMB issued its analysis of agencies’ implementation of intrusion detection and prevention capabilities, or more specifically, agencies’ implementation of the various versions of NCPS  In addition, the office’s annual FISMA report, issued most recently in March 2018, generally covered elements of the intrusion assessment plan  OMB personnel within the Office of the Federal CIO believed that these two reports, along with a process the office had initiated to validate incidents across the government, addressed the requirement for OMB to submit to Congress a description of the implementation of the intrusion assessment plan and the findings of the intrusion assessments conducted pursuant to the plan  However, the September 2017 and March 2018 reports did not address other elements described in DHS’s intrusion assessment plan  For example, OMB did not describe agency roles associated with segmenting their networks, identifying key servers based on threat and impact, ensuring all applications are appropriately tracked and configured, and categorizing and tagging data based on threat and impact  While OMB has provided important information to congressional stakeholders through its own reports, until it submits the plan and addresses all elements described in DHS’s intrusion assessment plan, it will continue to be remiss in providing timely and sufficiently detailed information regarding the intrusion assessment plan to congressional stakeholders to support their oversight responsibilities  The Federal Cybersecurity Enhancement Act of 2015 also required that OMB submit an analysis of agencies’ application of the intrusion detection and prevention capabilities to Congress no later than 18 months after the date of enactment of the act, and annually thereafter  OMB was to include a list of federal agencies and the degree to which each agency had applied the intrusion detection and prevention capabilities in this analysis  As discussed previously in this report, OMB issued its analysis of agencies’ implementation of intrusion detection and prevention capabilities in September 2017  However, the analysis did not include the degree to which agencies had applied the intrusion detection and prevention capabilities  For example, the analysis did not reflect that not all agencies were using this capability on all information traveling between their systems and any system other than an agency system, as required by the act  Until OMB includes the degree to which agencies have applied intrusion detection and prevention capabilities in its analysis, it cannot provide congressional stakeholders with an accurate portrayal of the extent to which the capabilities are detecting and preventing potential intrusions  The Federal Cybersecurity Enhancement Act of 2015 further required that the Federal Chief Information Officer, within OMB, submit a report to Congress no earlier than 18 months after the date of enactment, but no later than 2 years after that date, assessing the intrusion detection and intrusion prevention capabilities that DHS made available to agencies  The act required that the report address (1) the effectiveness of DHS’s system used for detecting, disrupting, and preventing cyber-threat actors, including advanced persistent threats, from accessing agency information and agency information systems; (2) whether the intrusion detection and prevention capabilities, continuous diagnostics and mitigation, and other systems deployed are effective in securing federal information systems; (3) the costs and benefits of the intrusion detection and prevention capabilities, including as compared to commercial technologies and tools, and including the value of classified cyber threat indicators; and (4) the capability of agencies to protect sensitive cyber threat indicators and defensive measures if they were shared through unclassified mechanisms for use in commercial technologies and tools  In a report issued in September 2018 (about 8 months past the required due date), the Federal Chief Information Officer provided Congress an assessment of intrusion detection and intrusion prevention capabilities across the federal enterprise  The report pointed out, among other things, that agencies did not possess or properly deploy capabilities to detect or prevent intrusions or minimize the impact of intrusions when they occur  In addition, the report acknowledged the need to improve the effectiveness of intrusion detection and intrusion prevention capabilities and stated that OMB would track performance through the CAP goal and annual FISMA reports  However, the report did not address all of the requirements specified in the act  For example, the report did not address whether DHS’s system (i e , NCPS) was effective in detecting advanced persistent threats  In addition, the report did not include a comparison of the costs and benefits of the intrusion detection and prevention capabilities versus commercial technologies and tools, or the value of classified cyber threat indicators  Further, the report did not address the capability of agencies to protect sensitive cyber threat indicators and defensive measures  Until OMB updates the Federal CIO report to address all of the requirements specified in the act, it will continue to be remiss in providing timely and sufficiently detailed information, such as that related to costs and benefits, among other elements in the act, to congressional stakeholders to support their oversight responsibilities  In addition to OMB’s responsibilities in the Federal Cybersecurity Enhancement Act of 2015, OMB has initiated plans for further improving agencies’ intrusion detection and prevention capabilities  In response to a tasking in Executive Order 13800, the Director of the American Technology Council coordinated the development of a report to the President from the Secretary of DHS, the Director of OMB, and the Administrator of the General Services Administration, regarding the modernization of federal information technology (IT)  The report, Report to the President on Federal IT Modernization, identified actions that OMB should take for (1) prioritizing the modernization of high-risk, high-value assets and (2) modernizing the Trusted Internet Connection (TIC) program and NCPS to improve protections, remove barriers, and enable commercial cloud migration  For example, OMB was to take the following actions subsequent to the December 13, 2017 report issuance date: Within 60 days: Update a TIC policy to address challenges with agencies’ perimeter-based architectures, such as the modernization of NCPS  In addition, introduce a “90 day sprint” during which approved projects would pilot proposed changes in TIC requirements  Within 90 days: Update the annual FISMA and CAP goal metrics to focus on those critical capabilities that were most commonly lacking among agencies and focus oversight assessments on high-value assets  Within 120 days: In conjunction with DHS, develop a strategy for optimally realigning resources across agencies to reduce the risk to high-value assets and respond to cybersecurity incidents for those assets  OMB has taken steps toward implementing several, but not all, of these actions  For example, it introduced a “90 day sprint” and, according to knowledgeable OMB staff, the outcomes of this action are directly informing changes in TIC requirements  In addition, OMB updated the annual FISMA and CAP goal metrics by including several metrics that focus on high-value assets  The updated FISMA and CAP goal metrics went into effect in April 2018  However, while OMB had taken steps toward updating the TIC policy and developing a strategy for optimally realigning resources, the policy and strategy were in draft and had not yet been finalized as of October 2018  The agency did not specify a time frame for finalizing the policy and strategy  Until OMB finalizes the TIC policy and the strategy for optimally realigning resources, the enhancements offered through the policy and strategy are unlikely to be realized  FISMA requires agencies to provide information security protections to prevent unauthorized access to their systems and information  Officials from the 23 selected agencies reported to us that they generally took steps to meet this requirement by augmenting the tools and services provided by DHS with their own intrusion detection and prevention capabilities  However, agencies did not consistently implement five key capabilities specified by DHS and NIST guidance  In addition, most of the agencies did not fully implement any of the phases of DHS’s CDM program that is intended to improve their capabilities to detect and prevent intrusions  Binding Operational Directive (BOD) 18-01 instructs agencies to enhance email security  These enhancements include enabling encrypted email transmission, ensuring that receiving mail servers know what actions the agency would like taken when an email falsely claims to have originated from the agency, and removing certain insecure protocols, among others  The final deadline for implementing all BOD 18-01 requirements was October 16, 2018  Additionally, NIST SP 800-53 Revision 4 recommends that security awareness training include training on how to recognize and prevent spear-phishing attempts  As of September 2018, only 2 of the 23 agencies reported implementing all of the email requirements  For the remaining 21 agencies: 9 agencies stated that their agency had plans to implement all enhancements by the October 2018 deadline, 1 agency was uncertain whether it would meet the deadline, and 11 stated they would not be able to meet the deadline  By contrast, the majority of agencies (22 of 23) reported that they had trained staff on spear-phishing exercises, as recommended by NIST SP 800-53 Revision 4  Officials at the remaining agency told us that the agency planned to have spear-phishing exercises in fiscal year 2019  Such training should help ensure that phishing will be a less effective attack vector against the majority of agencies  While agencies benefit from secure protocols and spear-phishing training, implementing the remaining BOD 18-01 email requirements would provide additional protection to agency information systems  NIST recommends that federal agencies deploy intrusion detection and prevention capabilities  These capabilities include monitoring cloud services, using host-based intrusion prevention systems, monitoring external and internal network traffic, and using a security information and event management (SIEM) system  However, in our semi-structured interviews of the 23 agencies, officials told us that they often had not implemented many of these capabilities  Such inconsistent implementation exposes federal systems and the information they contain to additional risk  As part of their continuing oversight efforts, OMB and DHS can use the information below to work with agencies to identify obstacles and impediments affecting the agencies’ abilities to implement these capabilities  NIST SP 800-53 Revision 4 states that agencies should monitor and control communications at the external boundary of the network  However, as of June 2018, fewer than half of the agencies that used cloud computing services were monitoring cloud traffic  Specifically: 10 of 22 agencies that used Infrastructure as a Service were monitoring inbound and outbound Infrastructure as a Service traffic, 7 of 21 agencies that used Platform as a Service were monitoring inbound and outbound Platform as a Service traffic, and 10 of 23 agencies that used Software as a Service were monitoring inbound and outbound Software as a Service traffic  Without monitoring traffic to and from cloud service providers, agencies risk a greater chance of malicious cloud activity detrimentally affecting agency information security  NIST SP 800-53 Revision 4 states that agency internal monitoring may be achieved by utilizing intrusion prevention capabilities  These capabilities include using host-based intrusion prevention systems to provide defense at an individual system or device level by protecting against malicious activities  Host-based capabilities include memory-based protection and application whitelisting  As of June 2018, officials at the 23 agencies reported the following to us: 16 agencies used host-based intrusion prevention capabilities, 15 agencies used memory-based protection, and 8 agencies used host-based application whitelisting  Until host-based intrusion protections are fully deployed, agencies will be at greater risk of malicious activity adversely affecting agency operations  NIST SP 800-53 Revision 4 also states that agencies should monitor and control communications at the external boundary of the network and at key internal boundaries (e g , network traffic)  NIST guidance also stated that an agency should deploy monitoring devices strategically within the network to detect essential information  However, the agencies in our review did not always monitor external and internal traffic  For example, of the 23 agencies: 5 reported that they were not monitoring inbound or outbound direct connections to outside entities  11 reported that they were not persistently monitoring inbound encrypted traffic  8 reported that they were not persistently monitoring outbound encrypted traffic  In addition, 13 agencies reported they were not using a network-based session capture solution  Of the 10 agencies that reported using this solution, officials from 2 agencies stated that they were not capturing workstation-to-workstation connections  Without thorough monitoring of external and internal traffic, agencies will have less assurance that they are aware of compromised or potentially compromised traffic within their network  NIST SP 800-53 Revision 4 states that agencies should establish enhanced monitoring capabilities  Such capabilities should include automated mechanisms that collect and analyze incident data for increased threat and situational awareness  According to NIST, a security information and event management (SIEM) system analyzes data from different sources and identifies and prioritizes significant events  Sources of data used by SIEM systems include logs from database systems, network devices, security systems, web applications, and workstation operating systems  Of the 23 agencies that we reviewed, 21 reported using a SIEM capability  Over half of the agencies employing a SIEM used one or more of their logs to match against known vulnerabilities and advanced persistent threats, as well as to create real-time alerts  For example, of the 21 agencies: 14 agencies reported collecting database logs, but only 7 agencies reported using the logs to match against known vulnerabilities and persistent threats and to create real-time alerts; 20 agencies reported collecting network logs, but only 13 agencies reported using them to match against known vulnerabilities and persistent threats and to create real-time alerts; 21 agencies reported collecting security logs, but only 13 reported using them to match against known vulnerabilities and persistent threats and to create real-time alerts; 15 agencies reported collecting web application logs, but only 9 agencies reported using them to match against known vulnerabilities and persistent threats and to create real-time alerts; and 13 agencies reported collecting workstation logs, but only 8 agencies reported using them to match against known vulnerabilities and persistent threats and to create real-time alerts  Only 5 agencies collected all 5 types of logs and used them to match against known vulnerabilities and persistent threats and to create real- time alerts  By not fully using SIEM capabilities, agencies will have less assurance that relevant personnel will be aware of possible weaknesses or intrusions  To further enhance their intrusion detection and prevention capabilities, the 23 civilian CFO Act agencies were in the process of implementing DHS’s CDM program  As previously discussed, Phase 1 of the program involves deploying products to automate hardware and software asset management, configuration settings, and common vulnerability management capabilities  Phase 2 intends to address privilege management and infrastructure integrity by allowing agencies to monitor users on their networks and to detect whether users are engaging in unauthorized activity  Phase 3 is intended to assess agency network activity and identify any anomalies that may indicate a cybersecurity compromise  As of June 2018, most agencies had not fully implemented any of the three phases  As shown in Figure 7, 15 agencies had partially implemented phase 1, 21 had partially or not yet begun to implement phase 2, and none of the agencies had fully implemented phase 3  Agencies’ implementation status has been affected, at least in part, due to delays in DHS’s deployment of the program phases  As a result, federal systems will remain at risk until the program is fully deployed  Many agencies have not effectively implemented the federal approach and strategy for securing information systems  For example, the inspectors general for 17 of the 23 selected agencies reported that their agencies had not effectively implemented their information security programs and had significant information security deficiencies associated with internal control over financial reporting  In addition, CIOs for 17 agencies reported not meeting all nine targets for the cybersecurity cross- agency priority goal  Further, OMB determined that that only 13 of the 23 agencies were managing risks to their overall enterprise, while the other 10 agencies were at risk  Until agencies more effectively implement the government’s approach and strategy, federal systems will remain at risk  DHS and OMB have initiatives underway that are intended to further improve agencies’ security posture  However, although DHS had provided training and guidance for NCPS and CDM, agencies expressed the need for more  In addition, OMB had also not finalized its policy and strategy aimed at addressing challenges with perimeter security and protecting high value assets, respectively  OMB had also not provided useful information to Congress, such as a description of agencies’ implementation of DHS’s intrusion assessment plan, the degree to which agencies are using NCPS, a complete analysis of agencies’ implementation of DHS’s intrusion assessment plan, or the costs and benefits of using commercial tools  Although agencies’ officials reported various efforts underway to enhance their agency’s intrusion detection and prevention capabilities, implementation efforts across the federal government were not consistent  OMB and DHS can use the information provided in this report to work with agencies to identify obstacles and impediments affecting the agencies’ abilities to implement these capabilities  We are making a total of nine recommendations, including two to DHS and seven to OMB  Specifically: The Secretary of DHS should direct the Network Security Deployment division to coordinate further with federal agencies to identify training and guidance needs for implementing NCPS and CDM  (Recommendation 1) The Secretary of DHS should direct the appropriate staff to work with OMB to follow up with agencies to identify obstacles and impediments affecting their abilities to implement intrusion detection and prevention capabilities  (Recommendation 2) The Director of OMB should submit the intrusion assessment plan to the appropriate congressional committees  (Recommendation 3) The Director of OMB should report on implementation of the defense- in-depth strategy described in the intrusion assessment plan, including all elements described in the plan  (Recommendation 4) The Director of OMB should update the analysis of agencies’ intrusion detection and prevention capabilities to include the degree to which agencies are using NCPS  (Recommendation 5) The Director of OMB should direct the Federal CIO to update her report to Congress to include required information, such as detecting advanced persistent threats, a comparison of the costs and benefits of the capabilities versus commercial technologies and tools, and the capability of agencies to protect sensitive cyber threat indicators and defense measures  (Recommendation 6) The Director of OMB should establish a time frame for finalizing the Trusted Internet Connections policy intended to address challenges with agencies’ perimeter-based architectures and issue it when finalized  (Recommendation 7) The Director of OMB should establish a time frame for finalizing the strategy for realigning resources across agencies to protect high- value assets and issue it when finalized  (Recommendation 8) The Director of OMB should direct the Federal CIO to work with DHS to follow-up with agencies to identify obstacles and impediments affecting their abilities to implement intrusion detection and prevention capabilities  (Recommendation 9) We provided a draft of this report to OMB and the 23 civilian CFO Act agencies, including DHS, covered by our review  In response, OMB provided comments via email, and DHS and three other agencies (the Department of Commerce, Social Security Administration, and U S  Agency for International Development) provided written comments, which are reprinted in appendices V through VIII, respectively  The 19 remaining agencies (the Departments of Agriculture, Education, Energy, Health and Human Services, Housing and Urban Development, the Interior, Justice, Labor, State, Transportation, the Treasury, and Veterans Affairs; as well as the Environmental Protection Agency, General Services Administration, National Aeronautics and Space Administration, National Science Foundation, Nuclear Regulatory Commission, Office of Personnel Management, and Small Business Administration) stated via email that they had no comments  In its comments, which the OMB liaison provided to GAO via email on December 7, 2018, OMB did not state whether it agreed or disagreed with the seven recommendations that we made to it  Rather, according to the liaison, OMB agreed with the facts in our draft report, but found that the report did not reflect the agency’s rationale for not submitting the DHS intrusion assessment plan to Congress and a report on the implementation of the plan, as required by the Federal Cybersecurity Enhancement Act of 2015  The liaison stated that OMB is working closely with DHS to provide strategic direction in assessing gaps in, and modernizing, the manner in which intrusion detection and prevention capabilities are delivered to the federal government  Further, in a subsequent email on December 10, 2018, OMB said it believes the Federal CIO’s September 2018 report to Congress, along with data provided in OMB’s fiscal year 2017 FISMA report to Congress, achieves the outcomes sought by the Federal Cybersecurity Enhancement Act of 2015 and demonstrates OMB's continuous engagement with DHS across the evolution of the intrusion detection and prevention program  As stated in our report, we acknowledge that OMB has provided important information to congressional stakeholders through its reports  However, OMB’s reports did not cover all outcomes described in the act  For example, as we pointed out, these reports did not fully address implementation of the defense-in-depth strategy described in DHS’s intrusion assessment plan  In addition, although OMB reported on several elements required by the Federal Cybersecurity Enhancement Act of 2015, it did not report on all of the required elements  For example, the reports did not address whether DHS’s NCPS was effective in detecting advanced persistent threats  The reports also did not include a comparison of the costs and benefits of the intrusion detection and prevention capabilities versus commercial technologies and tools, or the value of classified cyber threat indicators  Further, the reports did not address the capability of agencies to protect sensitive cyber threat indicators and defensive measures  Accordingly, we maintain that our recommendations for OMB to report on required elements in the Federal Cybersecurity Enhancement Act of 2015 are warranted  In addition, OMB suggested that we revise our recommendations to the agency to include a shared responsibility with DHS to help drive desired outcomes  However, six of the seven recommendations we are making to OMB are related to specific OMB responsibilities cited in either the Federal Cybersecurity Enhancement Act of 2015 or the Report to the President on Federal IT Modernization  As such, we believe the recommendations are appropriately addressed to OMB  Furthermore, our recommendations do not prevent OMB from working with DHS to implement them  Our seventh recommendation to OMB—to work with DHS to follow up with agencies to identify obstacles and impediments affecting their abilities to implement intrusion detection and prevention capabilities—includes a shared responsibility with DHS  OMB also provided technical comments, which we incorporated into the report, as appropriate  Subsequent to providing initial comments on our draft report, OMB issued a memorandum intended to provide a strategy for realigning resources across agencies to protect high-value assets  This action addresses our recommendation 8, which called for the Director of OMB to establish a time frame for finalizing the strategy for realigning resources across agencies to protect high-value assets, and to issue the strategy when finalized  In its comments, DHS stated that it concurred with the two recommendations we made to the department  DHS stated that it expects to implement the recommendations in 2019  The Department of Commerce commented that the report was reasonable and that the department agreed with the findings and recommendations  In its comments, the Social Security Administration stated that protecting its networks and information is a critical priority  According to the agency, it continued to make improvements in fiscal year 2018, such as improvements and progress in securing applications, leveraging the cloud, managing its assets and vulnerabilities, strengthening its network and incident response capabilities, improving its security training, and enhancing the overall effectiveness of its cybersecurity program  Finally, the U S  Agency for International Development commented that its inspector general had improved the agency’s capability maturity ratings for core security functions in fiscal year 2018  The agency also pointed out that it was the only selected agency in which fiscal year 2017 indicators of effectiveness in implementing the federal approach and strategy for securing information systems were all positive (as noted in Appendix III)  We are sending copies of this report to appropriate congressional committees, the Director of OMB, the heads of the 23 civilian CFO Act agencies and their inspectors general, and other interested congressional parties  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact Gregory C  Wilshusen at (202) 512-6244 or wilshuseng@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made key contributions to this report are listed in appendix IX  The Federal Cybersecurity Enhancement Act of 2015, which was enacted December 18, 2015, included a provision for GAO to report on the effectiveness of the federal government’s approach and strategy for securing agency information systems, including intrusion detection and prevention capabilities  The objectives of our review were to assess: (1) the reported effectiveness of selected agencies’ implementation of the federal government’s approach and strategy to securing agency information systems; (2) the extent to which the Office of Management and Budget (OMB) and the Department of Homeland Security (DHS) have facilitated the use of intrusion detection and prevention capabilities to secure federal agency information systems; and (3) the extent to which selected agencies reported implementing intrusion detection and prevention capabilities  Selected agencies for our review were the 23 civilian agencies covered by the Chief Financial Officers Act of 1990 (CFO Act)  We did not include the Department of Defense because the Federal Cybersecurity Enhancement Act of 2015 only pertains to civilian agencies  Because we focused our work on the 23 civilian agencies, results from these reviews are not generalizable to the entire federal government  To assess the reported effectiveness of agencies’ implementation of the federal government’s approach and strategy to securing agency information systems, we described the federal government’s approach and strategy by summarizing the Federal Information Security Modernization Act of 2014 (FISMA), Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure, and the National Institute of Standards and Technology’s (NIST) Framework for Improving Critical Infrastructure Cybersecurity (cybersecurity framework)  assessed the reported effectiveness of agencies’ implementation of the approach and strategy by reviewing annual reports from OMB and the inspectors general (IG) of the 23 civilian CFO Act agencies regarding the reported implementation of FISMA for fiscal year 2017  We described the IG reported maturity levels, including the Office of Inspectors General FISMA Reporting Metrics definition of “effectiveness ” These maturity levels are based on security domains aligned with the five core functions in NIST’s cybersecurity framework  We also summarized IG reported conclusions on the effectiveness of agencies’ information security programs for fiscal year 2017  reviewed the fiscal year 2016 and 2017 financial statement audit reports for each of the 23 civilian CFO Act agencies to identify the extent to which any significant deficiencies or material weaknesses related to information security over financial systems had been reported and to identify information security control weaknesses reported by the IGs  identified whether agencies had met the targets for the cybersecurity- focused cross-agency priority goal for fiscal years 2016 and 2017 by examining agency-reported performance metrics for fiscal years 2016 and 2017  evaluated OMB’s agency risk management assessment ratings to make a determination on how agencies were managing risk to their enterprise  These conclusions were based on FISMA metrics, and are aligned with the five core security functions defined in the cybersecurity framework  interviewed knowledgeable OMB officials and staff to obtain their views on the reported effectiveness of the federal government’s approach and strategy to securing agency information systems  To assess the extent to which OMB and DHS have facilitated the use of intrusion detection and prevention capabilities to secure federal agency information systems, we determined the extent OMB and DHS fulfilled their requirements described in the Federal Cybersecurity Enhancement Act of 2015 by collecting and reviewing artifacts from OMB and DHS and comparing them to the provisions outlined in the act  We also interviewed knowledgeable officials from OMB and DHS regarding their efforts to fulfill their requirements described in the act  determined the effectiveness of corrective actions taken by DHS to address nine previously reported recommendations we made in our report related to NCPS  Specifically, we collected appropriate artifacts and assessed the artifacts against the criteria used in that report, and determined the extent to which the actions taken by DHS met the intent of the recommendations, and we met with DHS staff responsible for the remediation activities and obtained their views of the status of actions taken to address the recommendations  held semi-structured interviews with knowledgeable officials from the 23 civilian CFO Act agencies  During these interviews, we obtained the agency’s views on whether they need more training and guidance from DHS for NCPS and CDM  We also interviewed knowledgeable officials and staff at DHS to obtain their views on how DHS had improved the intrusion detection and prevention capabilities it provides to federal agencies  We also interviewed DHS officials to obtain their views on the training and guidance that the department makes available to agencies  To assess the extent to which selected agencies reported implementing intrusion detection and prevention capabilities, we described the reported intrusion detection and prevention capabilities implemented by the 23 civilian CFO Act civilian agencies by summarizing implemented intrusion detection and prevention capability information obtained from the semi-structured interviews at the 23 civilian CFO Act agencies described above; identifying the extent to which the 23 civilian CFO Act agencies were in compliance with DHS’s binding operating directive (BOD) pertaining to enhanced email and web security (BOD 18-01) by collecting and summarizing Cyber Hygiene Trustworthy Email reports from the 23 agencies and determining the extent to which the agencies had taken required actions to implement the BOD  During the semi-structured interviews, we also obtained the agency’s views and experiences with other programs and services provided by DHS, including the extent to which agencies had implemented the tools offered by the department’s Continuous Diagnostics and Mitigation (CDM) program  To determine the reliability of submitted data and obtain clarification about agencies’ processes to ensure the accuracy and completeness of data used in their respective FISMA reports, we analyzed documents and conducted interviews with officials from 6 of the 23 civilian CFO Act agencies  To select these six agencies, we sorted agency fiscal year 2017 information technology budget data from highest to lowest amount and then divided the data into three tiers: high spending, medium spending, and low spending  We then randomly selected two agencies from each of the three tiers  The selected agencies were the Departments of Agriculture, Commerce, Housing and Urban Development, Transportation, and Veterans Affairs, and the U S  Agency for International Development  While not generalizable to all agencies, the information we collected and analyzed about the six selected agencies provided insights into various processes in place to produce FISMA reports  Based on this assessment, we determined that the data were sufficiently reliable for the purposes of our reporting objectives  We conducted this performance audit from December 2017 to December 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The National Institute of Standards and Technology established the cybersecurity framework to provide guidance for cybersecurity activities within the private sector and government agencies at all levels  The cybersecurity framework consists of five core functions: identify, protect, detect, respond, and recover  Within the five functions are 23 categories and 108 subcategories that define discrete outcomes for each function, as described in table 7  The federal approach and strategy for securing information systems is prescribed by federal law and policy, including the Federal Information Security Modernization Act of 2014 and the presidential executive order on Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure  Accordingly, federal reports describing agency implementation of this law and policy, and reports of related agency information security activities, indicated the effectiveness of agencies’ efforts to implement the federal approach and strategy  Table 8 summarizes the reported effectiveness of the 23 civilian Chief Financial Officers Act of 1990 agencies to implement the government’s approach and strategy to securing information systems  The President’s Management Agenda identifies cross-agency priority (CAP) goals to target areas where multiple agencies must collaborate to effect change  The agenda issued in fiscal year 2018 established an information technology modernization goal that includes a cybersecurity objective with specific priority areas and performance indicators  This cybersecurity-focused goal is intended to drive progress in the government’s efforts to modernize information technology to increase productivity and security  Figure 8 describes the 3 updated cybersecurity- focused cross-agency priority areas and 10 performance indicators  Each federal agency is expected to meet one of the 10 new performance indicators by the end of fiscal year 2018 and the remainder by 2020  In addition to the individual named above, Jeffrey Knott (assistant director), Daniel Swartz (analyst-in-charge), David Blanding, Chris Businsky, Kristi Dorsey, Di’Mond Spencer, Priscilla Smith, and Edward Varty made key contributions to this report  West Coile, Franklin Jackson, and Chris Warweg also provided assistance 
Report 2:  The national limits were set at a percentage (74%) of the median of all local fee schedule amounts for each laboratory test code; therefore, fee schedule amounts may differ by region  In general, annual increases in clinical lab fees have been based on the percentage change in the CPI-U  However, since 1987, Congress has specified lower updates  Beginning in 2014, with certain exceptions, laboratory tests provided in hospital outpatient departments are no longer paid separately under the CLFS and are instead included in the OPPS payments  PAMA introduced a new method for determining clinical laboratory payments and required CMS to base Medicare CLFS reimbursement on reported private insurance payment amounts  Medicare has been using weighted median private insurer rates to calculate Medicare payment rates for laboratory tests paid under the CLFS since January 1, 2018  These payment rates are national and do not vary by geographic area  Part B also covers diagnostic nonlaboratory x-ray tests and other diagnostic tests, as well as x-ray, radium, and radioisotope therapy  Generally, these services are paid for under the physician fee schedule, with beneficiaries responsible for a 20% coinsurance payment  Medicare covers a wide variety of equipment and devices under the heading of durable medical equipment (DME), prosthetics, and orthotics (PO) if they are medically necessary and are prescribed by a physician  DME is defined as equipment that (1) can withstand repeated use, (2) has an expected life of at least three years (effective for items classified as DME after January 1, 2012), (3) is used primarily to serve a medical purpose, (4) is not generally useful in the absence of an illness or injury, and (5) is appropriate for use in the home  DME includes such items as hospital beds, wheelchairs, blood glucose monitors, and oxygen and oxygen equipment  It also includes related supplies (S), such as drugs and biologics that are necessary for the effective use of the product  Prosthetics (P) are items that replace all or part of a body organ or its function, such as colostomy bags, pacemakers, and artificial eyes, arms, or legs  Orthotics (O) are braces that support a weak or deformed body member, such as leg or back braces  Except in competitive bidding areas (CBAs, described below), Medicare pays for most durable medical equipment, prosthetics, orthotics, and supplies (DMEPOS) based on fee schedules  Medicare pays 80% of the lower of the supplier's charge for the item or the fee schedule amount  The beneficiary is responsible for the remaining 20%  In general, fee schedule amounts are updated each year by a (1) measure of price inflation, and (2) a measure of economy-wide productivity, which may result in lower fee schedule amounts from one year to the next  Since 2016, the fee schedule rates that applied outside of competitive bidding areas for certain DMEPOS have been reduced based on price information from the competitive bidding program  The reductions were phased in during 2016, and fully phased in starting in January 2017  In response to concerns that the adjusted rates were too low, the Secretary again applied the phase-in rate for rural and noncontiguous areas not subject to competitive bidding starting in June 2018  Currently, two different fee schedules apply outside of CBAs  First, in rural or noncontiguous areas, the fee schedule is a 50/50 blend of the fee schedule with and without the reductions based on information from competitive bidding (i e , the phase-in methodology)  Second, in nonrural and contiguous areas, the fee schedule amounts are fully adjusted by information from the competitive bidding program  Numerous studies and investigations indicated that Medicare paid more for certain items of DME and PO than some other health insurers and some retail outlets  Such overpayments were attributed, in part, to the fee schedule mechanism of payment  MMA required the Secretary to establish a Competitive Acquisition Program for certain DMEPOS in specified areas  Instead of paying for medical equipment based on a fee schedule established by law, payment for items in competitive bidding areas was based on the supplier bids  The program started in 9 metropolitan areas in January 2011 and had expanded to 130 competitive bidding areas in 2018  However, the program has been suspended while a new bidding methodology is established  During the gap, the payments for previously competitively bid items in competitive bidding areas will be the amounts that applied on December 31, 2018, increased, yearly, by a measure of inflation  Certain specified outpatient prescription drugs and biologics are covered under Medicare Part B  (However, most outpatient prescription drugs are covered under Part D, discussed below ) Covered Part B drugs and biologics include drugs furnished incident to physician services, immunosuppressive drugs following a Medicare-covered organ transplant, erythropoietin for treatment of anemia for individuals with ESRD when not part of the ESRD composite rate, oral anticancer drugs (provided they have the same active ingredients and are used for the same indications as chemotherapy drugs that would be covered if furnished incident to physician services), certain vaccines under selected conditions, and drugs administered through DME  Generally, Medicare reimburses physicians and other providers, such as hospital outpatient clinics, for covered Part B drugs and biologics at 106% of the volume weighted average sales price of all drugs billed under the same billing code, although some Part B drugs, such as those administered through DME, are reimbursed at 95% of the drug's average wholesale price  Health care providers also are paid separately for administering Medicare Part B drugs  Medicare pays 80% of the amount paid to providers, and beneficiaries are responsible for the remaining 20%  A hospital outpatient is a person who has not been admitted by the hospital as an inpatient but is registered on the hospital records as an outpatient  Generally, payments under the hospital outpatient prospective payment system (OPPS) cover the operating and capital-related costs that are directly related and integral to performing a procedure or furnishing a service on an outpatient basis  These payments cover services such as the use of an operating suite, treatment, procedure, or recovery room; use of an observation bed as well as anesthesia; certain drugs or pharmaceuticals; incidental services; and other necessary or implantable supplies or services  Payments for services such as those provided by physicians and other professionals as well as therapy and clinical diagnostic laboratory services, among others, are separate  Under the OPPS, the unit of payment for acute care hospitals is the individual service or procedure as assigned to an ambulatory payment classification (APC)  To the extent possible, integral services and items (excluding physician services paid under the physician fee schedule) are bundled within each APC  Specified new technologies are assigned to "new technology APCs" until clinical and cost data are available to permit assignment into a "clinical APC " Medicare's hospital outpatient payment is calculated by multiplying the relative weight associated with an APC by a conversion factor  For most APCs, 60% of the conversion factor is geographically adjusted by the wage index used for the inpatient prospective payment system  Except for new technology APCs, each APC has a relative weight that is based on the median cost of services in that APC  The OPPS also includes pass-through payments for new technologies (specific drugs, biologicals, and devices) and payments for outliers  The Medicare Payment Advisory Commission (MedPAC) has recommended site-neutral payment policies that base payments on the resources needed to provide high-quality care in the most efficient setting  The Bipartisan Budget Act of 2015 (BBA 15; P L  114-74 ) gave CMS the authority to add new restrictions on Medicare payments for services furnished in provider-based off-campus hospital outpatient departments (HOPDs) to address discrepancies between payments under the MPFS and the OPPS for similar services  In its 2019 OPPS Final Rule, CMS explicitly applies the site-neutral policy to clinic visits, the most commonly billed service in hospital outpatient departments  An ambulatory surgical center (ASC) is a distinct entity that furnishes outpatient surgical procedures to patients who do not require an overnight stay after the procedure  According to MedPAC, most ASCs are freestanding facilities rather than part of a larger facility, such as a hospital  Medicare covers surgical and medical services performed in an ambulatory surgical center that are (1) commonly performed on an inpatient basis but may be safely performed in an ASC; (2) not of a type that are commonly performed or that may be safely performed in physicians' offices; (3) limited to procedures requiring a dedicated operating room or suite and generally requiring a postoperative recovery room or short-term (not overnight) convalescent room; and (4) not otherwise excluded from Medicare coverage  Medicare pays for surgery-related facility services provided in ASCs using a prospective payment system based on the OPPS  (Associated physician fees are paid for separately under the physician fee schedule ) Each of the approximately 3,500 procedures approved for payment in an ASC is classified into an APC group on the basis of clinical and cost similarity  The ASC system primarily uses the same payment groups as the OPPS; however, ASC payment rates are generally lower  The ASC weights are scaled (reduced) to account for the different mix of services in an ASC, and the ASC conversion factor (the base payment amount) is lower  A different payment method is used to set ASC payment for office-based procedures, separately payable radiology services, separately payable drugs, and device-intensive procedures  In addition, separate payments are made for certain ancillary items and services when they are integral to surgical procedures, including corneal tissue acquisition, brachytherapy sources, certain radiology services, many drugs, and certain implantable devices  The application of the site-neutral payment policy to clinic visits also affects such payments to ASCs (see discussion above)  Medicare Part B covers emergency and nonemergency ambulance services to or from a hospital, a critical access hospital, a skilled nursing facility, or a dialysis facility for End-Stage Renal Disease (ESRD) beneficiaries who require dialysis when other modes of transportation could endanger the Medicare beneficiary's health  In most cases, the program covers 80% of the allowed amount for the service, and the beneficiary is responsible for the remaining 20%  Generally, ambulance services are covered if (1) transportation of the beneficiary occurs; (2) the beneficiary is taken to an appropriate location (generally, the closest appropriate facility); (3) the ambulance service is medically necessary (other forms of transportation are contraindicated); (4) the ambulance provider or supplier meets state licensing requirements; and (5) the transportation is not part of a Medicare Part A covered stay  Medicare covers both scheduled and nonscheduled nonemergency transports if the beneficiary is bed-confined or meets other medical necessity criteria  Medicare may also cover emergency ambulance transportation by airplane or helicopter if the beneficiary's location is not easily reached by ground transportation or if long distance or obstacles, such as heavy traffic, would prevent the individual from obtaining needed care  Medicare pays for ambulance services according to a national fee schedule  The fee schedule establishes seven categories of ground ambulance services and two categories of air ambulance services  Medicare pays for different levels of ambulance services, which reflect the staff training and equipment required to meet the patient's medical condition or health needs  Generally, basic life support is provided by emergency medical technicians (EMTs)  Advanced life support is provided by EMTs with advanced training or by paramedics  Some rural ground and air ambulance services may qualify for increased payments  Also, ambulance providers that are CAHs, or that are entities that are owned and operated by a CAH, are paid on a reasonable-cost basis rather than the fee schedule if they are the only ambulance provider within a 35-mile radius  Medicare covers Part B services in rural health clinics (RHCs) and federally qualified health centers (FQHCs) provided by (1) physicians and specified nonphysician practitioners; (2) visiting nurses for homebound patients in home health shortage areas; (3) registered dieticians or nutritional professionals for diabetes training and medical nutrition therapy; and (4) others, as well as certain drugs administered by a physician or nonphysician practitioner  RHCs are paid based on an "all-inclusive" cost-based rate per beneficiary visit subject to a per visit upper limit, adjusted annually for inflation  For cost-reporting periods that began on or after October 1, 2014, FQHCs are paid a base payment rate per visit (with a limit, in most cases, of one billable visit per day) under a PPS methodology  Each FQHC's PPS rate is adjusted based on the location where the service is furnished using geographic adjustment factors, which are the geographic practice cost indices (GPCI) used in Medicare's physician fee schedule (MPFS)  This rate is increased by 34% for new patients (those not seen in the FQHC organization within the past three years)  The 34% increase also applies when a beneficiary receives a comprehensive initial Medicare visit (an initial preventive physician examination or an initial annual wellness visit) or a subsequent annual wellness visit  Effective January 1, 2017, the FQHC PPS base rate is updated annually using an FQHC-specific market basket  Medicare's payment to the FQHC is equal to 80% of the lesser of the adjusted PPS rate or the FQHC's actual charges associated with the visit, and the beneficiary is responsible for a 20% coinsurance  Medicare Advantage (MA) is an alternative way for Medicare beneficiaries to receive covered benefits  Under MA, private health plans are paid a per-person amount to provide all Medicare covered benefits (except hospice) to beneficiaries who enroll in their plan  Medicare beneficiaries who are eligible for Part A, enrolled in Part B, and do not have ESRD are eligible to enroll in an MA plan if one is available in their area  Some MA plans may choose their service area (local MA plans), while others agree to serve one or more regions defined by the Secretary (regional MA plans)  In 2019, nearly all Medicare beneficiaries have access to an MA plan and approximately a third of beneficiaries are enrolled in one  Private plans may use different techniques to influence the medical care used by enrollees  Some plans, such as health maintenance organizations (HMOs), may require enrollees to receive care from a restricted network of medical providers; enrollees may be required to see a primary care physician who will coordinate their care and refer them to specialists as necessary  Other types of private plans, such as private fee-for-service (PFFS) plans, may look more like original Medicare, with fewer restrictions on the providers an enrollee can see and minimal coordination of care  In general, MA plans offer additional benefits or require smaller co-payments or deductibles than original Medicare  Sometimes beneficiaries pay for these additional benefits through a higher monthly premium, but sometimes they are financed through plan savings  The extent of extra benefits and reduced cost sharing varies by plan type and geography  However, MA plans are seen by some beneficiaries as an attractive alternative to more expensive supplemental insurance policies found in the private market  By contract with CMS, a plan agrees to provide all required services covered in return for a capitated monthly payment adjusted for the demographics and health history of their enrollees  The same monthly payment is made regardless of how many or few services a beneficiary actually uses  In general, the plan is at-risk if costs, in the aggregate, exceed program payments; conversely, the plan can retain savings if aggregate costs are less than payments  Payments to MA plans are based on a comparison of each plan's estimated cost of providing Medicare covered services (a bid) relative to the maximum amount the federal government will pay for providing those services in the plan's service area (a benchmark)  If a plan's bid is less than the benchmark, its payment equals its bid plus a rebate  The size of the rebate is dependent on plan quality and ranges from 50% to 70% of the difference between the bid and the benchmark  The rebate must be returned to enrollees in the form of additional benefits, reduced cost sharing, reduced Part B or Part D premiums, or some combination of these options  If a plan's bid is equal to or above the benchmark, its payment will be the benchmark amount and each enrollee in that plan will pay an additional premium, equal to the amount by which the bid exceeds the benchmark  The MA benchmarks are determined through statutorily specified formulas that have changed over time  Since BBA 97, formulas have increased the benchmark amounts, in part, to encourage plan participation in all areas of the country  As a result, however, the benchmark amounts (and plan payments) in some areas have been higher than the average cost of original FFS Medicare  The ACA changed the way benchmarks are calculated by tying them closer to (or below) spending in FFS Medicare, and adjusting them based on plan quality  In a recent analysis, MedPAC found that "over the past few years, plan bids and payments have come down in relation to FFS spending while MA enrollment continues to grow  The pressure of lower benchmarks has led to improved efficiencies and more competitive bids that enable MA plans to continue to increase enrollment by offering benefits that beneficiaries find attractive " In 2006, the MA program began to offer MA regional plans  Regional MA plans must agree to serve one or more regions designated by the Secretary  There are 26 MA regions consisting of states or groups of states  Regional plan benchmarks include two components: (1) a statutorily determined amount (comparable to benchmarks described above) and (2) a weighted average of plan bids  Thus, a portion of the benchmark is competitively determined  Similar to local plans, plans with bids below the benchmark are given a rebate, while plans with bids above the benchmark require an additional enrollee premium  In general, MA eligible individuals may enroll in any MA plan that serves their area  However, some MA plans may restrict their enrollment to beneficiaries who meet additional criteria  For example, employer-sponsored MA plans are generally only available to the retirees of the company sponsoring the plan  In addition, Medicare Special Needs Plans (SNPs) are a type of coordinated care MA plan that exclusively enrolls, or enrolls a disproportionate percentage of, special needs individuals  Special needs individuals are any MA eligible individuals who are either institutionalized as defined by the Secretary, eligible for both Medicare and Medicaid, or have a severe or disabling chronic condition and would benefit from enrollment in a specialized MA plan  Medicare Part D provides coverage of outpatient prescription drugs to Medicare beneficiaries who choose to enroll in this optional benefit  (As previously discussed, Part B provides limited coverage of some outpatient prescription drugs ) In 2019, about 47 million (about 77%) of eligible Medicare beneficiaries are estimated to be enrolled in a Part D plan  Prescription drug coverage is provided through private prescription drug plans (PDPs), which offer only prescription drug coverage, or through Medicare Advantage prescription drug plans (MA-PDs), which offer prescription drug coverage that is integrated with the health care coverage they provide to Medicare beneficiaries under Part C  Plans must meet certain minimum requirements; however, there are significant variations among them in benefit design, including differences in premiums, drugs included on plan formularies, and cost sharing for particular drugs  Part D prescription drug plans are required to offer either "standard coverage" or alternative coverage that has actuarially equivalent benefits  In 2019, "standard coverage" has a $415 deductible and a 25% coinsurance for costs between $415 and $3,820  From this point, there is reduced coverage until the beneficiary has out-of-pocket costs of $5,100 (an estimated $8,139 54 in total spending); this coverage gap has been labeled the "doughnut hole " Once the beneficiary reaches the catastrophic limit, the program pays all costs except for the greater of 5% coinsurance or $3 40 for a generic drug and $8 50 for a brand-name drug  As required by the ACA, in 2010, Medicare sent a tax-free, one-time $250 rebate check to each Part D enrollee who reached the doughnut hole  Additionally, starting in 2011, the coverage gap is being gradually reduced each year  Under the ACA, the coverage gap for both brand-name and generic drugs was to be eliminated in 2020, but Congress moved up the date to 2019 for brand-name drugs as part of BBA 18  In 2019, a 70% discount is provided by drug manufacturers and Medicare pays an additional 5% of the cost of brand-name drugs dispensed during the coverage gap  In 2019, Medicare also pays 63% of the cost of generic drugs dispensed during the coverage gap and enrollees pay 37%  (See Figure 2  ) By 2020, through a combination of manufacturer discounts and increased Medicare coverage, Part D enrollees will be responsible for 25% of the costs for brand-name and generic drugs in the coverage gap (the same as during the initial coverage period)  Most plans offer actuarially equivalent benefits rather than the standard package, including alternatives such as reducing or eliminating the deductible, or using tiered cost sharing with lower cost sharing for generic drugs  Medicare's payments to plans are determined through a competitive bidding process, and enrollee premiums are tied to plan bids  Plans are paid a risk-adjusted monthly per capita amount based on their bids during a given plan year  Part D plan sponsors determine payments for drugs and are expected to negotiate prices  The federal government is prohibited from interfering in the price negotiations between drug manufacturers, pharmacies, and plans (the so-called "non-interference clause")  Part D also provides enhanced coverage for low-income enrolled individuals, such as persons who previously received drug benefits under Medicaid (known as "dual eligibles"—enrollees in both Medicare and Medicaid)  Additionally, certain persons who do not qualify for Medicaid, but whose incomes are below 150% of poverty, may also receive assistance for some portion of their premium and cost-sharing charges  The MMA included significant incentives for employers to continue to offer coverage to their retirees by providing a 28% federal subsidy  In 2019, the maximum potential subsidy per covered retiree is $2,264 for employers or unions offering drug coverage that is at least actuarially equivalent (called "creditable" coverage) to standard coverage  Employers or unions may select an alternative option (instead of taking the subsidy) with respect to Part D, such as electing to pay a portion of the Part D premiums  They may also elect to provide enhanced coverage, though this has some financial consequences for the employer or union  Alternatively, employers or unions may contract with a PDP or MA-PD to offer the coverage or become a Part D plan sponsor themselves for their retirees  A variety of public and private entities are involved in carrying out Medicare administrative and oversight functions  CMS, an agency within HHS, has primary operational responsibilities  Such responsibilities include managing program finances, developing policies and regulations, setting payment rates, and developing the program's information-technology infrastructure  CMS conducts its activities through its headquarters and 10 regional offices  The Social Security Administration, however, enrolls beneficiaries into the program and issues Medicare beneficiary cards  CMS also contracts with various private entities, including private health insurance companies, to help administer the program  For example, Medicare Administrative Contractors (MACs) process and pay Parts A and B reimbursement claims, enroll providers and suppliers, educate providers and suppliers on billing requirements, support appeal processes, and answer provider and supplier inquiries through call centers, as well as other activities  Qualified Independent Contractors (QICs) perform second-level reviews on appeals initially reviewed by MACs  Medicare's quality assurance activities are primarily handled by State Survey Agencies and Quality Improvement Organizations (QIOs), which operate in all states and the District of Columbia, Puerto Rico, and the U S  Virgin Islands  The State Survey Agencies are responsible for inspecting Medicare provider facilities (e g , nursing homes, home health agencies, and hospitals) to ensure that they are in compliance with federal safety and quality standards referred to as Conditions (or Requirements) of Participation  Alternatively, some types of providers, including hospitals, may receive certification through private accrediting agencies, such as the Joint Commission  QIOs are mostly private, not-for-profit organizations that monitor the quality of care delivered to Medicare beneficiaries and educate providers on the latest quality-improvement techniques  Medicare program integrity activities, such as audits, provider education, medical review, and predictive data analysis, also are carried out by a variety of government and private entities  For example, the Center for Program Integrity within CMS works together with the U S  Department of Justice (DOJ) and the HHS Office of Inspector General (OIG) to identify and prevent fraud, waste, and abuse in Medicare  CMS also works with private contractors to carry out certain program-integrity functions  Unified Program Integrity Contractors (UPICs) perform integrity-related activities including data analysis to identify potentially fraudulent claims (bills) for Medicare Parts A and B, home health and hospice services, and DME  Similarly, the National Benefit Integrity Medicare Drug Integrity Contractor (MEDIC) is responsible for identifying and investigating fraud, waste, and abuse in Medicare Advantage and Part D  When appropriate, the UPICs and MEDIC work with and refer cases to law enforcement, including OIG and DOJ  In addition, Recovery Audit Contractors (RACs) are responsible for identifying improper Medicare payments, including both underpayments and overpayments, and for recouping any overpayments made to providers  Each year, the Comprehensive Error Rate Testing (CERT) program quantifies a national improper Medicare payment rate by examining a random sample of claims  In turn, the Supplemental Medical Review Contractor targets medical reviews in areas where OIG, RACs, and CERT have identified vulnerabilities and/or questionable billing patterns to identify ways to lower improper payment rates  As required by the ACA, the Center for Medicare and Medicaid Innovation (CMMI) was established in January 2011 to test and evaluate innovative payment and service delivery models to reduce program expenditures under Medicare  Examples of these models include providing payment incentives for groups of doctors, hospitals, and other health care providers (Accountable Care Organizations, or ACOs) to coordinate the services they provide to Medicare beneficiaries; bundling payments for services provided in different settings during a beneficiary's episode of care; and reimbursing health providers based on the quality of care rather than on the volume of services  CMMI also plays an important role in developing and implementing the new physician payment models required by MACRA  Medicare beneficiary education and outreach duties are shared between CMS and the Social Security Administration  Each year, CMS mails out a "Medicare and You" handbook to beneficiaries, which provides information on their benefits for the upcoming year  Additional educational materials and responses to frequently asked questions may be found on the CMS-maintained "Medicare gov" website, and beneficiaries may call a CMS-operated 1-800 number for assistance with specific questions and help with selecting and enrolling in a Medicare Advantage and/or Part D plan  A Medicare beneficiary ombudsman is also available to provide assistance to Medicare consumers with their complaints, grievances, and requests  The Social Security Administration is responsible for notifying low-income Medicare beneficiaries about programs that may be able to assist them with their medical and prescription drug expenses  The Social Security Administration also provides general Medicare eligibility and enrollment information on its webpage and on Social Security benefit statements  Finally, CMS partners with community-based organizations, such as State Health Insurance Assistance Programs, in every state to provide educational resources and personalized assistance to Medicare beneficiaries  Medicare's financial operations are accounted for through two trust funds maintained by the Department of the Treasury—the Hospital Insurance (HI) trust fund for Part A and the Supplementary Medical Insurance (SMI) trust fund for Parts B and D  For beneficiaries enrolled in Medicare Advantage (Part C), payments are made on their behalf in appropriate portions from the HI and SMI trust funds  HI is primarily funded by payroll taxes, while SMI is primarily funded through general revenue transfers and premiums  (See Figure 3  ) The HI and SMI trust funds are overseen by a Board of Trustees that provides annual reports to Congress  The trust funds are accounting mechanisms  Income to the trust funds is credited to the fund in the form of interest-bearing government securities  Expenditures for services and administrative costs are recorded against the fund  These securities represent obligations that the government has issued to itself  As long as a trust fund has a balance, the Department of the Treasury is authorized to make payments for it from the U S  Treasury  Medicare expenditures are primarily paid for through mandatory spending—generally Medicare pays for all covered health care services provided to beneficiaries  Aside from certain constraints in HI described below, the program is not subject to spending limits  Additionally, most Medicare expenditures (aside from premiums paid by beneficiaries) are paid for by current workers through income taxes and dedicated Medicare payroll taxes, that is, current income is used to pay current expenditures  Medicare taxes paid by current workers are not set aside to cover their future Medicare expenses  The primary source of funding for Part A is payroll taxes paid by employees and employers  Each pays a tax of 1 45% on the employee's earnings; the self-employed pay 2 9%  Beginning in 2013, some higher-income employees pay higher payroll taxes  Unlike Social Security, there is no upper limit on earnings subject to the tax  Other sources of income include (1) interest on federal securities held by the trust fund, (2) a portion of federal income taxes that individuals pay on their Social Security benefits, and (3) premiums paid by voluntary enrollees who are not automatically entitled to Medicare Part A  Income for Part A is credited to the HI trust fund  Part A expenditures for CY2019 are estimated to reach approximately $330 billion  Revenue to the trust fund is expected to consist of about $285 billion in payroll tax income and another $38 billion in interest and other income  The Medicare Trustees project that in CY2019, the HI trust fund will incur a deficit of approximately $7 billion  As long as the HI trust fund has a balance, the Treasury Department is authorized to make payments for Medicare Part A services  To date, the HI trust fund has never run out of money (i e , become insolvent), and there are no provisions in the Social Security Act that govern what would happen if that were to occur  Part A expenditures exceeded HI income each year from 2008 through 2015, and the assets credited to the trust fund were drawn down to make up the deficit in those years  Although the HI trust fund accumulated small surpluses in 2016 and 2017, the Medicare Trustees estimate that, beginning in 2018, expenditures will outpace income in all future years, and project that the HI trust fund will become insolvent in 2026 (i e , the balance of the trust fund will reach $0)  At that time there would no longer be sufficient funds to fully cover Part A expenditures  Medicare Part B is financed primarily from federal general revenues and from beneficiary premiums, which are set at 25% of estimated per capita program costs for the aged  (The disabled pay the same premium as the aged ) Income for Part B is credited to the SMI trust fund  Total spending for Part B is estimated to reach about $368 billion in CY2019, with premiums financing about $98 billion of that amount and general revenues financing most of the rest  Most beneficiaries who enroll in Medicare Part B pay a monthly premium  Individuals receiving Social Security benefits have their Part B premium payments automatically deducted from their Social Security benefit checks  Due to a "hold-harmless" provision in the Social Security Act, an individual's Social Security check cannot decrease from one year to the next as a result of the annual Part B premium increase (except in the case of higher-income individuals subject to income-related premiums)  The 2019 monthly Part B premium is $135 50  However, about 3 5% of Medicare enrollees are protected by the hold-harmless provision and pay lower premium amounts because the dollar amount of the 2019 cost-of-living increase in their Social Security benefits was not sufficient to cover the full premium increase  Since 2007, higher-income Part B enrollees have paid higher premiums  In 2019, individuals whose modified adjusted gross income (AGI) exceeds $85,000 and each member of a couple filing jointly whose modified AGI exceeds $170,000 are subject to higher premium amounts  These higher-income premiums range from 35% to 85% of the value of Part B and affect about 5% of Medicare enrollees  (See Appendix B for 2019 Part B premiums and high-income thresholds ) Payments for spending under the Medicare Advantage program are made in appropriate portions from the HI and SMI trust funds  There is no separate trust fund for Part C  Medicare Part D is financed through a combination of beneficiary premiums and federal general revenues  In addition, certain transfers are made from the states  These transfers, referred to as "clawback payments," represent a portion of the amounts states could otherwise have been expected to pay for drugs under Medicaid if drug coverage for the dual eligible population had not been transferred to Part D  Part D revenues are credited to a separate Part D account within the SMI trust fund  In CY2019, total spending for Part D is estimated to reach approximately $98 billion, with about $71 billion of that amount paid for by general revenues, $16 billion from beneficiary premiums, and $12 billion from state transfers  In 2019, the base beneficiary premium is $33 19; however, beneficiaries pay different premiums depending on the plan they have selected and whether they are entitled to low-income premium subsidies  Additionally, beginning in 2011, higher income Part D enrollees pay higher premiums  (The income thresholds are the same as for Part B, as described above ) On average, beneficiary premiums are set at 25 5% of expected total Part D costs for basic coverage  The Budget Control Act of 2011 (BCA; P L  112-25 ) provided for increases in the debt limit and established procedures designed to reduce the federal budget deficit, including the creation of a Joint Select Committee on Deficit Reduction  The failure of the Joint Committee to propose deficit reduction legislation by its mandated deadline triggered automatic spending reductions ("sequestration" of mandatory spending and reductions in discretionary spending) in fiscal years 2013 through 2021  The American Taxpayer Relief Act of 2012 (ATRA; P L  112-240 ) delayed the automatic reductions by two months, while the Bipartisan Budget Act of 2013 (BBA; P L  113-67 ) extended sequestration for mandatory spending for an additional two years—through FY2023  In 2014, the President signed into law an amended version of S  25 ( P L  113-82 ), which included a provision to extend BCA's sequester of mandatory spending through FY2024  BBA 15 extended the sequestration of mandatory spending another year, through FY2025  Most recently, BBA 18 extended the BCA mandatory spending sequester through FY2027  Section 256(d) of the Balanced Budget and Emergency Deficit Control Act of 1985 (BBEDCA; P L  99-177 ) contains special rules for the Medicare program in the event of a sequestration  Among other things, it specifies that for Medicare, sequestration is to begin the month after the sequestration order has been issued  Therefore, as the sequestration order was issued March 2013, Medicare sequestration began April 1, 2013, and will continue through March 31, 2028  Under sequestration, Medicare's benefit structure generally remains unchanged; however, benefit related payments are subject to 2% reductions  In other words, most Medicare payments to health care providers, as well as to MA and Part D plans, are being reduced by 2%  Certain Medicare payments are exempt from sequestration and therefore not reduced  These exemptions include (1) Part D low-income subsidies, (2) the Part D catastrophic subsidy, and (3) Qualified Individual (QI) premiums  Some non-benefit related Medicare expenses, such as administrative and operational spending, are subject to higher reductions, 6 2% in 2019  While Medicare provides broad protection against the costs of many, primarily acute care, services, the program does not cover all services that may be used by its aged and disabled beneficiaries  In general, Medicare does not cover eyeglasses, hearing aids, dentures, or most long-term care services  Further, unlike most private insurance policies, it does not include an annual "catastrophic" cap on out-of-pocket spending on cost-sharing charges for services covered under Parts A and B (except for persons enrolled in Medicare Advantage plans)  Most Medicare beneficiaries have some coverage in addition to Medicare  The following are the main sources of additional coverage for Medicare enrollees: Medicare Advantage  Many MA plans offer services in addition to those covered under original Medicare, reduced cost sharing, or reduced Part B or D premiums  All MA plans have a catastrophic cap  Employer Coverage  Coverage may be provided through a current or former employer  In recent years, a number of employers have cut back on the scope of retiree coverage  Some have dropped such coverage entirely, particularly for future retirees  As noted earlier, the MMA attempted to stem this trend, at least for prescription drug coverage, by offering subsidies to employers who offer drug coverage, at least as good as that available under Part D  Medigap  Individual insurance policies that supplement fee-for-service Medicare are referred to as Medigap policies  Beneficiaries with Medigap insurance typically have coverage for a portion of Medicare's deductibles and coinsurance; they may also have coverage for some items and services not covered by Medicare  Individuals select from a set of standardized plans, though not all plans are offered in all states  Medicaid  Certain low-income Medicare beneficiaries also may be eligible for full or partial benefits under their state's Medicaid program  Individuals eligible for both Medicare and Medicaid are referred to as dual eligibles  The lowest-income dual eligibles qualify for full Medicaid benefits, so that the majority of their health care expenses are paid by either Medicare or Medicaid; Medicare pays first, with Medicaid picking up most of the remaining costs  In addition to full-benefit dual eligibles, state Medicaid programs pay Medicare premiums and some cost sharing for other partial dual eligibles, who have higher income than full-benefit dual eligibles but are still considered to have low income  Other Public Sources  Individuals may have additional coverage through the Department of Veterans Affairs, or TRICARE for military retirees eligible for Medicare (and enrolled in Part B)  In 2015, about 87% of Medicare beneficiaries had some form of additional coverage  Some persons may have had more than one type of additional coverage  Appendix A  Abbreviations Appendix B  2019 Medicare Beneficiary Costs

ground_truth: Federal agencies are dependent on information systems to carry out operations. The risks to these systems are increasing as security threats evolve and become more sophisticated. To reduce the risk of a successful cyberattack, agencies can deploy intrusion detection and prevention capabilities on their networks and systems. GAO first designated federal information security as a government-wide high-risk area in 1997. In 2015, GAO expanded this area to include protecting the privacy of personally identifiable information. Most recently, in September 2018, GAO updated the area to identify 10 critical actions that the federal government and other entities need to take to address major cybersecurity challenges. The federal approach and strategy for securing information systems is grounded in the provisions of the Federal Information Security Modernization Act of 2014 and Executive Order 13800. The act requires agencies to develop, document, and implement an agency-wide program to secure their information systems. The Executive Order, issued in May 2017, directs agencies to use the National Institute of Standards and Technology's cybersecurity framework to manage cybersecurity risks. The Federal Cybersecurity Enhancement Act of 2015 contained a provision for GAO to report on the effectiveness of the government's approach and strategy for securing its systems. GAO determined (1) the reported effectiveness of agencies' implementation of the government's approach and strategy; (2) the extent to which DHS and OMB have taken steps to facilitate the use of intrusion detection and prevention capabilities to secure federal systems; and (3) the extent to which agencies reported implementing capabilities to detect and prevent intrusions. To address these objectives, GAO analyzed OMB reports related to agencies' information security practices including OMB's annual report to Congress for fiscal year 2017. GAO also analyzed and summarized agency-reported security performance metrics and IG-reported information for the 23 civilian CFO Act agencies. In addition, GAO evaluated plans, reports, and other documents related to DHS intrusion detection and prevention programs, and interviewed OMB, DHS, and agency officials. The 23 civilian agencies covered by the Chief Financial Officers Act of 1990 (CFO Act) have often not effectively implemented the federal government's approach and strategy for securing information systems (see figure below). Until agencies more effectively implement the government's approach and strategy, federal systems will remain at risk. To illustrate: As required by Office of Management and Budget (OMB), inspectors general (IGs) evaluated the maturity of their agencies' information security programs using performance measures associated with the five core security functions—identify, protect, detect, respond, and recover. The IGs at 17 of the 23 agencies reported that their agencies' programs were not effectively implemented. IGs also evaluated information security controls as part of the annual audit of their agencies' financial statements, identifying material weaknesses or significant deficiencies in internal controls for financial reporting at 17 of the 23 civilian CFO Act agencies. Chief information officers (CIOs) for 17 of the 23 agencies reported not meeting all elements of the government's cybersecurity cross-agency priority goal. The goal was intended to improve cybersecurity performance through, among other things, maintaining ongoing awareness of information security, vulnerabilities, and threats; and implementing technologies and processes that reduce malware risk. Executive Order 13800 directed OMB, in coordination with the Department of Homeland Security (DHS), to assess and report on the sufficiency and appropriateness of federal agencies' processes for managing cybersecurity risks. Using performance measures for each of the five core security functions, OMB determined that 13 of the 23 agencies were managing overall enterprise risks, while the other 10 agencies were at risk. In assessing agency risk by core security function, OMB identified a few agencies to be at high risk (see figure at the top of next page). DHS and OMB facilitated the use of intrusion detection and prevention capabilities to secure federal agency systems, but further efforts remain. For example, in response to prior GAO recommendations, DHS had improved the capabilities of the National Cybersecurity Protection System (NCPS), which is intended to detect and prevent malicious traffic from entering agencies' computer networks. However, the system still had limitations, such as not having the capability to scan encrypted traffic. The department was also in the process of enhancing the capabilities of federal agencies to automate network monitoring for malicious activity through its Continuous Diagnostics and Mitigation (CDM) program. However, the program was running behind schedule and officials at most agencies indicated the need for additional training and guidance. Further, the Federal CIO issued a mandated report assessing agencies' intrusion detection and prevention capabilities, but the report did not address required information, such as the capability of NCPS to detect advanced persistent threats, and a cost/benefit comparison of capabilities to commercial technologies and tools. Selected agencies had not consistently implemented capabilities to detect and prevent intrusions into their computer networks. Specifically, the agencies told GAO that they had not fully implemented required actions for protecting email, cloud services, host-based systems, and network traffic from malicious activity. For example, 21 of 23 agencies had not, as of September 2018, sufficiently enhanced email protection through implementation of DHS' directive on enhanced email security. In addition, less than half of the agencies that use cloud services reported monitoring these services. Further, most of the selected 23 agencies had not fully implemented the tools and services available through the first two phases of DHS's CDM program. Until agencies more thoroughly implement capabilities to detect and prevent intrusions, federal systems and the information they process will be vulnerable to malicious threats. GAO is making two recommendations to DHS, to among other things, coordinate with agencies to identify additional needs for training and guidance. GAO is also making seven recommendations to OMB to, among other things, direct the Federal CIO to update the mandated report with required information, such as detecting advanced persistent threats. DHS concurred with GAO's recommendations. OMB did not indicate whether it concurred with the recommendations or not.