Managing for Results: Government-wide Actions Needed to Improve Agencies' Use of Performance Information in Decision Making
Released: 2018-09-05T00:00:00

Letter

Background

Use of Performance Information in the Federal Government

Concerned that the federal government was more focused on program activities and processes than the results to be achieved, Congress passed the Government Performance and Results Act of 1993 (GPRA). GPRA sought to focus federal agencies on performance by requiring agencies to develop long-term and annual goals, and measure and report on progress towards those goals annually. Based on our analyses of the act’s implementation, we concluded in March 2004 that GPRA’s requirements had laid a solid foundation for results-oriented management. At that time, we found that performance planning and measurement had slowly yet increasingly become a part of agencies’ cultures. For example, managers reported having significantly more performance measures in 2003 than in 1997, when GPRA took effect government-wide.

However, the benefit of collecting performance information is fully realized only when that information is actually used by managers to make decisions aimed at improving results. Although our 2003 survey found greater reported availability of performance information than in 1997, it also showed managers’ use of that information for various management activities generally had remained unchanged. Based on those results, and in response to a request from Congress, in September 2005, we developed a framework intended to help agencies better incorporate performance information into their decision making. As shown in figure 1, we identified five leading practices that can promote the use of performance information for policy and program decisions; and four ways agency managers can use performance information to make program decisions aimed at improving results.

Our September 2005 report also highlighted examples of how agencies had used performance information to improve results. For example, we described how the Department of Transportation’s National Highway Traffic Safety Administration used performance information to identify, develop, and share effective strategies that increased national safety belt usage—which can decrease injuries and fatalities from traffic accidents— from 11 percent in 1985 to 80 percent in 2004.

Subsequently, the GPRA Modernization Act of 2010 (GPRAMA) was enacted, which significantly expanded and enhanced the statutory framework for federal performance management. The Senate Committee on Homeland Security and Governmental Affairs report accompanying the bill that would become GPRAMA stated that agencies were not consistently using performance information to improve their management and results. The report cited the results of our 2007 survey of federal managers. That survey continued to show little change in managers’ use of performance information. The report further stated that provisions in GPRAMA are intended to address those findings and increase the use of performance information to improve performance and results. For example, GPRAMA requires certain agencies to designate a subset of their respective goals as their highest priorities—known as agency priority goals—and to measure and assess progress toward those goals at least quarterly through data-driven reviews.

Our recent work and surveys suggest that data-driven reviews are having their intended effect. For example, in July 2015, we found that agencies reported that their reviews had positive effects on progress toward agency goals and efforts to improve the efficiency of operations, among other things. In addition, for those managers who were familiar with their agencies’ data-driven reviews, our 2013 and 2017 surveys showed that the more managers viewed their programs as being subject to a review, the more likely they were to report their agencies’ reviews were driving results and conducted in line with our leading practices. Recognizing the important role these reviews were playing in improving data-driven decision making, our management agenda for the presidential and congressional transition in 2017 included a key action to expand the use of data-driven reviews beyond agency priority goals to other agency goals.

More broadly, our recent surveys of federal managers have continued to show that reported government-wide uses of performance information generally have not changed or in some cases have declined. As we found in September 2017, and as illustrated in figure 2, the 2017 update to our index suggests that government-wide use of performance information did not improve between 2013 and 2017. In addition, it is statistically significantly lower relative to our 2007 survey, when we created the index.

Moreover, in looking at the government-wide results on the 11 individual survey questions that comprise the index, we found few statistically significant changes in 2017 when compared to (1) our 2013 survey or (2) the year each question was first introduced. For example, in comparing 2013 and 2017 results, two questions had results that were statistically significantly different:

The percentage of managers who reported that employees who report to them pay attention to their agency’s use of performance information was statistically significantly higher (from 40 to 46 percent).

The percentage of managers who reported using performance information to adopt new program approaches or change work processes was statistically significantly lower (from 54 to 47 percent).

As we stated in our September 2017 report, the decline on the latter question was of particular concern as agencies were developing plans to improve their efficiency, effectiveness, and accountability, as called for by an April 2017 memorandum from OMB.

The Administration’s Plans for Federal Performance Management

In early 2017, the administration announced several efforts intended to improve government performance. OMB issued several memorandums detailing the administration’s plans to improve government performance by reorganizing the government, reducing the federal workforce, and reducing federal agency burden. As part of the reorganization efforts, OMB and agencies were to develop government-wide and agency reform plans, respectively, designed to leverage various GPRAMA provisions. For instance, the April 2017 memorandum mentioned above stated that OMB intends to monitor implementation of the reforms using, among other things, agency priority goals. While many agency-specific organizational improvements were included in the President’s fiscal year 2019 budget, released in February 2018, OMB published additional government-wide and agency reform proposals in June 2018.

The President’s Management Agenda (PMA), released in March 2018, outlines a long-term vision for modernizing federal operations and improving the ability of agencies to achieve outcomes. To address the issues outlined in the PMA, the administration established a number of cross-agency priority (CAP) goals. CAP goals, required by GPRAMA, are to address issues in a limited number of policy areas requiring action across multiple agencies, or management improvements that are needed across the government.

The PMA highlights several root causes for the challenges the federal government faces. Among them is that agencies do not consistently apply data-driven decision-making practices. The PMA states that smarter use of data and evidence is needed to orient decisions and accountability around service and results. To that end, in March 2018, the administration established the Leveraging Data as a Strategic Asset CAP goal to improve the use of data in decision making to increase the federal government’s effectiveness.

Federal Performance Management Leadership Roles and Responsibilities

Over the past 25 years, various organizations, roles, and responsibilities have been created by executive action or in law to provide leadership in federal performance management. At individual agencies and across the federal government, these organizations and officials have key responsibilities for improving performance, as outlined below.

OMB: At least every four years, OMB is to coordinate with other agencies to develop CAP goals—such as the one described earlier on leveraging data as an asset—to improve the performance and management of the federal government. OMB is also required to coordinate with agencies to develop annual federal government performance plans to define, among other things, the level of performance to be achieved toward the CAP goals. Following GPRAMA’s enactment, OMB issued guidance for initial implementation, as required by the act, and continues to provide updated guidance in its annual Circular No. A-11, additional memorandums, and other means.

Chief Operating Officer (COO): The deputy agency head, or equivalent, is designated as the COO, with overall responsibility for improving agency management and performance through, among other things, the use of performance information.

President’s Management Council (PMC): The PMC is comprised of OMB’s Deputy Director for Management and the COOs of major departments and agencies, among other individuals. Its responsibilities include improving overall executive branch management and implementing the PMA.

Performance Improvement Officer (PIO): Agency heads designate a senior executive as the PIO, who reports directly to the COO. The PIO is responsible for assisting the head of the agency and COO to ensure that agency goals are achieved through, among other things, the use of performance information.

Performance Improvement Council (PIC): The PIC is charged with assisting OMB to improve the performance of the federal government. It is chaired by the Deputy Director for Management at OMB and includes PIOs from each of the 24 Chief Financial Officers Act agencies, as well as other PIOs and individuals designated by the chair. Among its responsibilities, the PIC is to work to resolve government-wide or cross-cutting performance issues, and facilitate the exchange among agencies of practices that have led to performance improvements.

Previously, the General Service Administration’s (GSA) Office of Executive Councils provided analytical, management, and administrative support for the PIC, the PMC, and other government-wide management councils. In January 2018, the office was abolished and its functions, staff, and authorities, along with those of the Unified Shared Services Management Office, were reallocated to GSA’s newly created Shared Solutions and Performance Improvement Office.

Agencies’ Use of Performance Information in Decision Making and Related Leading Practices Generally Has Not Improved

Reported Use of Performance Information in Decision Making Generally Has Not Improved at Individual Agencies Since 2013

As at the government-wide level—where, as described earlier, the use of performance information did not change from 2013 to 2017—managers’ reported use of performance information at most agencies also did not improve since 2013 (illustrated in figure 3).

At the agency level, 3 of the 24 agencies had statistically significant changes in their index scores—1 increase (National Science Foundation) and 2 decreases (Social Security Administration and the Office of Personnel Management). Also, in 2017, 6 agencies had results that were statistically significantly different—4 higher and 2 lower—than the government-wide average (see sidebar).

Throughout the report, we highlight two different types of statistically significant results—changes from our last survey in 2013 and differences from the 2017 government-wide average. The former indicates when an agency’s reported use of performance information or leading practices has measurably improved or declined. The latter indicates when it is statistically significantly higher or lower than the rest of government. These results suggest agencies have taken actions that led to improvements in their use of performance information. For example, when a result is a statistically significant increase since 2013, as with the National Science Foundation index score in 2017, this suggests that the agency has adopted practices that led to a measurable increase in the use of performance information by managers.

When a result is statistically significantly higher than the government-wide average, like GSA’s 2017 index score, this suggests that the agency’s use of performance information is among the highest results when compared to the rest of government. These agencies could also have insights into practices that led to relatively high levels of performance information use. Finally, when a result is a statistically significant decrease since 2013, as with the Social Security Administration’s index score in 2017, or statistically significantly lower than the government-wide average, like the Department of Homeland Security’s 2017 index score, this suggests the agencies face challenges that are hampering their ability to use performance information. Appendix III provides each agency’s index scores from 2007, 2013, and 2017 to show changes between survey years.

When we disaggregated the index and analyzed responses from the 11 questions that comprise the index—which could help pinpoint particular actions that improved the use of performance information—we similarly found relatively few changes in agencies’ recent results. Specifically, we identified 16 instances where agency responses on individual questions were statistically significantly different from 2013 to 2017—10 increases and 6 decreases. This represents about 6 percent of the total possible responses to the 11 survey questions from each of the agencies.

In addition, we found 12 instances where an agency’s result on a question was statistically significantly higher (11) or lower (1) than the government-wide average in 2017. For example, the percentage of Social Security Administration (SSA) managers reporting that their peers use performance information to share effective approaches was statistically significantly higher than the government-wide average. Although SSA’s index score had a statistically significant decline in 2017 compared to 2013, the agency’s index score remains relatively high, as it has in prior years. The scope of our work has not allowed us to determine definitively what factors caused the decline in SSA’s index score and whether the decline is likely to continue, although its result on this particular question may indicate a continued strength. Each agency’s results on the 11 questions that comprise the index are presented in appendix I. The agencies’ respective statistically significant results are identified in figure 4.

While some agencies had statistically significant improvements on individual questions, and could point to actions that led to improvements in their use of performance information, these improvements should be considered in relation to the range of agency results and the government- wide average. In figure 4, there are five agencies with statistically significant increases on responses to individual questions, where those results were not statistically significantly higher than the government-wide average (see arrows without plus signs for the Departments of Agriculture, Defense, and Justice; the Environmental Protection Agency; and the National Science Foundation). While these represent improvements, they should be considered in relation to the range of agency results and the government-wide average (provided in detail in the agency summaries in appendix I). For example, in 2017, the percentage of managers at the Department of Agriculture who reported that upper management use performance information to inform decisions about program changes was statistically significantly higher than in 2013. However, the department’s 2017 result (37 percent) was relatively lower when compared to the maximum agency result on that question (60 percent). Appendix I presents the results on the index and the 11 questions that comprise it for each of the 24 agencies.

Individual Agencies’ Reported Use of Leading Practices Generally Remains Unchanged

When we compared government-wide and agency-level results on selected survey questions that reflect practices that promote the use of performance information, we found that results between 2013 and 2017 generally remained unchanged. As described earlier, there are 10 survey questions that both reflect the five leading practices identified in our past work and had statistically significant associations with higher index scores. As shown in figure 5, government-wide results on 2 of the 10 questions were statistically significantly different, both increases, from 2013 to 2017. Despite these two increases, the overall results suggest these practices are not widely followed government-wide. On most of the 10 questions, only about half (or fewer) of the managers reported their agencies were following them to a “great” or “very great” extent.

When we analyzed agency-level responses to these 10 questions, we also found relatively few changes in recent results. Specifically, our analysis found 20 instances—16 increases and 4 decreases—where agencies’ responses on individual questions were statistically significantly different from 2013 to 2017. This represents about 8 percent of the total possible responses to the 10 survey questions from each of the agencies.

In addition, we found 10 instances where an agency’s result on a question was statistically significantly higher (8) or lower (2) than the government-wide average in 2017. Each agency’s results on these 10 questions are presented in appendix I, and the statistically significant results are identified in figure 6.

Those agencies with results on individual questions that are either statistically significantly higher than 2013, higher than the 2017 government-wide average, or both may have taken actions in line with our leading practices for promoting the use of performance information. For example, the National Science Foundation had both types of statistically significant results on a question about having sufficient information on the validity of their performance data. Here, the agency’s result increased 27 percentage points from 2013 to 2017. While the scope of our review does not allow us to definitively determine the reasons for the National Science Foundation’s higher results, they suggest the agency has taken recent actions that greatly improved the availability and accessibility of information on the validity of performance data. In both 2013 and 2017, our analyses found this particular question to be the strongest predictor of higher performance information use when we tested for associations between the questions that reflect leading practices and our index.

Managers Whose Programs Were Subject to Data- Driven Reviews Reported Greater Use of Performance Information and Leading Practices

Our 2017 survey results show that managers who reported their programs were subject to data-driven reviews also were more likely to report using performance information in decision making to a greater extent (see figure 7). For the 35 percent of managers who reported being familiar with data-driven reviews, those who reported their programs had been subject to data-driven reviews to a “great” or “very great” extent had index scores that were statistically significantly higher than those whose programs were subject to these reviews to a lesser extent.

Similarly, we found that being subject to data-driven reviews to a greater extent was also related to greater reporting of agencies following practices that can promote the use of performance information. As figure 8 shows, managers who reported their programs were subject to these reviews to a “great” or “very great” extent more frequently reported that their agencies followed the five leading practices that promote the use of performance information, as measured by the 10 related survey questions associated with higher scores on the index. For example, of the estimated 48 percent of managers who reported their programs were subject to data-driven reviews to a “great” or “very great” extent, 72 percent also reported that managers at their level (peers) effectively communicate performance information on a routine basis to a “great” or “very great” extent. Conversely, for the 24 percent of managers who reported their programs were subject to data-driven reviews to a “small” or “no” extent, only 30 percent reported that managers at their level do this to a “great” or “very great” extent.

Opportunities Exist for the Executive Branch to Increase the Use of Performance Information within Agencies

Disparities Exist in the Use of Performance Information by Senior Agency Leaders and Managers at Lower Levels

Our past work has found that the Executive Branch has taken steps to improve the use of performance information in decision making by senior leaders at federal agencies. However, our survey results indicate those steps have not led to similar improvements in use by managers at lower levels. Through its guidance to implement GPRAMA, OMB developed a framework for performance management in the federal government that involves agencies setting goals and priorities, measuring performance, and regularly reviewing and reporting on progress. This includes expectations for how agency senior leaders should use performance information to assess progress towards achieving agency priority goals through data-driven reviews, and strategic objectives through strategic reviews. For example, GPRAMA requires, and OMB’s guidance reinforces, that data-driven reviews should involve the agency head, Chief Operating Officer, Performance Improvement Officer, and other senior officials responsible for leading efforts to achieve each goal.

OMB’s guidance also identifies ways in which agency leaders should use the results of those reviews to inform various decision-making activities, such as revising strategies, formulating budgets, and managing risks. Our past work also found that agencies made progress in implementing these reviews and using performance information. In July 2015, we found that agencies generally were conducting their data-driven reviews in line with GPRAMA requirements and our related leading practices, including that agency leaders used the reviews to drive performance improvement. In addition, in September 2017, we reported on selected agencies’ experiences in implementing strategic reviews and found that the reviews helped direct leadership attention to progress on strategic objectives.

Despite those findings, our survey results continue to show that the reported use of performance information by federal managers has generally not improved, and actually declined at some agencies. This could be because of the two different groups of agency officials covered by our work. GPRAMA’s requirements, and the federal performance management framework established by OMB’s guidance, apply at the agency-wide level and generally involve senior leaders. Our past work reviewing implementation of the act therefore focused on improvements in the use of performance information by senior leaders at the agency- wide level. In contrast, our surveys covered random samples of mid- and upper-level managers within those agencies, including at lower organizational levels such as component agencies. Their responses indicate that the use of performance information more broadly within agencies—at lower organizational levels—generally has not improved over time. The exception to this was managers whose programs were subject to the data-driven reviews required by GPRAMA. As described above, those managers were more likely to report greater use of performance information in their agencies. This reinforces the value of the processes and practices put in place by GPRAMA. Our survey results suggest that limited actions have been taken to diffuse processes and practices related to the use of performance information to lower levels within federal agencies, where mid-level and senior managers make decisions about managing programs and operations.

Although OMB staff agreed that diffusing processes and practices to lower levels could lead to improved use of performance information, they told us they have not directed agencies to do so for a few reasons. First, OMB staff expressed concerns about potentially imposing a “one-size-fits- all” approach on agencies. They stated that agencies are best positioned to improve their managers’ use of performance information, given their individual and unique missions and cultures, and the environments in which they operate. We agree that it makes sense for agencies to be able to tailor their approaches for those reasons. OMB’s existing guidance provides an overarching framework that recognizes the need for flexibility and for agencies to tailor their approaches. Moreover, given the long- standing and cross-cutting nature of this challenge, a government-wide approach also would provide a consistent focus on improving the use of performance information more extensively within agencies.

OMB staff also told us that they believed it would go beyond their mandate to direct agencies to extend GPRAMA requirements to lower levels. GPRAMA requires OMB to provide guidance to agencies to implement its requirements, which only apply at the agency-wide level. As noted earlier, however, GPRAMA also requires OMB to develop cross- agency priority (CAP) goals to improve the performance and management of the federal government. The President’s Management Agenda established a CAP goal to leverage data as a strategic asset, in part, to improve the use of data for decision making and accountability throughout the federal government. This new CAP goal presents an opportunity for OMB and agencies to identify actions to expand the use of performance information in decision making throughout agencies.

Plan for New CAP Goal Does Not Yet Contain Required Elements for Successful Implementation

As of June 2018, the action plan for implementing the Leveraging Data as a Strategic Asset CAP goal is limited. According to the President’s Management Agenda and initial CAP goal action plan, the goal primarily focuses on developing and implementing a long-term, enterprise-wide federal data strategy to better govern and leverage the federal government’s data. It is through this strategy that, among other things, the administration intends to improve the use of data for decision making and accountability. However, the strategy is under development and not expected to be released until January 2019, with a related plan to implement it expected in April 2019.

The existing action plan, released in March 2018 and updated in June 2018, does not yet include specific steps needed to improve the use of data—including performance information—more extensively within agencies. According to the action plan for the goal, potential actions currently under consideration focus on establishing agency “learning agendas” that prioritize the development and use of data and other evidence for decision-making; building agency capacity to use data and other evidence; and improving the timeliness of performance information and other data, and making that information available to decision makers and the public.

Although developing learning agendas and building capacity could help improve the use of performance information in agencies, improving availability of data may be less effective. For example, as our past survey results have shown, increasing the availability of performance information has not resulted in corresponding increases in its use in decision making.

We recognize that the CAP goal was created in March 2018. Nonetheless, it is important that OMB and its fellow goal leaders develop the action plan and related federal data strategy consistent with all key requirements to better ensure successful implementation. The action plan does not yet include complete information related to the following GPRAMA requirements: performance goals that define the level of performance to be achieved each year for the CAP goal; the various federal agencies, organizations, programs, and other activities that contribute to the CAP goal; performance measures to assess overall progress towards the goal as well as the progress of each agency, program, and other activity contributing to the goal; and clearly defined quarterly targets.

Consistent with GPRAMA, Standards for Internal Control in the Federal Government identifies information that agencies are required to include in their plans to help ensure they achieve their goals. The standards state that objectives—such as improving the use of data in decision making— should be clearly defined to enable the identification of risks. Objectives are to be defined in specific terms so they can be understood at all levels of the entity—in this case, government-wide as well as within individual agencies. This involves defining what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement.

Ensuring that future updates to the new CAP goal’s action plan includes all required elements is particularly important, as our previous work has found that some past CAP goal teams did not meet all planning and reporting requirements. For example, in May 2016 we found that most of the CAP goal teams we reviewed had not established targets for all performance measures they were tracking. This limited the transparency of their efforts and the ability to track progress toward established goals. We recommended that OMB, working with the Performance Information Council (PIC), report on actions that CAP goal teams are taking, or plan to take, to develop such targets and performance measures. OMB staff generally agreed and, in July 2017, told us they were working, where possible, to assist the development of measures for CAP goals. However, the recommendation has not been addressed and OMB staff said the next opportunity to address it would be when the administration established new CAP goals (which took place in March 2018). Following the initial release of the new CAP goals, CAP goal teams are to more fully develop the related action plans through quarterly updates. Given the ongoing importance of meeting these planning and reporting requirements, we will continue to monitor the status of actions to address this recommendation as implementation of the new CAP goals proceeds.

Our Survey Results Identify Additional Opportunities for the PIC to Improve Federal Use of Performance Information

While the PIC, which is chaired by OMB, has contributed to efforts to enhance the use of performance information, our survey results identify additional opportunities to further those efforts. The PIC’s past efforts have included hosting various working groups and learning events for agency officials to provide performance management guidance, and developing resources with relevant practices. For example, the PIC created a working group focused on agency performance reviews, which was used to share recommendations for how agencies can implement reviews, along with a guide with practices for effectively implementing strategic reviews. In January 2018, staff supporting the PIC joined with staff from another GSA office to create a new group called Fed2Fed Solutions. This group consults with agencies and provides tailored support, such as data analysis and performance management training for agency officials, to help them address specific challenges related to organizational transformation, data-driven decision making, and other management improvement efforts.

Our survey results identify useful information related to potential promising practices and challenges that OMB and the PIC could use to inform efforts to enhance the use of performance information more extensively within agencies (e.g., at lower levels). As was previously described, the PIC has responsibilities to (1) facilitate the exchange among agencies of proven practices, and (2) work to resolve government- wide or cross-cutting performance issues, such as challenges. Our analyses of 2017 survey results identified instances where agencies may have found effective ways to enhance the use of performance information by agency leaders and managers in decision making, as well as instances where agencies (and their managers) face challenges in doing so.

Specifically, based on analyses of our survey responses, we identified 14 agencies that may have insights into specific practices that led to recent improvements in managers’ use of performance information, or ways that they maintain relatively high levels of use by their managers when compared to the rest of the government. Figure 9 summarizes the agencies identified earlier in the report that had statistically significant increases, or results higher than the government-wide average, on our index or individual survey questions. As the figure shows, several agencies had statistically significant results across all three sets of analyses and therefore may have greater insights to offer: the General Services Administration, National Aeronautics and Space Administration, and the National Science Foundation.

In addition, our analyses identified nine agencies where results suggest managers face challenges that have hampered their ability to use performance information. Figure 10 summarizes the agencies identified earlier in the report that had statistically significant decreases, or results lower than the government-wide average, on our index or individual survey questions. As the figure shows, the Office of Personnel Management had statistically significant decreases in all three sets of analyses.

Four agencies—the Departments of the Treasury and Veterans Affairs, the Nuclear Regulatory Commission, and the Social Security Administration—were common to both of the figures above. That is, they had results that indicate they may have insights on some aspects of using performance information and face challenges in other aspects. As was mentioned earlier, to provide proper context, these results should be considered in relation to the range of agency results and the government- wide average (provided in detail in the agency summaries in appendix I).

Given the prioritization of other activities, such as the recent creation of the Fed2Fed Solutions program, the PIC has not yet undertaken a systematic approach that could improve the use of performance information by managers at lower levels within agencies. Such an approach would involve identifying and sharing practices that have led to improved use, as well as identifying common or cross-cutting challenges that have hampered such use. The results of our analyses could help the PIC do so, and in a more targeted manner. By identifying and sharing proven practices, the PIC could further ensure that agency leaders and managers are aware of effective or proven ways they can use performance information to inform their decisions across the spectrum of activities they manage within their agencies. Those proven practices also may help agency leaders and managers resolve any identified challenges.

Furthermore, in September 2017, we found that, for the estimated 35 percent of managers who reported familiarity with data-driven reviews, the more they viewed their programs being subject to a review, the more likely they were to report the reviews were driving results and were conducted in line with our leading practices for using performance information. Despite the reported benefits of and results achieved through data-driven reviews, they were not necessarily widespread. As noted above, GPRAMA requires agencies to conduct such reviews for agency priority goals, which represent a small subset of goals, and they are required at the departmental level. These reasons may explain why most managers reported they were not familiar with the reviews.

As a result, we recommended that OMB should work with the PIC to identify and share among agencies practices for expanding the use of data-driven reviews. OMB staff agreed with our recommendation but have yet to address it. In June 2018, OMB updated its annual guidance to agencies to explicitly encourage them to expand data-driven reviews to include other goals, priorities, and management areas as applicable to improve organizational performance. However, as of June 2018, OMB and the PIC have yet to take any steps to identify and share practices for expanding the use of these reviews in line with our recommendation. Given the additional analyses we conducted for this report—which show that being subject to data-driven reviews is related to greater reported use of performance information and leading practices that promote such use—we continue to believe these further actions would help agencies implement these reviews more extensively. We reiterate the importance of the September 2017 recommendation and will continue to monitor OMB’s progress to address it.

Conclusions

For more than 20 years, our work has highlighted weaknesses in the use of performance information in federal decision making. While the Executive Branch has taken some actions in recent years, such as establishing a framework for performance management across the federal government, our survey results underscore that more needs to be done to improve the use of performance information more extensively within agencies and government-wide. The President’s Management Agenda and its related CAP goal to leverage data as a strategic asset present an opportunity to do so, as it aims to improve data-driven decision making. As OMB and its fellow goal leaders more fully develop the action plan for achieving this goal, providing additional details for its plans to improve data-driven decision making would help provide assurance that it can be achieved.

As part of those initiatives, our survey results could provide a useful guide for targeting efforts. Officials at each agency could use these results to identify areas for additional analysis and potential actions that could help improve the use of performance information across the agency and at lower levels. Similarly, OMB and the PIC could use the results to identify broader issues in need of government-wide attention. It will also be important, however, for OMB and the PIC to go beyond this analysis and work with agencies to identify and share proven practices for increasing the use of performance information at lower levels within agencies, as well as challenges that may be hampering agencies’ ability to do so.

Recommendations for Executive Action

We are making the following two recommendations to OMB: The Director of OMB should direct the leaders of the Leveraging Data as a Strategic Asset CAP Goal to ensure future updates to the action plan, and the resulting federal data strategy, provide additional details on improving the use of data, including performance information, more extensively within federal agencies. The action plan should identify performance goals; contributing agencies, organizations, programs, and other activities; those responsible for leading implementation within these contributors; planned actions; time frames; and means to assess progress. (Recommendation 1)

The Director of OMB, in coordination with the PIC, should prioritize efforts to identify and share among agencies proven practices for increasing, and challenges that hamper, the use of performance information in decision making more extensively within agencies. At a minimum, this effort should involve the agencies that our survey suggests may offer such insights. (Recommendation 2)

Agency Comments

We provided a draft of this report to the Director of the Office of Management and Budget for review and comment. We also provided a draft of the report to the heads of each of the 24 federal agencies covered by our survey.

OMB had no comments, and informed us that it would assess our recommendations and consider how best to respond.

We are sending copies of this report to congressional requesters, the Director of the Office of Management and Budget, the heads of each of the 24 agencies, and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mcneilt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix IV.

Appendix I: Summaries of Agency Survey Results

U.S. Department of Agriculture

(USDA) (Goverment-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Commerce

(Commerce) (Goverment-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to 5 Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Defense

(Goverment-wide) (DOD)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Education

(Education) (Goverment-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Energy

(Government-wide) (Energy)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Health & Human Services

(HHS) (Goverment-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Partices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Department of Homeland Security

(DHS) (Goverment-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Housing and Urban Development

(Government-wide) (HUD)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Department of the Interior

(Interior) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Justice

(DOJ) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Labor

(DOL) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Department of State

(State) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Transportation

(DOT) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of the Treasury

(Treasury) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Department of Veterans Affairs

(Government-wide) (VA)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

U.S. Agency for International Development

(Goverment-wide) (USAID)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Environmental Protection Agency

(Government-wide) (EPA)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

General Services Administration

(Government-wide) (GSA)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

National Aeronautics and Space Administration

(Government-wide) (NASA)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

National Science Foundation

(Government-wide) (NSF)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Nuclear Regulatory Commission

(NRC) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Office of Personnel Management

(Government-wide) (OPM)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Partices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Small Business Administration

(SBA) (Government-wide)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Social Security Administration

(Government-wide) (SSA)

11 Questions that Comprise the Index

Performance information is used to:

10. The individual I report to 11. Employees that report to me 0 Percent of managers reporting “Great” or “Very Great”

10 Questions Related to Leading Practices That Promote the Use of Performance Information

Leading Practices:

Aligning agencywide goals, objectives, and measures

Improving the usefulness of performance information

Communicate performance information frequently and effectively

Appendix II: Objectives, Scope, and Methodology

This report responds to a request that we analyze agency-level results from our 2017 survey of federal managers at the 24 agencies covered by the Chief Financial Officers (CFO) Act of 1990, as amended, to determine the extent agencies are using performance information. This report assesses the extent to which: 1. the reported use of performance information and related leading practices at 24 agencies has changed compared to our prior survey in 2013; 2. being subject to data-driven reviews related to managers’ reported use of performance information and leading practices; and 3. the Executive Branch has taken actions to enhance agencies’ use of performance information in various decision-making activities.

From November 2016 through March 2017, we administered our online survey to a stratified random sample of 4,395 individuals from a population of 153,779 mid- and upper-level civilian managers and supervisors at the 24 CFO Act agencies. The management levels covered general schedule (GS) or equivalent schedules at levels comparable to GS-13 through GS-15, and career Senior Executive Service (SES) or equivalent. We obtained the sample from the Office of Personnel Management’s Enterprise Human Resources Integration database as of September 30, 2015—the most recent fiscal year data available at the time. The sample was stratified by agency and whether the manager or supervisor was a member of the SES. To help determine the reliability and accuracy of the database elements used to draw our sample of federal managers for the 2017 survey, we checked the data for reasonableness and the presence of any obvious or potential errors in accuracy and completeness and reviewed our past analyses of the reliability of this database. We concluded in our September 2017 report that the data used to draw our sample were sufficiently reliable for the purpose of the survey. For the 2017 survey, we received usable questionnaires from about 67 percent of the eligible sample. The weighted response rate at each agency generally ranged from 57 percent to 82 percent, except the Department of Justice, which had a weighted response rate of 36 percent. The overall survey results are generalizable to the population of managers government-wide and at each individual agency.

To assess the potential bias from agencies with lower response rates, we conducted a nonresponse bias analysis using information from the survey and sampling frame as available. The analysis confirmed discrepancies in the tendency to respond to the survey related to agency and SES status. The analysis also revealed some differences in response propensity by age and GS level; however, the direction and magnitude of the differences on these factors were not consistent across agencies or strata. Our data may be subject to bias from unmeasured sources for which we cannot control. Results, and in particular estimates from agencies with low response rates such as the Department of Justice, should be interpreted with caution. However, the survey’s results are comparable to five previous surveys we conducted in 1997, 2000, 2003, 2007, and 2013.

To address the first objective, we used data from our 2017 survey to update agency scores on our use of performance information index. This index, which was last updated using data from our 2013 survey, averages managers’ responses on 11 questions related to the use of performance information for various management activities and decision making. Using 2017 survey data, we conducted statistical analyses to ensure these 11 questions were still positively correlated. That analysis confirmed that no negative correlations existed and therefore no changes to the index were needed. Figure 11 shows the questions that comprise the index.

After calculating agency index scores for 2017, we compared them to previous results from 2007 and 2013, and to the government-wide average for 2017, to identify any statistically significant differences. We focus on statistically significant results because these indicate that observed relationships between variables and differences between groups are likely to be valid, after accounting for the effects of sampling and other sources of survey error. For each of the 11 questions that comprise the index, we identified individual agency results, excluding missing and no basis to judge responses, and determined when they were statistically significantly different from (1) the agency’s results on the same question in 2013, or (2) the government-wide average results on the question in 2017. In this report, we analyzed and summarized the results of our 2017 survey of federal managers. Due to the limited scope of the engagement, we did not conduct additional audit work to determine what may have caused statistically significant changes between our 2017 and past survey results.

To further address this objective we completed several statistical analyses that allowed us to assess the association between the index and 22 survey questions that we determined relate to leading practices we previously found promote the use of performance information. See figure 12 for the 22 specific questions related to these five practices that we included in the analysis.

When we individually tested these 22 survey questions (bivariate regression), we found that each was statistically significantly and positively related to the index in 2017. This means that each question, when tested in isolation from other factors, was associated with higher scores on the index. However, when all 22 questions were tested together (multivariate regression), we found that 5 questions continued to be positively and significantly associated with the index in 2017, after controlling for other factors.

To conduct this multivariate analysis, we began with a base model that treated differences in managers’ views of agency performance management use as a function of the agency where they worked. We found, however, that a model based on agency alone had little predictive power (R-squared of 0.04). We next examined whether managers’ responses to these questions reflecting practices that promote the use of performance information related to their perceptions of agency use of performance information, independent of agency. The results of this analysis are presented in table 1 below. Each coefficient reflects the increase in our index associated with a one-unit increase in the value of a particular survey question.

Our final multivariate regression model had an R-squared of 0.67, suggesting that the variables in this model explain approximately 67 percent of the variation in the use index. We also tested this model controlling for whether a respondent was a member of the SES and found similar results.

As shown above in table 1, five questions related to three of the leading practices that promote agencies’ use of performance information were statistically significant in 2017. These results suggest that, when controlling for other factors, certain specific efforts to increase agency use of performance information—such as providing information on the validity of performance data—may have a higher return and lead to higher index scores.

With respect to aligning agency-wide goals, objectives, and measures, we found that each increase in terms of the extent to which individuals felt that managers aligned performance measures with agencywide goals and objectives was associated with a 0.08 increase in their score on the use index.

In terms of improving the usefulness of performance information, we found that having information on the validity of performance data for decision making was the strongest predictor in our model (0.18). As measured here, taking steps to ensure the performance information is useful and appropriate was associated with almost as large a change in a managers’ index score (0.16).

In terms of developing agency capacity to use performance information, we found that having sufficient analytical tools to collect, analyze, and use performance information (0.07), and providing or paying for training that would help link their programs to achievement of agency strategic goals (0.10), were also statistically significantly related to a manager’s reported use of performance information.

When we combined these results with what we previously found through a similar analysis of 2013 survey results in September 2014, we identified 10 questions that have had a statistically significant association with higher index scores. This reinforces the importance of the five leading practices to promote the use of performance information. For each of these questions, which are outlined in figure 13 below, we determined when agency results were statistically significantly different from 2013 results or the 2017 government-wide average.

For the second objective, we examined, based on the extent they responded their programs had been subject to agency data-driven reviews, differences in managers’ use index scores and responses on questions related to practices that promote the use of performance information. We grouped managers based on the extent they reported their programs had been subject to these reviews, from “no extent” through “very great extent.” We then calculated the average index scores for the managers in each of those five categories.

We also examined differences in how managers responded to the 10 questions reflecting practices that can promote the use of performance information based on the extent they reported their programs had been subject to data-driven reviews. We grouped managers into three categories based on the extent they reported their programs had been subject to these reviews (no-small extent, moderate extent, great-very great extent). We then compared how these groups responded to the ten questions.

For the third objective, we reviewed our past work that assessed Executive Branch activities to enhance the use of performance information; various resources (i.e., guidance, guides, and playbooks) developed by the Office of Management and Budget (OMB) and the Performance Improvement Council (PIC) that could support agencies’ use of performance information; and the President’s Management Agenda, and related materials with information on cross-agency efforts to improve the use of data in federal decision making.

Lastly, for the third objective we also interviewed OMB and PIC staff about any actions they have taken, or planned to take, to further support the use of performance information across the federal government.

We conducted this performance audit from October 2017 to September 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Comparison of 2007, 2013, and 2017 Agency Use of Performance Information Index Scores

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the above contact, Benjamin T. Licht (Assistant Director) and Adam Miles (Analyst-in-Charge) supervised this review and the development of the resulting report. Arpita Chattopadhyay, Caitlin Cusati, Meredith Moles, Dae Park, Amanda Prichard, Steven Putansu, Alan Rozzi, Shane Spencer, and Khristi Wilkins also made key contributions. Robert Robinson developed the graphics for this report. Alexandra Edwards, Jeff DeMarco, Mark Kehoe, Ulyana Panchishin, and Daniel Webb verified the information presented in this report.

Related GAO Products

Results of the Periodic Surveys on Organizational Performance and Management Issues Managing for Results: Further Progress Made in Implementing the GPRA Modernization Act, but Additional Actions Needed to Address Pressing Governance Challenges. GAO-17-775. Washington, D.C.: September 29, 2017.

Supplemental Material for GAO-17-775: 2017 Survey of Federal Managers on Organizational Performance and Management Issues. GAO-17-776SP. Washington, D.C.: September 29, 2017.

Program Evaluation: Annual Agency-wide Plans Could Enhance Leadership Support for Program Evaluations. GAO-17-743. Washington, D.C.: September 29, 2017.

Managing for Results: Agencies’ Trends in the Use of Performance Information to Make Decisions. GAO-14-747. Washington, D.C.: September 26, 2014.

Managing for Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges. GAO-13-518. Washington, D.C.: June 26, 2013.

Managing for Results: 2013 Federal Managers Survey on Organizational Performance and Management Issues, an E-supplement to GAO-13-518. GAO-13-519SP. Washington, D.C.: June 26, 2013.

Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making. GAO-13-570. Washington, D.C.: June 26, 2013.

Government Performance: Lessons Learned for the Next Administration on Using Performance Information to Improve Results. GAO-08-1026T. Washington, D.C.: July 24, 2008.

Government Performance: 2007 Federal Managers Survey on Performance and Management Issues, an E-supplement to GAO-08-1026T. GAO-08-1036SP. Washington, D.C.: July 24, 2008.

Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results. GAO-04-38. Washington, D.C.: March 10, 2004.

Managing for Results: Federal Managers’ Views on Key Management Issues Vary Widely Across Agencies. GAO-01-592. Washington, D.C.: May 25, 2001.

Managing for Results: Federal Managers’ Views Show Need for Ensuring Top Leadership Skills.GAO-01-127. Washington, D.C.: October 20, 2000.

The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven. GAO/GGD-97-109. Washington, D.C.: June 2, 1997.