Managing for Results: Agencies' Trends in the Use of Performance Information to Make Decisions
Released: 2014-09-26T00:00:00

Letter

Background

GAO-08-1036SP and GAO-13-519SP. use of performance information has not changed significantly over time government-wide.

These survey results are consistent with trends identified in other federal employee surveys government-wide. For example, the Office of Personnel Management (OPM) surveys federal workers with the Federal Employee Viewpoint Survey (FEVS). FEVS is a tool that measures employees’ perceptions of whether, and to what extent, conditions characterizing successful organizations are present in their agencies. OPM creates an index using a smaller subset selected from the FEVS survey responses that are related to agencies’ results-oriented performance culture. OPM also creates additional indices using different subsets of FEVS survey questions related to: (1) leadership and knowledge management; (2) talent management; and (3) job satisfaction. On the results-oriented performance culture index, 27 of the 37 agencies OPM surveyed experienced a decline between 2008 and 2013. Only seven agencies improved during this time period—OPM, the U.S. Departments of Education and Transportation, the Federal Communications Commission, National Labor Relations Board, Railroad Retirement Board, and the Broadcasting Board of Governors.

The Office of Management and Budget and the Performance Improvement Council (PIC), work with federal agencies to improve performance across the federal government. Among the PIC’s responsibilities is the charge to facilitate the exchange of useful performance improvement practices and work among the federal agencies to resolve government-wide or crosscutting performance issues.

Agencies’ Reported Use of Performance Information Generally Has Not Improved Since 2007

Few federal agencies showed improvement in managers’ use of performance information for decision making between 2007 and 2013, as measured by our use index. Specifically, our analysis of the average use index score at each agency found that most agencies showed no statistically significant change in use during this period. Only two agencies—OPM and the Department of Labor—experienced a statistically significant improvement in managers’ use of performance information. During the same time period, four agencies—the Departments of Energy and Veterans Affairs (VA), the National Aeronautics and Space Administration, and the Nuclear Regulatory Commission—experienced a statistically significant decline in managers’ use of performance information as measured by our index. See table 1 below for agency scores on the use of performance information index.

In addition, figure 4 illustrates that SES managers used performance information, as measured by our index, more than non-SES managers both government-wide and within each agency. SES managers government-wide and at nine agencies scored statistically significantly higher than the non-SES managers at those agencies. As shown in figure 4 below, SES and non-SES managers from DHS and VA had the largest gaps in use of performance information between their SES and non-SES managers. In one agency—the National Science Foundation—the trend was reversed, with non-SES managers reporting more favorably than SES managers. However, this difference was not statistically significant.

Survey Questions Addressing Key Practices Significantly Related to the Use of Performance Information

Using the data from our 2013 survey of federal managers, we found that specific practices identified in our previous work on the use of performance information to enhance or facilitate the use of performance information for decision making were significantly related to the use of performance information as measured by our use index. Figure 5 shows the questions that we tested based on each of the practices. We have highlighted those questions and responses that we found to have a statistically significant and positive relationship with the use of performance information index.performance information index for agencies increased when managers reported that their agencies engaged to a greater extent in these practices as reflected in the survey questions. For example, in 2013, OPM managers responded more favorably than the government-wide average on several of the survey questions related to these practices. OPM was one of the two agencies that experienced an increase in use of performance information from 2007 to 2013, as measured by our index.

Aligning Agencywide Goals, Objectives, and Measures

Leading practices state that aligning an agency’s goals, objectives, and measures increases the usefulness of the performance information collected to decision makers at each level, and reinforces the connection between strategic goals and the day-to-day activities of managers and staff. In analyzing the 2013 survey results, we found that managers’ responses to a related survey question were significantly related to the use of performance information controlling for other factors. Specifically, increase in the extent to which individuals agreed that managers aligned performance measures with agency-wide goals and objectives were associated with increase on the five-point scale we used for our use index.

Government-wide, an estimated 46 percent of managers at federal agencies reported that managers at their levels took steps to align program performance measures with agency-wide goals and objectives. The Social Security Administration (SSA) and OPM led the 24 agencies with approximately 65 percent of managers reporting that they aligned program performance measures with agency-wide goals and objectives. DHS trailed the other agencies with only 34 percent of their managers reporting similarly.

Improving the Usefulness of Performance Information

Leading practices state that to facilitate the use of performance information, agencies should ensure that information meets various users’ needs for completeness, accuracy, consistency, timeliness, validity, and ease of use. When analyzing the results of our 2013 survey, we found that managers’ responses to the statement, “I have sufficient information on the validity of the performance data I use to make decisions,” related to their use of performance information. Specifically, individuals who rated their agencies as providing a higher extent of sufficient information on the validity of performance data for decision making, tended to rate their agencies higher on the performance use scale than individuals who rated their agencies lower, controlling for other factors. Having sufficient information on the validity of performance data for decision making had the largest potential effect of the questions included in our model. This question was the strongest predictor in our regression analysis.

Government-wide, the percentage of managers responding favorably about having sufficient information on the validity of performance data was particularly low, at about 36 percent. The National Aeronautics and Space Administration (NASA) and OPM led the agencies with more than 50 percent of managers from NASA and OPM responding that they have sufficient information about the validity of performance data for decision- making (58 percent and 54 percent, respectively). The U.S. Department of Agriculture (USDA) and DHS trailed the other agencies with less than 30 percent of their managers responding similarly (28 percent and 21 percent, respectively).

Developing Agency Capacity to Use Performance Information

Leading practices state that building the capacity for managers to use performance information is critical to using performance information in a meaningful fashion, and that inadequate staff expertise, among other When factors, can hinder agencies from using performance information.we analyzed the results of our 2013 survey, we found that managers who said that their agencies have provided training that would help them to use performance information to make decisions, rated their agencies more positively on our use index. Compared to managers who said their agencies had not trained them on using performance information in decision making, those who said their agencies did rated them higher on the use scale, controlling for other factors.

Government-wide, an estimated 44 percent of the managers who responded to our survey reported that their agencies have provided training that would help them to use performance information in decision making. The U.S. Agency for International Development (USAID) led the agencies in this area, with 62 percent of USAID managers responding that their agencies have provided training that would help them use of performance information in decision making in the last 3 years. The U.S. Department of the Treasury (Treasury), DHS, the Nuclear Regulatory Commission (NRC), and the Environmental Protection Agency (EPA) trailed the other agencies with less than 35 percent of their managers responding similarly (Treasury and DHS with 34 percent, NRC with 33 percent, and EPA with 32 percent) that they had received training on use of performance information in the last 3 years.

Other types of training did not appear to be positively related to use of performance information. Specifically, training on developing performance measures was significantly—but negatively—related to use of performance information. Training on (1) setting program performance goals; (2) assessing the quality of performance data; and (3) linking program performance to agency strategic plans was not found to relate to managers’ use of performance information after controlling for other information.

Demonstrating Management Commitment

Leading practices state that the demonstrated commitment of leadership and management to achieving results and using performance information can encourage others to embrace using a model that uses performance information to make decisions. When we analyzed the results of our 2013 survey, we found that managers’ responses to the statement, “My agency’s top leadership demonstrates a strong commitment to achieving results,” were significantly and positively related to the use of performance information. Specifically, on average, increases in a manager’s rating of the strength of their agency’s top leadership’s commitment to achieving results were associated with increased ratings of their agencies on the use scale, controlling for other factors.

Government-wide, the percentage of federal managers responding favorably about their agencies’ top leadership demonstrating a strong commitment to achieving results was an estimated 60 percent. Managers from NRC (78 percent) and SSA (74 percent) had significantly higher scores on this question than the government-wide average, while managers from DHS (44 percent) and USDA (42 percent) had lower scores than the government-wide average.

Communicating Performance Information Frequently and Effectively

Leading practices state that communicating performance information frequently and effectively throughout an agency can help managers to inform staff and other stakeholders of their commitment to achieve agency goals and to keep these goals in mind as they pursue their day- to-day activities. When analyzing the results of our 2013 survey, we found that two related questions were significantly and positively related to an agency’s use of performance information:

Agency managers/supervisors at my level effectively communicate performance information routinely.

Employees in my agency receive positive recognition for helping the agency accomplish its strategic goals.

Specifically, those who reported favorably that agency managers/supervisors at their levels effectively communicated performance information routinely tended to rate their agencies somewhat higher on the use index, controlling for other factors. Similarly, those who reported favorably that employees in their agency receive positive recognition for helping the agency accomplish its strategic goals rated their agencies somewhat higher on the use scale, controlling for other factors.

An estimate 41 percentage of managers government-wide who responded to our survey reported that agency managers/supervisors at their level effectively communicated performance information routinely. About 60 percent of managers at the Small Business Administration, Department of Labor, and OPM, responded positively when asked about effectively communicating performance information routinely (62 percent, 61 percent, and 60 percent respectively). DHS trailed the other agencies with only 34 percent of its managers reporting similarly.

Government-wide, an estimated 42 percent of the managers responded favorably when asked about employees in their respective agencies receiving positive recognition for helping the agencies accomplish their strategic goals. While the managers at NRC and the U.S. Department of Commerce scored at or higher than 50 percent when asked about positive recognition (58 percent and 50 percent, respectively), DHS trailed federal agencies with only 34 percent of its managers reporting similarly.

Concluding Observations

Our analyses of agency-level results from our periodic surveys of federal managers in 2007 and 2013 reinforce that there are several leading practices and related survey questions that significantly influenced agencies’ use of performance information for management decision making. However, our surveys show that such usage generally has not improved over time. This information can be helpful to the Office of Management and Budget (OMB) and the Performance Improvement Council as they work with federal agencies to identify and implement stronger performance management practices to help improve agency use of performance information. Moreover, the use of performance information will remain a challenge unless agencies can narrow the gap in use between Senior Executive Service (SES) and non-SES managers.

Agency Comments

We provided a draft of this report to the Director of OMB and to the 24 agencies that responded to our 2007 and 2013 federal managers surveys. On September 4, 2014, OMB staff provided us with oral comments and generally agreed with our report. OMB staff also stated that they would continue to work with agencies to address the use of performance information through agencies’ annual strategic reviews of progress toward agencies’ strategic objectives, which began in 2014.

We also received comments from the U.S. Departments of Commerce (Commerce) and the Treasury (Treasury), the General Services Administration (GSA), and the National Aeronautics and Space Administration (NASA). On August 27, 2014, the liaison from NASA e- mailed us a summary of NASA officials’ comments. On August 28, 2014, the liaison from GSA e-mailed us a summary of GSA officials’ comments.

On August 29, 2014, the liaisons from Commerce and Treasury e-mailed us summaries of their respective agency officials’ comments. Commerce and GSA generally agreed with our report, and provided technical comments, which we incorporated as appropriate. NASA and Treasury raised concerns about the findings and conclusions in our report, including the design of the surveys. We discuss their comments and our evaluation of them below, which generally fell into the following four categories:

NASA and Treasury raised concerns about the underlying methodology for the 2007 and 2013 federal managers surveys. They said that it did not adequately provide agency-wide perspectives that fully represented the agencies’ use of performance information. Specifically, NASA and Treasury expressed concerns about the lack of demographic information about the survey respondents (e.g. survey respondents by agency component and geographic location). Treasury also expressed concern as to whether we had included senior leadership in our report. To address this comment, we added some additional information to our report that discusses our survey design and administration, specifically that we did not collect demographic information beyond whether a federal manager’ was a member of the SES or not (non-SES). Moreover, our stratified random sample of federal managers ensured that we had a representative sample of federal managers both government-wide and within each of the 24 agencies we surveyed. It was not our objective to design the survey and draw a sample of managers that would allow us to report in a generalizable way at the geographic location or organizational level within an agency. Designing a sample to produce estimates at the geographic location and/or organizational level within an agency would result in a much larger sample than the approximately 107,326 managers selected in our 2007 survey and the approximately 148,300 managers selected in our 2013 survey. Nevertheless, as previously discussed, our sample was sufficient for the purposes of this report.

NASA and Treasury also expressed concern that despite all the efforts their respective agencies have undertaken to implement the GPRA Modernization Act of 2010, our draft report did not provide information on the root causes for the lack of progress in the use of performance information in their agencies. For example, NASA cited some of its agency initiatives, including the development of an automated performance management data repository to assist in the agency’s decision-making process.

Treasury cited its Quarterly Performance Review process as an example of the agency’s commitment to using performance information in decision making. We recognize the activities that the agencies have underway to improve staff engagement on the use of performance information for decision making, and have previously reported on some of these initiatives. However, despite the efforts discussed above, our survey results showed that the use of performance information, as reported by managers at the agencies, has not improved within agencies between 2007 and 2013. Our report analyzed the results from specific questions in both the 2007 and 2013 surveys. We agree that our report does not provide information on the root causes for the trends we found in the use of performance information. However, the results of the regression analysis in this report point to some specific practices that can enhance the use of performance information, areas where federal agencies may want to focus further analysis and efforts. Both NASA and Treasury requested their respective agencies’ 2007 and 2013 survey data sets to perform additional analyses that might provide further insights into root causes underlying the trends in the use of performance information within their agencies.

Treasury also commented that the rankings we report based on the average scores on the 2013 use of performance information index might imply that agencies with a higher ranking are theoretically better at using performance information, and therefore, have superior performance management practices. Treasury also raised concerns about our use of the index to score agencies. It asked if it should view the higher-ranking agencies as examples of what agencies should do to improve the use of performance information. There is not a huge difference in scores between those agencies that scored higher on the use index than others at the lower end. But, we believe our methodology is useful for generally distinguishing between agencies’ levels of use of performance information, and for assessing change in use of performance information over time. However, we revised our report to focus on agencies’ scores rather than on rank ordering. We also did additional statistical testing to determine whether or not the changes between the 2007 and 2013 use indexes were statistically different among agencies. As for the implication of the rankings on the quality of management practices in particular agencies, in 2007, we did employ a use index to identify agencies for further case study analysis. We selected an agency that had significantly improved on the use index along with agencies that scored lower on the index to assess whether there were any promising practices or challenges facing those agencies.

NASA, Treasury, and Commerce all commented that it was difficult to tell how managers may have interpreted the term “performance information” when responding to our surveys. Treasury further commented that it was unclear what information managers were using to make management decisions if they were not using performance information. In both the 2007 and 2013 surveys, we defined the terms “performance information” and “performance measures” in the broadest sense. To clarify this point, we added the definition of performance information from the 2013 managers survey in the report. Moreover, as discussed above, additional agency analysis of the root causes underlying the use of performance information could provide some additional context to the types of information agencies are using for decision making.

The following 20 agencies had no comments on the draft report: the U.S. Departments of Agriculture, Defense, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, Justice, Labor, State, Transportation, and Veterans Affairs, the Environmental Protection Agency, Nuclear Regulatory Commission, Office of Personnel Management, National Science Foundation, Small Business Administration, Social Security Administration, and the United States Agency for International Development. The written response from the Social Security Administration is reproduced in appendix II.

We are sending copies of this report to the agencies that participated in our 2013 managers survey, the Director of OMB, as well as appropriate congressional committees and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II.

Appendix I: Regression Analyses of 2013 Federal Managers Survey Results to Identify Predictors of Use of Performance Information

In analyzing the results of our 2013 survey, we explored whether federal managers’ responses to certain survey questions could help explain differences in how managers in agencies reported using performance information. To examine which factors related to agency use of performance information, as measured by the use of performance information index, we conducted regression analysis. The regression analysis allowed us to assess the unique association between our outcome variable—the performance information index—and a given predictor variable, while controlling for multiple other predictor variables.

To create the use of performance index, we identified survey questions that reflected managers’ use of performance information for key management activities and decision making. The 2013 use of performance index included most of the questions included in our 2007 index, and additional questions from the 2013 managers survey that we determined reflected the concept of use of performance information (see figure 1 for specific questions included in our index). core set of items from the original index, we tested the impact of including and excluding several additional questions related to performance management use to ensure the cohesiveness and strength of our revised index. Our revised index is an average of the questions used for the index and runs from 1 to 5, where a 1 reflects that managers feel the agency engages “to no extent” and a 5 reflecting that managers feel the agency engages “to a very great extent” in the use of performance information activities. We found the index met generally accepted standards for scale reliability.

For more information on the original index we created for the 2007 ffederal managers survey, see GAO-08-1026T.

To develop our regression model examining predictors of performance use as measured by our index, we first identified a series of variables that were related to one of the five practices we have previously found to enhance or facilitate use of performance information. These practices include: aligning agencywide goals, objectives, and measures; improving the usefulness of performance information; developing the capacity to use performance information; demonstrating management commitment; and communicating performance information frequently and effectively. See figure 3 for the specific questions related to these five practices that we included in the regression. Although we identified other questions also related to the five elements of effective performance management, many of these questions were already accounted for in our use index of performance information, and we excluded them from consideration in our regression.

Overall, our results demonstrate that some types of management practices and training are more positively correlated than others, with manager perceptions of performance information use as measured by the use index, even when controlling for other factors. Further, these results suggest that certain specific efforts to increase agency use of performance information—such as increasing timeliness of performance information and providing information on the validity of performance measures—may have a higher return than others.

To execute our analysis, we began with a base model that treated differences in managers’ views of agency performance management use as a function of the agency where they worked. We found that despite statistically significant differences on average among managers at different agencies, a regression model based on agency alone had very poor predictive power (R-squared of .03).

We next examined whether managers’ responses to other items reflecting the practices of effective performance management related to their perceptions of agency use of performance information, independent of agency. We found that several items consistently predicted increases on individuals’ ratings of their agencies use of performance management information, including whether managers align program performance measures with agency goals and objectives; having information on the validity of performance measures; and training on how to use performance management information in decision making. We also tested this model controlling for whether a respondent was a member of the Senior Executive Service (SES), and found similar results.

We also tested our model with a variable to control for agency size in five categories. We found that, relative to the largest agencies (100,000 or more employees), managers at smaller agencies tended to rate their agency’s use of performance information slightly lower. The significance and magnitude of other significant variables was similar whether we controlled for agency size or using intercepts to control for individual agencies.

Our final model had an R-squared of .65, suggesting that the independent variables in the model predicted approximately 65 percent of the variance in the use index. Specific results are presented in table 2 below. Each coefficient reflects the average increase in the dependent variable, our five-point use scale, associated with a one-unit increase in the value of the independent variables. Note that in our discussion, we highlight the maximum potential impact of each variable rather than the increase in the use score associated with each increase in a dependent variable.

As seen in table 2, at least one question related to each of the five practices to enhance agencies’ use of performance information was significant. With respect to aligning agencywide goals, objectives, and measures, we found that each increase in terms of the extent to which individuals felt that managers aligned performance measures with agencywide goals and objectives was associated with a .13 increase in their score on the use scale, or approximately a .52 increase on the 5- point use scale when comparing individuals in the lowest to the highest categories.

In terms of improving the usefulness of performance information, we found that having information on the validity of performance data for decision making was the strongest predictor in our model. Compared to individuals who said that they did not have sufficient information on the validity of performance data for decision making, on average, individuals who said they had a very great extent of information rated their agencies approximately 0.64 points higher on the performance use scale, controlling for other factors. In contrast, the potential effect of the timeliness of information, while significant, had a smaller potential impact on managers’ perceptions of their agency’s use of performance information. On average, managers who responded “to a very great extent” on whether their agency’s performance information was available in time to manage programs or projects rated their agency about .28 points higher on the performance use scale than those who responded “to no extent.”

In terms of developing agency capacity to use performance information, we found that one type of training was positively related to use of performance information, though other types of training were either not related or were negatively related, after controlling for other factors. Compared to managers who said their agencies had not trained them with training on how to use performance information in decision making, those who said their agencies did provide such training rated their agencies an average of .14 points higher on the use scale, controlling for other factors. The potential effect of this type of training was relatively small compared to the potential effect of some of the other predictors in our model. In contrast, training in developing performance measures was negatively associated with managers’ perceptions of performance information use.

With respect to demonstrating management commitment, managers that rated their agency’s leadership highly in terms of demonstrating a strong commitment to achieving results tended to rate their agencies higher on performance information use, as measured by our use index. Each increase in the extent to which a manager felt their agency leadership was committed to results was associated with a .08 increase in the performance use index, or up to a .32 increase in the five-point performance use index when comparing managers who reported “no extent” of leadership commitment to those that reported “a very great extent.”

Two questions related to communicating performance information frequently and effectively were significantly and positively associated with manager’s perceptions of an agency’s use of performance information, controlling for other factors. Compared to those who rated their agencies the lowest in terms of whether managers and supervisors effectively communicated performance information routinely—those managers who rated their agencies most highly averaged .32 points higher on the five- point performance use index. Similarly, managers who reported that employees in their agency received “a very great extent” of positive recognition for helping the agency to accomplish strategic goals rated their agencies an average of .24 points higher on performance information use, as measured by our use index. We did not find a statistically significant relationship between the accessibility of performance information (to managers, employees or the public) and managers’ perceptions of use of performance information.

Sensitivity and Specification Testing

To conduct our analysis, we used Stata software to generate regression estimates that incorporated variance calculations appropriate for the To ensure that large amounts of complex design of the survey data.missing data do not result from listwise deletion, we imputed values for individual questions if the individual is missing or indicated “no basis to judge” on three or fewer responses from the 23 variables initially tested in the regression, using the agency-level average to impute. Individuals missing data on more than 3 of the 23 potential variables were dropped from the analysis. We conducted a variety of sensitivity checks to ensure that our results were robust across different specifications and assumptions. For the most part, we found generally similar patterns across models in terms of the magnitude and significance of different variables related to the elements of effective performance management.

In general, our models assume that the relationship between the independent and dependent variables is linear, and that changes in the dependent variable associated with a change in the independent variable are similar across each ordinal category. Under this specification, the change in the use index associated with a shift from “to no extent” to “to a small extent” is assumed to be similar to the change associated with an increase from “to a great extent” to “a very great extent”. To determine whether the linear specification was appropriate, or consistent with the observed data, we tested versions of our models that treated independent variables with a Likert-scale response as categorical. We found our results to be robust across a variety of specifications, including those that relaxed the assumption of linearity for responses based on a five-point scale.

Appendix II: Comments from the Social Security Administration

Appendix III: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgements

In addition to the contact named above, Sarah Veale (Assistant Director), Margaret McKenna Adams, Tom Beall, Mallory Barg Bulman, Chad Clady, Karin Fangman, Cynthia Jackson, Janice Latimer, Donna Miller, Anna Maria Ortiz, Kathleen Padulchick, Mark Ramage, Joseph Santiago, Albert Sim, and Megan Taylor made key contributions to this report.