Managing for Results: Agencies' Trends in the Use of Performance Information to Make Decisions
Published: 2014-09-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO has long reported that agencies are better equipped to address management and performance challenges when managers effectively use performance information for decision making. However, GAO's periodic surveys of federal managers indicate that use of performance information has not changed significantly.
GAO was mandated to evaluate the implementation of the GPRA Modernization Act of 2010. GAO assessed agencies' use of performance information from responses to GAO's surveys of federal managers at 24 agencies. To address this objective, GAO created an index to measure agency use of performance information derived from a set of questions from the most recent surveys in 2007 and 2013, and used statistical analysis to identify practices most significantly related to the use of performance information index.
section_title: What GAO Found
paragraphs: Agencies' reported use of performance information, as measured by GAO's use of performance information index, generally did not improve between 2007 and 2013. The index was derived from a set of survey questions in the 2007 and 2013 surveys that reflected the extent to which managers reported that their agencies used performance information for various management activities and decision making. GAO's analysis of the average index score among managers at each agency found that most agencies showed no statistically significant change in use during this period. As shown in the table below, only two agencies experienced a statistically significant improvement in the use of performance information. During the same time period, four agencies experienced a statistically significant decline in the use of performance information.
Legend statistically significant decrease statistically significant increase
GAO has previously found that there are five leading practices that can enhance or facilitate the use of performance information: (1) aligning agency-wide goals, objectives, and measures; (2) improving the usefulness of performance information; (3) developing agency capacity to use performance information; (4) demonstrating management commitment; and (5) communicating performance information frequently and effectively. GAO tested whether additional survey questions related to the five practices were significantly related to the use of performance information as measured by the index. GAO found that the average use of performance information index for agencies increased when managers reported their agencies engaged to a great extent in these practices as reflected in the survey questions. For example, the Office of Personnel Management (OPM) was one of the two agencies that experienced an increase in use of performance information from 2007 to 2013, as measured by the GAO index. In 2013, OPM managers responded more favorably than the government-wide average on several of the survey questions related to these practices.
section_title: What GAO Recommends
paragraphs: GAO is not making recommendations in this report. Office of Management and Budget staff generally agreed with the report. Four agencies (the Departments of Commerce and the Treasury, the General Services Administration (GSA), and the National Aeronautics and Space Administration (NASA)) provided comments that are addressed. Commerce and GSA agreed with the report. Treasury and NASA raised concerns about the findings and conclusions in this report, including the design of the surveys. GAO continues to believe its findings and conclusions are valid as discussed in the report. Twenty other agencies did not have comments.

section_title: Letter
paragraphs: The federal government is one of the world’s largest and most complex  entities, with about $3.5 trillion in outlays in fiscal year 2013, funding a  vast array of programs and operations. It faces a number of significant  fiscal, management, and performance challenges in responding to the  diverse and increasingly complex issues it seeks to address. Addressing  these challenges will require actions on multiple fronts. Our prior work on  results-oriented management has found that data-driven decision making   If agencies do not effectively use performance  leads to better results.measures and performance information to track progress and achieve  their goals, they increase the risk of failing to achieve them.
In that regard, we have previously reported that the performance planning  and reporting framework originally put into place by the Government  Performance and Results Act of 1993 (GPRA), and significantly  enhanced by the GPRA Modernization Act of 2010 (GPRAMA), provides  important tools that can help inform congressional and executive branch  decision making to address challenges the federal government faces.
This report is part of a series of reports under our mandate to examine  the implementation of GPRAMA. This report compares the agency-level  results from our 2013 survey of federal managers at the 24 agencies  covered by the Chief Financial Officers (CFO) Act of 1990, as amended,  with our 2007 managers survey. The 2007 survey is the most recent  survey conducted before GPRAMA was enacted in 2011 and includes  results from the Department of Homeland Security (DHS).objective for this report was to assess agencies’ use of performance  information from responses to our federal managers surveys.
To address this objective, we analyzed survey data that we had  previously collected and publicly reported in 2007 and 2013. In 2007, we  surveyed a stratified random sample of mid-level and upper-level  managers and supervisors (General Schedule levels comparable to 13  through 15 and career Senior Executive Service (SES) or equivalent) at  the 24 CFO Act agencies (4,412 persons from a population of  approximately 107,326 mid-level and upper-level civilian managers and  supervisors). For the 2007 survey results, the average response rate  across all agencies was about 70 percent. In 2013, we again surveyed a  stratified random sample of mid-level and upper-level managers and  supervisors at the 24 CFO Act agencies (4,391 persons from a population  of approximately 148,300 mid-level and upper-level civilian managers and  supervisors). For the 2013 survey, the average response rate across all  agencies was 69 percent. Similar to the previous surveys, the sample  was stratified by agency and by whether the manager or supervisor was a  member of the SES or non-SES. The overall survey results are  generalizable to the population of managers as described above at each  of the 24 agencies and government-wide. The responses of each eligible  sample member who provided a usable questionnaire were weighted in  the analyses to account statistically for all members of the population.  For additional information on survey development and implementation,  please see our prior reports. We did not interview officials at the  agencies that participated in our 2013 managers survey to obtain  additional information.
The use of performance information index was based primarily on an  index that we developed and reported on the 2007 managers survey. In  both the 2007 and 2013 surveys, we defined the terms “performance  information” and “performance measures” in the broadest sense. In our  2013 survey, we defined performance information as the data collected to  measure progress toward achieving an agency’s established mission or  program-related goals or objectives. We further stated that performance  information can focus on performance measures, such as quality,  timeliness, customer satisfaction, or efficiency. It can inform key  management decisions such as setting program priorities, allocating  resources, or identifying program problems and taking corrective actions  to solve those problems. After identifying a core set of items from the  original index, we tested the impact of including and excluding several  additional questions related to performance management use to ensure  the cohesiveness and strength of our revised index that we used for the  2013 survey.
Figure 1 shows the set of 2013 managers survey questions that comprise  the use index.
After developing the use indices for 2007 and 2013, we then analyzed the  managers’ responses, grouped the managers’ responses according to  their agencies, and compared the agencies’ scores on each index.
We then reviewed our prior work where we identified leading practices  that can help agencies enhance or facilitate the use of performance  information for management decision making, as shown in figure 2  below.
To determine if there any additional factors related to these leading  practices that could influence how an agency scored on our use index, we  looked at the remaining questions from the 2013 managers survey and  identified those additional questions that were associated with these  leading practices. We used statistical testing to determine if the  relationship between these additional questions and an agency’s use of  performance information was statistically significant. See figure 3 below  for the survey questions we tested related to the five leading practices.
We conducted regression analyses to assess the relationship between  these additional questions related to leading practices to enhance and  facilitate the use of performance information and an agency’s score and  ranking on the use index. Regression analyses allowed us to assess the  unique association between our outcome variable and a given predictor   Using  variable, while controlling for multiple other predictor variables.our survey data, these analyses were intended to reflect the strength of  the relationship between our previously identified practices with our 2013  use index. Our analyses did not seek to identify or assess other survey  items addressing additional practices that we had not previously identified  for improving the use of performance information for management  decision making. More information on our regression analysis can be  found in appendix I.
To ensure reliability of the specific survey questions we used in our  analyses, we conducted electronic testing of the data and reviewed prior  information on the design and implementation of our 2007 and 2013  managers surveys. purpose of this report.
We believe the data are sufficiently reliable for the  We conducted this performance audit from August 2013 to September  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GAO-08-1036SP and GAO-13-519SP. use of performance information has not changed significantly over time  government-wide.
These survey results are consistent with trends identified in other federal  employee surveys government-wide. For example, the Office of  Personnel Management (OPM) surveys federal workers with the Federal  Employee Viewpoint Survey (FEVS). FEVS is a tool that measures  employees’ perceptions of whether, and to what extent, conditions  characterizing successful organizations are present in their agencies.  OPM creates an index using a smaller subset selected from the FEVS  survey responses that are related to agencies’ results-oriented  performance culture. OPM also creates additional indices using different  subsets of FEVS survey questions related to: (1) leadership and  knowledge management; (2) talent management; and (3) job satisfaction.  On the results-oriented performance culture index, 27 of the 37 agencies  OPM surveyed experienced a decline between 2008 and 2013. Only  seven agencies improved during this time period—OPM, the U.S.  Departments of Education and Transportation, the Federal  Communications Commission, National Labor Relations Board, Railroad  Retirement Board, and the Broadcasting Board of Governors.
The Office of Management and Budget and the Performance  Improvement Council (PIC), work with federal agencies to improve  performance across the federal government. Among the PIC’s  responsibilities is the charge to facilitate the exchange of useful  performance improvement practices and work among the federal  agencies to resolve government-wide or crosscutting performance issues.
section_title: Agencies’ Reported Use of Performance Information Generally Has Not Improved Since 2007
paragraphs: Few federal agencies showed improvement in managers’ use of  performance information for decision making between 2007 and 2013, as  measured by our use index. Specifically, our analysis of the average use  index score at each agency found that most agencies showed no  statistically significant change in use during this period. Only two  agencies—OPM and the Department of Labor—experienced a  statistically significant improvement in managers’ use of performance  information. During the same time period, four agencies—the  Departments of Energy and Veterans Affairs (VA), the National  Aeronautics and Space Administration, and the Nuclear Regulatory  Commission—experienced a statistically significant decline in managers’  use of performance information as measured by our index. See table 1  below for agency scores on the use of performance information index.
In addition, figure 4 illustrates that SES managers used performance  information, as measured by our index, more than non-SES managers  both government-wide and within each agency. SES managers  government-wide and at nine agencies scored statistically significantly  higher than the non-SES managers at those agencies. As shown in figure  4 below, SES and non-SES managers from DHS and VA had the largest  gaps in use of performance information between their SES and non-SES  managers. In one agency—the National Science Foundation—the trend  was reversed, with non-SES managers reporting more favorably than  SES managers. However, this difference was not statistically significant.
subsections: section_title: Survey Questions Addressing Key Practices Significantly Related to the Use of Performance Information
paragraphs: Using the data from our 2013 survey of federal managers, we found that  specific practices identified in our previous work on the use of  performance information to enhance or facilitate the use of performance  information for decision making were significantly related to the use of  performance information as measured by our use index. Figure 5 shows  the questions that we tested based on each of the practices. We have  highlighted those questions and responses that we found to have a  statistically significant and positive relationship with the use of  performance information index.performance information index for agencies increased when managers  reported that their agencies engaged to a greater extent in these  practices as reflected in the survey questions. For example, in 2013,  OPM managers responded more favorably than the government-wide  average on several of the survey questions related to these practices.  OPM was one of the two agencies that experienced an increase in use of  performance information from 2007 to 2013, as measured by our index.
section_title: Aligning Agencywide Goals, Objectives, and Measures
paragraphs: Leading practices state that aligning an agency’s goals, objectives, and  measures increases the usefulness of the performance information  collected to decision makers at each level, and reinforces the connection  between strategic goals and the day-to-day activities of managers and  staff. In analyzing the 2013 survey results, we found that managers’  responses to a related survey question were significantly related to the  use of performance information controlling for other factors. Specifically,  increase in the extent to which individuals agreed that managers aligned  performance measures with agency-wide goals and objectives were  associated with increase on the five-point scale we used for our use  index.
Government-wide, an estimated 46 percent of managers at federal  agencies reported that managers at their levels took steps to align  program performance measures with agency-wide goals and objectives.  The Social Security Administration (SSA) and OPM led the 24 agencies  with approximately 65 percent of managers reporting that they aligned  program performance measures with agency-wide goals and objectives.  DHS trailed the other agencies with only 34 percent of their managers  reporting similarly.
section_title: Improving the Usefulness of Performance Information
paragraphs: Leading practices state that to facilitate the use of performance  information, agencies should ensure that information meets various users’  needs for completeness, accuracy, consistency, timeliness, validity, and  ease of use. When analyzing the results of our 2013 survey, we found  that managers’ responses to the statement, “I have sufficient information  on the validity of the performance data I use to make decisions,” related  to their use of performance information. Specifically, individuals who rated  their agencies as providing a higher extent of sufficient information on the  validity of performance data for decision making, tended to rate their  agencies higher on the performance use scale than individuals who rated  their agencies lower, controlling for other factors. Having sufficient  information on the validity of performance data for decision making had  the largest potential effect of the questions included in our model. This  question was the strongest predictor in our regression analysis.
Government-wide, the percentage of managers responding favorably  about having sufficient information on the validity of performance data  was particularly low, at about 36 percent. The National Aeronautics and  Space Administration (NASA) and OPM led the agencies with more than  50 percent of managers from NASA and OPM responding that they have  sufficient information about the validity of performance data for decision- making (58 percent and 54 percent, respectively). The U.S. Department  of Agriculture (USDA) and DHS trailed the other agencies with less than  30 percent of their managers responding similarly (28 percent and 21  percent, respectively).
section_title: Developing Agency Capacity to Use Performance Information
paragraphs: Leading practices state that building the capacity for managers to use  performance information is critical to using performance information in a  meaningful fashion, and that inadequate staff expertise, among other   When  factors, can hinder agencies from using performance information.we analyzed the results of our 2013 survey, we found that managers who  said that their agencies have provided training that would help them to  use performance information to make decisions, rated their agencies  more positively on our use index. Compared to managers who said their  agencies had not trained them on using performance information in  decision making, those who said their agencies did rated them higher on  the use scale, controlling for other factors.
Government-wide, an estimated 44 percent of the managers who  responded to our survey reported that their agencies have provided  training that would help them to use performance information in decision  making. The U.S. Agency for International Development (USAID) led the  agencies in this area, with 62 percent of USAID managers responding  that their agencies have provided training that would help them use of  performance information in decision making in the last 3 years. The U.S.  Department of the Treasury (Treasury), DHS, the Nuclear Regulatory  Commission (NRC), and the Environmental Protection Agency (EPA)  trailed the other agencies with less than 35 percent of their managers  responding similarly (Treasury and DHS with 34 percent, NRC with 33  percent, and EPA with 32 percent) that they had received training on use  of performance information in the last 3 years.
Other types of training did not appear to be positively related to use of  performance information. Specifically, training on developing performance  measures was significantly—but negatively—related to use of  performance information. Training on (1) setting program performance  goals; (2) assessing the quality of performance data; and (3) linking  program performance to agency strategic plans was not found to relate to  managers’ use of performance information after controlling for other  information.
section_title: Demonstrating Management Commitment
paragraphs: Leading practices state that the demonstrated commitment of leadership  and management to achieving results and using performance information  can encourage others to embrace using a model that uses performance  information to make decisions. When we analyzed the results of our  2013 survey, we found that managers’ responses to the statement, “My  agency’s top leadership demonstrates a strong commitment to achieving  results,” were significantly and positively related to the use of  performance information. Specifically, on average, increases in a  manager’s rating of the strength of their agency’s top leadership’s  commitment to achieving results were associated with increased ratings  of their agencies on the use scale, controlling for other factors.
Government-wide, the percentage of federal managers responding  favorably about their agencies’ top leadership demonstrating a strong  commitment to achieving results was an estimated 60 percent. Managers  from NRC (78 percent) and SSA (74 percent) had significantly higher  scores on this question than the government-wide average, while  managers from DHS (44 percent) and USDA (42 percent) had lower  scores than the government-wide average.
section_title: Communicating Performance Information Frequently and Effectively
paragraphs: Leading practices state that communicating performance information  frequently and effectively throughout an agency can help managers to  inform staff and other stakeholders of their commitment to achieve  agency goals and to keep these goals in mind as they pursue their day- to-day activities. When analyzing the results of our 2013 survey, we  found that two related questions were significantly and positively related  to an agency’s use of performance information:
Agency managers/supervisors at my level effectively communicate  performance information routinely.
Employees in my agency receive positive recognition for helping the  agency accomplish its strategic goals.
Specifically, those who reported favorably that agency  managers/supervisors at their levels effectively communicated  performance information routinely tended to rate their agencies somewhat  higher on the use index, controlling for other factors. Similarly, those who  reported favorably that employees in their agency receive positive  recognition for helping the agency accomplish its strategic goals rated  their agencies somewhat higher on the use scale, controlling for other  factors.
An estimate 41 percentage of managers government-wide who  responded to our survey reported that agency managers/supervisors at  their level effectively communicated performance information routinely.  About 60 percent of managers at the Small Business Administration,  Department of Labor, and OPM, responded positively when asked about  effectively communicating performance information routinely (62 percent,  61 percent, and 60 percent respectively). DHS trailed the other agencies  with only 34 percent of its managers reporting similarly.
Government-wide, an estimated 42 percent of the managers responded  favorably when asked about employees in their respective agencies  receiving positive recognition for helping the agencies accomplish their  strategic goals. While the managers at NRC and the U.S. Department of  Commerce scored at or higher than 50 percent when asked about  positive recognition (58 percent and 50 percent, respectively), DHS trailed  federal agencies with only 34 percent of its managers reporting similarly.
section_title: Concluding Observations
paragraphs: Our analyses of agency-level results from our periodic surveys of federal  managers in 2007 and 2013 reinforce that there are several leading  practices and related survey questions that significantly influenced  agencies’ use of performance information for management decision  making. However, our surveys show that such usage generally has not  improved over time. This information can be helpful to the Office of  Management and Budget (OMB) and the Performance Improvement  Council as they work with federal agencies to identify and implement  stronger performance management practices to help improve agency use  of performance information. Moreover, the use of performance  information will remain a challenge unless agencies can narrow the gap in  use between Senior Executive Service (SES) and non-SES managers.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Director of OMB and to the 24  agencies that responded to our 2007 and 2013 federal managers  surveys. On September 4, 2014, OMB staff provided us with oral  comments and generally agreed with our report. OMB staff also stated  that they would continue to work with agencies to address the use of  performance information through agencies’ annual strategic reviews of  progress toward agencies’ strategic objectives, which began in 2014.
We also received comments from the U.S. Departments of Commerce  (Commerce) and the Treasury (Treasury), the General Services  Administration (GSA), and the National Aeronautics and Space  Administration (NASA). On August 27, 2014, the liaison from NASA e- mailed us a summary of NASA officials’ comments. On August 28, 2014,  the liaison from GSA e-mailed us a summary of GSA officials’ comments.
On August 29, 2014, the liaisons from Commerce and Treasury e-mailed  us summaries of their respective agency officials’ comments. Commerce  and GSA generally agreed with our report, and provided technical  comments, which we incorporated as appropriate. NASA and Treasury  raised concerns about the findings and conclusions in our report,  including the design of the surveys. We discuss their comments and our  evaluation of them below, which generally fell into the following four  categories:
NASA and Treasury raised concerns about the underlying  methodology for the 2007 and 2013 federal managers surveys.  They said that it did not adequately provide agency-wide  perspectives that fully represented the agencies’ use of  performance information. Specifically, NASA and Treasury  expressed concerns about the lack of demographic information  about the survey respondents (e.g. survey respondents by agency  component and geographic location). Treasury also expressed  concern as to whether we had included senior leadership in our  report. To address this comment, we added some additional  information to our report that discusses our survey design and  administration, specifically that we did not collect demographic  information beyond whether a federal manager’ was a member of  the SES or not (non-SES). Moreover, our stratified random  sample of federal managers ensured that we had a representative  sample of federal managers both government-wide and within  each of the 24 agencies we surveyed. It was not our objective to  design the survey and draw a sample of managers that would  allow us to report in a generalizable way at the geographic  location or organizational level within an agency. Designing a  sample to produce estimates at the geographic location and/or  organizational level within an agency would result in a much larger  sample than the approximately 107,326 managers selected in our  2007 survey and the approximately 148,300 managers selected in  our 2013 survey. Nevertheless, as previously discussed, our  sample was sufficient for the purposes of this report.
NASA and Treasury also expressed concern that despite all the  efforts their respective agencies have undertaken to implement  the GPRA Modernization Act of 2010, our draft report did not  provide information on the root causes for the lack of progress in  the use of performance information in their agencies. For  example, NASA cited some of its agency initiatives, including the  development of an automated performance management data  repository to assist in the agency’s decision-making process.
Treasury cited its Quarterly Performance Review process as an  example of the agency’s commitment to using performance  information in decision making. We recognize the activities that  the agencies have underway to improve staff engagement on the  use of performance information for decision making, and have  previously reported on some of these initiatives. However,  despite the efforts discussed above, our survey results showed  that the use of performance information, as reported by managers  at the agencies, has not improved within agencies between 2007  and 2013. Our report analyzed the results from specific questions  in both the 2007 and 2013 surveys. We agree that our report does  not provide information on the root causes for the trends we found  in the use of performance information. However, the results of the  regression analysis in this report point to some specific practices  that can enhance the use of performance information, areas  where federal agencies may want to focus further analysis and  efforts. Both NASA and Treasury requested their respective  agencies’ 2007 and 2013 survey data sets to perform additional  analyses that might provide further insights into root causes  underlying the trends in the use of performance information within  their agencies.
Treasury also commented that the rankings we report based on  the average scores on the 2013 use of performance information  index might imply that agencies with a higher ranking are  theoretically better at using performance information, and  therefore, have superior performance management practices.  Treasury also raised concerns about our use of the index to score  agencies. It asked if it should view the higher-ranking agencies as  examples of what agencies should do to improve the use of  performance information. There is not a huge difference in scores  between those agencies that scored higher on the use index than  others at the lower end. But, we believe our methodology is useful  for generally distinguishing between agencies’ levels of use of  performance information, and for assessing change in use of  performance information over time. However, we revised our  report to focus on agencies’ scores rather than on rank ordering.  We also did additional statistical testing to determine whether or  not the changes between the 2007 and 2013 use indexes were  statistically different among agencies. As for the implication of the  rankings on the quality of management practices in particular  agencies, in 2007, we did employ a use index to identify agencies  for further case study analysis. We selected an agency that had  significantly improved on the use index along with agencies that  scored lower on the index to assess whether there were any  promising practices or challenges facing those agencies.
NASA, Treasury, and Commerce all commented that it was  difficult to tell how managers may have interpreted the term  “performance information” when responding to our surveys.  Treasury further commented that it was unclear what information  managers were using to make management decisions if they were  not using performance information. In both the 2007 and 2013  surveys, we defined the terms “performance information” and  “performance measures” in the broadest sense. To clarify this  point, we added the definition of performance information from the  2013 managers survey in the report. Moreover, as discussed  above, additional agency analysis of the root causes underlying  the use of performance information could provide some additional  context to the types of information agencies are using for decision  making.
The following 20 agencies had no comments on the draft report: the U.S.  Departments of Agriculture, Defense, Education, Energy, Health and  Human Services, Homeland Security, Housing and Urban Development,  the Interior, Justice, Labor, State, Transportation, and Veterans Affairs,  the Environmental Protection Agency, Nuclear Regulatory Commission,  Office of Personnel Management, National Science Foundation, Small  Business Administration, Social Security Administration, and the United  States Agency for International Development. The written response from  the Social Security Administration is reproduced in appendix II.
We are sending copies of this report to the agencies that participated in  our 2013 managers survey, the Director of OMB, as well as appropriate  congressional committees and other interested parties. In addition, this  report is available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix II.
section_title: Appendix I: Regression Analyses of 2013 Federal Managers Survey Results to Identify Predictors of Use of Performance Information
paragraphs: In analyzing the results of our 2013 survey, we explored whether federal  managers’ responses to certain survey questions could help explain  differences in how managers in agencies reported using performance  information. To examine which factors related to agency use of  performance information, as measured by the use of performance  information index, we conducted regression analysis. The regression  analysis allowed us to assess the unique association between our  outcome variable—the performance information index—and a given  predictor variable, while controlling for multiple other predictor variables.
To create the use of performance index, we identified survey questions  that reflected managers’ use of performance information for key  management activities and decision making. The 2013 use of  performance index included most of the questions included in our 2007  index, and additional questions from the 2013 managers survey that we  determined reflected the concept of use of performance information (see  figure 1 for specific questions included in our index). core set of items from the original index, we tested the impact of including  and excluding several additional questions related to performance  management use to ensure the cohesiveness and strength of our revised  index. Our revised index is an average of the questions used for the  index and runs from 1 to 5, where a 1 reflects that managers feel the  agency engages “to no extent” and a 5 reflecting that managers feel the  agency engages “to a very great extent” in the use of performance  information activities. We found the index met generally accepted  standards for scale reliability.
For more information on the original index we created for the 2007 ffederal managers  survey, see GAO-08-1026T.
To develop our regression model examining predictors of performance  use as measured by our index, we first identified a series of variables that  were related to one of the five practices we have previously found to  enhance or facilitate use of performance information. These practices  include: aligning agencywide goals, objectives, and measures; improving  the usefulness of performance information; developing the capacity to use  performance information; demonstrating management commitment; and  communicating performance information frequently and effectively. See  figure 3 for the specific questions related to these five practices that we  included in the regression. Although we identified other questions also  related to the five elements of effective performance management, many  of these questions were already accounted for in our use index of  performance information, and we excluded them from consideration in our  regression.
Overall, our results demonstrate that some types of management  practices and training are more positively correlated than others, with  manager perceptions of performance information use as measured by the  use index, even when controlling for other factors. Further, these results  suggest that certain specific efforts to increase agency use of  performance information—such as increasing timeliness of performance  information and providing information on the validity of performance  measures—may have a higher return than others.
To execute our analysis, we began with a base model that treated  differences in managers’ views of agency performance management use  as a function of the agency where they worked. We found that despite  statistically significant differences on average among managers at  different agencies, a regression model based on agency alone had very  poor predictive power (R-squared of .03).
We next examined whether managers’ responses to other items reflecting  the practices of effective performance management related to their  perceptions of agency use of performance information, independent of  agency. We found that several items consistently predicted increases on  individuals’ ratings of their agencies use of performance management  information, including whether managers align program performance  measures with agency goals and objectives; having information on the  validity of performance measures; and training on how to use  performance management information in decision making. We also tested  this model controlling for whether a respondent was a member of the  Senior Executive Service (SES), and found similar results.
We also tested our model with a variable to control for agency size in five  categories. We found that, relative to the largest agencies (100,000 or  more employees), managers at smaller agencies tended to rate their  agency’s use of performance information slightly lower. The significance  and magnitude of other significant variables was similar whether we  controlled for agency size or using intercepts to control for individual  agencies.
Our final model had an R-squared of .65, suggesting that the independent  variables in the model predicted approximately 65 percent of the variance  in the use index. Specific results are presented in table 2 below. Each  coefficient reflects the average increase in the dependent variable, our  five-point use scale, associated with a one-unit increase in the value of  the independent variables. Note that in our discussion, we highlight the  maximum potential impact of each variable rather than the increase in the  use score associated with each increase in a dependent variable.
As seen in table 2, at least one question related to each of the five  practices to enhance agencies’ use of performance information was  significant. With respect to aligning agencywide goals, objectives, and  measures, we found that each increase in terms of the extent to which  individuals felt that managers aligned performance measures with  agencywide goals and objectives was associated with a .13 increase in  their score on the use scale, or approximately a .52 increase on the 5- point use scale when comparing individuals in the lowest to the highest  categories.
In terms of improving the usefulness of performance information, we  found that having information on the validity of performance data for  decision making was the strongest predictor in our model. Compared to  individuals who said that they did not have sufficient information on the  validity of performance data for decision making, on average, individuals  who said they had a very great extent of information rated their agencies  approximately 0.64 points higher on the performance use scale,  controlling for other factors. In contrast, the potential effect of the  timeliness of information, while significant, had a smaller potential impact  on managers’ perceptions of their agency’s use of performance  information. On average, managers who responded “to a very great  extent” on whether their agency’s performance information was available  in time to manage programs or projects rated their agency about .28  points higher on the performance use scale than those who responded “to  no extent.”
In terms of developing agency capacity to use performance information,  we found that one type of training was positively related to use of  performance information, though other types of training were either not  related or were negatively related, after controlling for other factors.  Compared to managers who said their agencies had not trained them  with training on how to use performance information in decision making,  those who said their agencies did provide such training rated their  agencies an average of .14 points higher on the use scale, controlling for  other factors. The potential effect of this type of training was relatively  small compared to the potential effect of some of the other predictors in  our model. In contrast, training in developing performance measures was  negatively associated with managers’ perceptions of performance  information use.
With respect to demonstrating management commitment, managers that  rated their agency’s leadership highly in terms of demonstrating a strong  commitment to achieving results tended to rate their agencies higher on  performance information use, as measured by our use index. Each  increase in the extent to which a manager felt their agency leadership  was committed to results was associated with a .08 increase in the  performance use index, or up to a .32 increase in the five-point  performance use index when comparing managers who reported “no  extent” of leadership commitment to those that reported “a very great  extent.”
Two questions related to communicating performance information  frequently and effectively were significantly and positively associated with  manager’s perceptions of an agency’s use of performance information,  controlling for other factors. Compared to those who rated their agencies  the lowest in terms of whether managers and supervisors effectively  communicated performance information routinely—those managers who  rated their agencies most highly averaged .32 points higher on the five- point performance use index. Similarly, managers who reported that  employees in their agency received “a very great extent” of positive  recognition for helping the agency to accomplish strategic goals rated  their agencies an average of .24 points higher on performance  information use, as measured by our use index. We did not find a  statistically significant relationship between the accessibility of  performance information (to managers, employees or the public) and  managers’ perceptions of use of performance information.
subsections: section_title: Sensitivity and Specification Testing
paragraphs: To conduct our analysis, we used Stata software to generate regression  estimates that incorporated variance calculations appropriate for the   To ensure that large amounts of  complex design of the survey data.missing data do not result from listwise deletion, we imputed values for  individual questions if the individual is missing or indicated “no basis to  judge” on three or fewer responses from the 23 variables initially tested in  the regression, using the agency-level average to impute. Individuals  missing data on more than 3 of the 23 potential variables were dropped  from the analysis. We conducted a variety of sensitivity checks to ensure  that our results were robust across different specifications and  assumptions. For the most part, we found generally similar patterns  across models in terms of the magnitude and significance of different  variables related to the elements of effective performance management.
In general, our models assume that the relationship between the  independent and dependent variables is linear, and that changes in the  dependent variable associated with a change in the independent variable  are similar across each ordinal category. Under this specification, the  change in the use index associated with a shift from “to no extent” to “to a  small extent” is assumed to be similar to the change associated with an  increase from “to a great extent” to “a very great extent”. To determine  whether the linear specification was appropriate, or consistent with the  observed data, we tested versions of our models that treated independent  variables with a Likert-scale response as categorical. We found our  results to be robust across a variety of specifications, including those that  relaxed the assumption of linearity for responses based on a five-point  scale.
section_title: Appendix II: Comments from the Social Security Administration
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgements
paragraphs: In addition to the contact named above, Sarah Veale (Assistant Director),  Margaret McKenna Adams, Tom Beall, Mallory Barg Bulman, Chad  Clady, Karin Fangman, Cynthia Jackson, Janice Latimer, Donna Miller,  Anna Maria Ortiz, Kathleen Padulchick, Mark Ramage, Joseph Santiago,  Albert Sim, and Megan Taylor made key contributions to this report.