Managing For Results: Executive Branch Should More Fully Implement the GPRA Modernization Act to Address Pressing Governance Challenges
Published: 2013-06-26T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government faces significant and long-standing fiscal, management, and performance challenges. The acts implementation offers opportunities for Congress and the executive branch to help address these challenges. This report is the latest in a series in which GAO, as required by the act, reviewed the acts initial implementation. GAO assessed the executive branchs (1) progress in implementing the act and (2) effectiveness in using tools provided by the act to address key governance challenges. To address these objectives, GAO reviewed the act, related OMB guidance, and past and recent GAO work related to federal performance management and the act; and interviewed OMB staff. In addition, to determine the extent to which agencies are using performance information and several of the acts requirements to improve agency results, GAO surveyed a stratified random sample of 4,391 federal managers from 24 agencies, with a 69 percent response rate which allows GAO to generalize these results.
section_title: What GAO Found
paragraphs: The executive branch has taken a number of steps to implement key provisions of the GPRA Modernization Act (the act). The Office of Management and Budget (OMB) developed interim cross-agency priority (CAP) goals, and agencies developed agency priority goals (APG). Agency officials reported that their agencies have assigned performance management leadership roles and responsibilities to officials who generally participate in performance management activities, including quarterly performance reviews (QPR) for APGs. Further, OMB developed Performance.gov, a government-wide website, which provides quarterly updates on the CAP goals and APGs.
However, the executive branch needs to do more to fully implement and leverage the acts provisions to address governance challenges.
OMB and agencies have identified many programs and activities that contribute to goals, as required, but are missing additional opportunities to address crosscutting issues. For example, few have identified tax expenditures, which represented about $1 trillion in forgone revenue in fiscal year 2012, due to a lack of OMB guidance and oversight. Therefore, the contributions made by tax expenditures towards broader federal outcomes are unknown.
Ensuring performance information is useful and used by federal managers to improve results remains a weakness. GAO found little improvement in managers reported use of performance information or practices that could help promote this use. There was a decline in the percentage of managers that agreed that their agencies top leadership demonstrates a strong commitment to achieving results. However, agencies QPRs show promise as a leadership strategy for improving the use of performance information in agencies.
Agencies have taken steps to align daily operations with agency results, but continue to face difficulties measuring performance. Agencies have established performance management systems to align individual performance with agency results. However, agencies continue to face long-standing issues with measuring performance across various programs and activities. The Performance Improvement Council (PIC) could do more to examine and address these issues, given its responsibilities for addressing crosscutting performance issues and sharing best practices. Without a comprehensive examination of these issues and an approach to address them, agencies will likely continue to experience difficulties in measuring program performance.
Communication of performance information could better meet users needs. Federal managers and potential users of Performance.gov reported concerns about the accessibility, availability, understandability, and relevance of performance information to the public. Further outreach to key stakeholders could help improve how this information is communicated.
Agency performance information is not always useful for congressional decision making. Consultations with Congress are intended, in part, to ensure this information is useful for congressional decision making. However, GAO found little evidence that meaningful consultations occurred related to agency strategic plans and APGs. GAO also found that the performance information provided on Performance.gov may not fully be meeting congressional needs.
section_title: What GAO Recommends
paragraphs: GAO recommends OMB improve implementation of the act and help address challenges by ensuring that the contributions of tax expenditures to crosscutting and agency goals are identified and assessed, and developing a detailed approach for addressing long-standing performance measurement issues. OMB staff agreed with these recommendations. GAO also reports on the status of existing recommendations related to CAP goals, APGs, QPRs, the PIC, agency performance management training, and Performance.gov.

section_title: Letter
paragraphs: The federal government is one of the world’s largest and most complex  entities, with about $3.5 trillion in outlays in fiscal year 2012 funding a  vast array of programs and operations. It faces a number of significant  fiscal, management, and performance challenges in responding to the  diverse and increasingly complex issues it seeks to address. Addressing  these challenges will require actions on multiple fronts. For example,  program structures that are outmoded, fragmented, overlapping, or  duplicative and not up to the challenges of the times must be reformed or  restructured. Since 2011, our series of annual reports has identified 162  areas of potential duplication, overlap, or fragmentation as well as cost  savings and revenue-enhancing opportunities. In addition, weaknesses  in management capacity, both government-wide and in individual  agencies, undermine efficient and effective government. The recent  update to our high-risk list identified numerous opportunities to reduce  costs and improve government performance.
Moving forward, the federal government will need to make tough choices  in setting priorities as well as reforming programs and management  practices to better link resources to results. In that regard, we have  previously reported that the performance planning and reporting  framework originally put into place by the Government Performance and  Results Act of 1993 (GPRA), and significantly enhanced by the GPRA  Modernization Act of 2010 (GPRAMA or the act), provides important  tools that can help inform congressional and executive branch decision  making to address challenges the federal government faces. For  example, we recently reported on several issues that hinder the federal  government’s ability to address fragmentation, overlap, and duplication,  including the need for improved and regular performance information, the  absence of a comprehensive list of federal programs, and the lack of  related funding information. If effectively implemented, GPRAMA could  help address these issues as well as improve information sharing and  coordination among federal agencies—both of which are needed to  further address governance challenges related to fragmentation, overlap,  and duplication.
GPRAMA lays out a schedule for gradual implementation of its provisions  during a period of interim implementation—from its enactment in January  2011 to February 2014 when a new planning and reporting cycle for  federal agencies begins. GPRAMA also includes provisions requiring us  to review implementation of the act at several critical junctures and  provide recommendations to improve its implementation. This report is  the final in a series responding to the mandate to assess initial  implementation of the act by June 2013, and pulls together findings from  our recent work related to the act, the results of our periodic survey of  federal managers, and our related recent work on federal governance,  performance, and coordination issues. Our specific objectives for this  report were to assess the executive branch’s (1) progress in  implementing key provisions of the act and (2) effectiveness in using tools  provided by the act to address key governance challenges the federal  government faces. To address these objectives, we reviewed GPRAMA,  related congressional documents and Office of Management and Budget  (OMB) guidance, and our past and recent work related to managing for  results and the act. We also interviewed OMB staff. To determine the  extent to which agencies are using performance information and several  of the act’s requirements to improve agency results, we surveyed a  stratified random sample of 4,391 persons from a population of  approximately 148,300 mid-level and upper-level civilian managers and  supervisors (General Schedule levels 13 through 15 and career Senior  Executive Service (SES), or equivalent) working in the 24 executive  branch agencies covered by the Chief Financial Officers (CFO) Act of  1990, as amended. The web-based survey was administered between  November 2012 and February 2013 and is comparable to surveys we  conducted in 1997, 2000, 2003, and 2007. For this report, our focus is  on comparing the 2013 survey results with those from the 1997 baseline  survey and with the results of the 2007 survey, which is the most recent  survey conducted before GPRAMA was enacted in 2011. We noted the  results from the other two surveys—2000 and 2003—when statistically  significant trends compared to 2013 occurred. For the 2013 survey, we  received usable questionnaires from about 69 percent of the eligible  sample. The response rate across the 24 agencies ranged from 57  percent to 88 percent. The overall survey results are generalizable to the  population of managers as described above at each of the 24 agencies  and government-wide. Concurrently with this report, we are issuing an  electronic supplement that shows all of the aggregated responses to all  survey items at the government-wide and individual agency levels.
To help determine the reliability and accuracy of the database elements  used to draw our sample of federal managers for the 2013 survey, we  checked the data for reasonableness and the presence of any obvious or  potential errors in accuracy and completeness and reviewed our past  analyses of the reliability of this database. We believe the data used to  draw our sample are sufficiently reliable for the purpose of this report.  Appendix I provides additional information about our objectives, scope,  and methodology.
We conducted this performance audit from August 2012 to June 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: GPRAMA is a significant enhancement of GPRA, which was the  centerpiece of a statutory framework that Congress put in place during  the 1990s to help resolve long-standing management problems in the  federal government and provide greater accountability for results. GPRA  sought to focus federal agencies on performance by requiring agencies to  develop long-term and annual goals—contained in strategic and annual  performance plans—and measure and report on progress towards those  goals on an annual basis.
In our past reviews of its implementation, we found that GPRA provided a  solid foundation to achieve greater results in the federal government, but  several key governance challenges remained—particularly related to: addressing crosscutting issues; ensuring performance information was useful and used by agency  leadership and managers and the Congress;  strengthening the alignment between individual performance and  agency results as well as holding individuals and organizations  responsible for achieving those results; measuring performance for certain types of programs; and providing timely, useful information about the results achieved by  agencies.
To help address these and other challenges, GPRAMA revises existing  provisions and adds new requirements, including the following:
Cross-agency priority (CAP) goals: OMB is required to coordinate  with agencies to establish federal government priority goals— otherwise referred to as CAP goals—that include outcome-oriented  goals covering a limited number of policy areas as well as goals for  management improvements needed across the government. The act  also requires that OMB—with agencies—develop annual federal  government performance plans to, among other things, define the  level of performance to be achieved toward the CAP goals.
Agency priority goals (APGs): Certain agencies are required to  develop a limited number of APGs every 2 years. Both the agencies  required to develop these goals and the number of goals to be  developed are determined by OMB. These goals are to reflect the  highest priorities of each selected agency, as identified by the head of  the agency, and be informed by the CAP goals as well as input from  relevant congressional committees.
Leadership positions: Although most of these positions previously  existed in government, they were created by executive orders,  presidential memoranda, or OMB guidance. GPRAMA established  these roles in law, provided responsibilities for various aspects of  performance improvement, and elevated some of them.
Chief operating officer (COO): The deputy agency head, or  equivalent, is designated COO, with overall responsibility for  improving agency management and performance.
Performance improvement officer (PIO): Agencies are required to  designate a senior executive within the agency as PIO, who  reports directly to the COO and has responsibilities to assist the  agency head and COO with performance management activities.
Goal leader: For each CAP goal, OMB must identify a lead  government official—referred to by OMB as a goal leader— responsible for coordinating efforts to achieve each of the goals.  For agency performance goals, including APGs, agencies must  also designate a goal leader, who is responsible for achieving the  goal.
Performance Improvement Council (PIC): Originally created by a  2007 executive order, GPRAMA establishes the PIC in law and  included additional responsibilities. The PIC is charged with assisting  OMB to improve the performance of the federal government and  achieve the CAP goals. Among its other responsibilities, the PIC is to  facilitate the exchange among agencies of useful performance  improvement practices and work to resolve government-wide or  crosscutting performance issues. The PIC is chaired by the Deputy  Director for Management at OMB and includes agency PIOs from  each of the 24 CFO Act agencies as well as other PIOs and  individuals designated by the chair.
Quarterly performance reviews (QPR): For each APG, agencies are  required to conduct QPRs to review progress towards the goals and  develop strategies to improve performance, as needed. These  reviews are to be led by the agency head and COO and include the  PIO, relevant goal leaders, and other relevant parties both within and  outside the agency.
Performance.gov: OMB is required to develop a single, government- wide performance website to communicate government-wide and  agency performance information. The website—implemented by OMB  as Performance.gov—is required to make available information on  APGs and CAP goals, updated on a quarterly basis; agency strategic  plans, annual performance plans, and annual performance reports;  and an inventory of all federal programs.
Performance management capacity: The Office of Personnel  Management (OPM) is charged with three responsibilities under the  act. OPM is to (1) in consultation with the PIC, identify key skills and  competencies needed by federal employees to carry out a variety of  performance management activities; (2) incorporate these skills and  competencies into relevant position classifications; and (3) work with  agencies to incorporate these key skills into agency training.
section_title: The Executive Branch Has Taken Important Steps to Implement Key GPRAMA Provisions
paragraphs: Since GPRAMA’s enactment in January 2011, OMB and agencies have  taken a number of important steps to implement key provisions related to  the act’s planning and reporting requirements. In February 2012, OMB  identified 14 interim CAP goals concurrent with the submission of the  President’s Budget. Nine of the goals related to crosscutting policy areas  and 5 covered management improvements. In addition, at the same  time, 24 agencies selected by OMB developed 103 APGs for 2012 and  2013, and OMB published information about these goals as well as the  CAP goals on Performance.gov, which OMB considers to comprise the  federal government performance plan. In December 2012, OMB  expanded the information available on the site by providing an update on  fiscal year 2012 performance for both sets of goals, and in March 2013,  quarterly updates of the site began. All 24 CFO Act agencies are  conducting QPRs, according to our survey of PIOs at these agencies.  Our 2013 survey indicates that approximately one-third (33 percent) of  federal managers across the government are at least somewhat familiar  with the QPRs. These and related efforts were based on OMB guidance  on implementing the act issued in 2011 and 2012.
As another positive development, OMB and agencies have also put into  place key aspects of the act’s performance management leadership roles.  We recently reported that, at the agency level, all 24 CFO Act agencies  have assigned senior-level officials to the COO, PIO, and goal leader  roles. Furthermore, OMB guidance directed agencies with PIOs who are  political appointees or other officials with limited-term appointments to  appoint a career senior executive to serve as deputy PIO. Nearly all (22)  of the CFO Act agencies have assigned officials to the deputy PIO role,  according to our PIO survey. PIOs we surveyed reported that most  performance management officials (COOs, PIOs, deputy PIOs and goal  leaders) had large involvement in four primary tasks that summarize the  performance management responsibilities required by GPRAMA: (1)  strategic and performance planning and goal setting, (2) performance  measurement and analysis, (3) communicating agency progress toward  goals, and (4) agency quarterly performance reviews.
At the government-wide level, the PIC has taken steps to meet its  requirement to facilitate the exchange of useful practices and tips and  tools to strengthen agency performance management. For example, it  established the Goal Setting Working Group to help agencies set their  2012 to 2013 APGs; the Internal Agency Reviews Working Group to  share best practices for QPRs; and the Business Intelligence Working  Group to share tools for data analytics. PIOs we surveyed reported that,  in general, they found the PIC helpful and that there was strong agency  participation in the PIC and its working groups. However, in April 2013 we  reported that the PIC has not routinely assessed its performance and  recommended that OMB work with the PIC to  conduct formal feedback on the PIC’s performance from member  agencies on an ongoing basis; and update the PIC’s strategic plan and review the PIC’s goals, measures,  and strategies for achieving performance, and revise them if  appropriate.
OMB staff agreed with these recommendations.
In addition, OPM has completed its work identifying key skills and  competencies needed by performance management staff and  incorporating those skills and competencies into relevant position  classifications. OPM identified 15 competencies for performance  management staff and published them in a January 2012 memorandum  from the OPM Director. It also identified relevant position classifications  that are related to the competencies for performance management staff  and worked with a PIC working group to develop related guidance and  tools for agencies. Furthermore, OPM has taken steps to work with  agencies to incorporate the key competencies into agency training.  However, we reported in April 2013 that these efforts have been broad- based and not informed by specific assessments of agency training  needs. We recommended that, in coordination with the PIC and the  Chief Learning Officers Council, OPM (1) identify competency areas  needing improvement within agencies, (2) identify agency training that  focuses on needed performance management competencies, and (3)  share information about available agency training on competency areas  needing improvement. OPM agreed with these recommendations and  reported that it will take actions to implement them.
section_title: The Executive Branch Needs to More Fully Use the GPRAMA Framework to Address Pressing Federal Governance Challenges
subsections: section_title: OMB and Agencies Have Made Some Progress Addressing Crosscutting Issues, but Are Missing Additional Opportunities
paragraphs: Many of the meaningful results that the federal government seeks to  achieve, such as those related to protecting food and agriculture and  providing homeland security, require the coordinated efforts of more than  one federal agency, level of government, or sector. However, agencies  face a range of challenges and barriers when they attempt to work  collaboratively. The need for improved collaboration has been  highlighted throughout our work over many years, in particular in two  bodies of work. First, our reports over the past 3 years identified more  than 80 areas where opportunities exist for executive branch agencies or  Congress to reduce fragmentation, overlap, and duplication. Figure 1  defines and illustrates these terms.
We found that resolving many of these issues requires better  collaboration among agencies. Second, collaboration and improved  working relationships across agencies are fundamental to many of the  issues that we have designated as high risk due to their vulnerabilities to  fraud, waste, abuse, and mismanagement, or most in need of  transformation.
For almost 2 decades we have reported on agencies’ missed  opportunities for improved collaboration through the effective  implementation of GPRA. In our 1997 assessment of the status of the  implementation of GPRA, we reported that agencies faced challenges  addressing crosscutting issues, which led to fragmentation and overlap.  Again, we reported in 2004—10 years after the enactment of GPRA—that  there was still an inadequate focus on addressing issues that cut across  federal agencies. On a government-wide level, we reported that OMB  did not fully implement a government-wide performance plan, as was  required by GPRA. Additionally, few agency strategic and performance  plans addressed crosscutting efforts and coordination. At that time,  almost half of federal managers in our 2003 survey reported that they  coordinated program efforts to a great or very great extent with other  internal or external organizations. Now, almost 20 years since GPRA’s  passage, our work continues to demonstrate that the needed  collaboration is not sufficiently widespread. Accordingly, in 2012 we  developed a guide on key considerations for implementing collaborative  mechanisms. The results of our 2013 survey of federal managers show  that the percentage of managers reporting that they use information  obtained from performance measurement when coordinating program  efforts with other internal or external organizations to a great or very great  extent has not increased since 1997. Based on this survey, an estimated  23 percent of the managers reported that they coordinated program  efforts to a small extent or not at all.
The following three examples, among many, highlight the need for  improved collaboration to help address crosscutting issues:
Food safety: One area that has been identified in both bodies of work  is the fragmented nature of federal food safety oversight. The U.S.  food safety system is characterized by inconsistent oversight,  ineffective coordination, and inefficient use of resources; these  characteristics have placed the system on our high-risk list since 2007  and in all three of our annual reports on fragmentation, overlap, and  duplication. We have reported that the U.S. Department of  Agriculture (USDA) and the Food and Drug Administration (FDA), the  two primary agencies responsible for food safety, have taken some  steps to increase collaboration. However, agencies have not  developed a government-wide performance plan for food safety that  includes results-oriented goals and performance measures, as we  recommended when we put federal oversight of food safety on the  high-risk list in January 2007. In the absence of this plan, we have  reported cases of fragmentation, overlap, and duplication. The 2010  nationwide recall of more than 500 million eggs because of  Salmonella contamination highlights a negative consequence of this  fragmentation. Several agencies have different roles and  responsibilities in the egg production system. Through the Food  Safety Working Group, federal agencies have taken steps designed  to increase collaboration in some areas that cross regulatory  jurisdictions. For example, both USDA and FDA set goals to reduce  illness from Salmonella within their own areas of egg safety  jurisdiction by the end of 2011 and developed a memorandum of  understanding on information sharing regarding egg safety. While  such actions are encouraging, without a government-wide  performance plan for food safety, fragmentation, overlap, and  duplication is likely to continue.
Climate change: Climate change is a complex, crosscutting issue  that poses risks to many environmental and economic systems— including agriculture, infrastructure, ecosystems, and human health— and presents a significant financial risk to the federal government.  Among other impacts, climate change could threaten coastal areas  with rising sea levels, alter agricultural productivity, and increase the  intensity and frequency of severe weather events such as floods,  drought, and hurricanes. Weather-related events have cost the nation  tens of billions of dollars in damages over the past decade. For  example, in 2012, the administration requested $60.4 billion for  Superstorm Sandy recovery efforts. However, the federal government  is not well positioned to address the fiscal exposure presented by  climate change, partly because of the complex, crosscutting nature of  the issue. Given these challenges and the nation’s precarious fiscal  condition, we added “Limiting the Federal Government’s Fiscal  Exposure to Climate Change” to our high-risk list in 2013. In adding  climate change to this list, we reported that the federal government  would be better positioned to respond to the risks posed by climate  change if federal efforts were more coordinated and directed toward  common goals. In October 2009, we recommended that the  appropriate entities within the Executive Office of the President, in  consultation with relevant federal agencies, state and local  governments, and key congressional committees of jurisdiction,  develop a strategic plan to guide the nation’s efforts to adapt to  climate change, including the establishment of clear roles,  responsibilities, and working relationships among federal, state, and  local governments. In written comments, the Council on  Environmental Quality generally agreed with the report’s  recommendations, noting that leadership and coordination is  necessary within the federal government to ensure an effective and  appropriate adaptation response and that such coordination would  help to catalyze regional, state, and local activities. Some actions  have subsequently been taken to improve the coordination of federal  adaptation efforts, including the development of an interagency  climate change adaptation task force.
Federal disability programs: In June 2012, we identified 45  programs in nine agencies that helped people with disabilities obtain  or retain employment, reflecting a fragmented system of services and  supports. Many of these programs overlapped in whom they served  and the types of services they provided. Such fragmentation and  overlap may frustrate and confuse program beneficiaries and limit the  overall effectiveness of the federal effort. Having extensive  coordination and overarching goals can help address program  fragmentation. Although we identified promising coordination efforts  among some programs, most reported not coordinating with each  other, and some officials told us they lacked funding and staff time to  pursue coordination. Coordination efforts can be enhanced when  programs work toward a common goal; however, the number and type  of outcome measures used by the 45 programs varied greatly. To  improve coordination, efficiency, and effectiveness, we suggested that  OMB consider establishing government-wide goals for employment of  people with disabilities. Consistent with this suggestion, OMB officials  stated that the Domestic Policy Council began an internal review  intended to improve the effectiveness of some disability programs  through better coordination and alignment. However, as we noted in  our 2013 high-risk update, OMB still needs to maintain and expand its  role in improving coordination across programs—such as the 45 we  identified—that support employment for those with disabilities, and  ultimately work with all relevant agencies to develop measurable  government-wide goals to spur further coordination and improved  outcomes for those who are seeking to find and maintain  employment.
On the other hand, we have recently highlighted progress that the  executive branch and Congress have made in addressing areas that we  previously identified as being at risk of fragmentation, overlap, and  duplication. For example, the nation’s surface transportation system is  critical to the economy and affects the daily life of most Americans.  However, in our 2011 annual report on fragmentation, overlap, and  duplication, we reported that over the years federal surface transportation  programs grew increasingly fragmented. At the core of this  fragmentation was the fact that federal goals and roles for the programs  were unclear or conflicted with other federal priorities, programs lacked  links to the performance of the transportation system or of the grantees,  and programs did not use the best tools to target investments in  transportation to the areas of greatest benefit. Accordingly, since 2004,  we have made several recommendations and matters for congressional  consideration to address the need for a more goal-oriented approach to  surface transportation, introduce greater performance and accountability  for results, and break down modal stovepipes. As we reported in  February 2013, there was progress in clarifying federal goals and roles  and linking federal programs to performance when the Moving Ahead for  Progress in the 21st Century Act was enacted in July 2012. The act  addressed fragmentation by eliminating or consolidating programs, and  made progress in clarifying federal goals and roles and linking federal  programs to performance to better ensure accountability for results.
The challenge of collaboration has also been highlighted in our reviews of  related GPRAMA requirements, such as those for CAP goals, APGs, and  QPRs. While agencies have implemented some of these provisions,  these efforts have not included all of the relevant agency, program, and  other contributors. When agencies do not include all relevant contributors,  they may miss important opportunities to work with others who are  instrumental to achieving intended outcomes.
Including all contributors is also a requirement of GPRAMA.
At the government-wide level, OMB is required to list all of the  agencies, organizations, program activities, regulations, tax  expenditures, policies, and other activities that contribute to each CAP  goal. With relevant stakeholders, OMB is required to review the  progress of all contributors towards each goal on a quarterly basis.
At the agency level, agencies are required to identify the various  federal organizations, programs, and activities—both within and  external to the agency—that contribute to each goal, and for APGs,  review progress on a quarterly basis with relevant stakeholders.
However, as shown in table 1, we have found that agencies are not  including all stakeholders as they implement GPRAMA.
While we continue to see challenges to collaboration across federal  agencies, as a positive development, our survey of federal managers  shows that reported collaboration increases when individuals contribute to  the CAP goals, APGs, or QPRs. Our 2013 survey data indicate that 58  percent of federal managers reported they were somewhat or very  familiar with CAP goals. Among these individuals, federal managers who  viewed their programs as contributing to CAP goals to a great or very  great extent were more likely to report collaborating outside their program  to a great or very great extent to help achieve CAP goals, as figure 2  shows.
We saw a similar pattern in responses from managers who were familiar  with the APGs and the extent to which their programs contributed to the  APGs. Eighty-two percent of federal managers reported they were  somewhat or very familiar with APGs. Among these individuals, those  who viewed their programs as contributing to APGs to a great or very  great extent were more likely to report collaborating outside their program  to a great or very great extent to help achieve APGs, as figure 3 shows.
While the questions on our survey were designed to examine  collaboration outside individual programs, they were not designed to  distinguish between collaboration within or outside agency boundaries. As  discussed in table 1, we found that collaboration was more common  within agencies than between agencies. This may be appropriate in some  cases; however, in other cases this might point to a need for broader  inclusion of external stakeholders. We found that more managers  reported collaborating with officials external to their agency to a great or  very great extent when they also reported that their programs were  involved in QPRs to a similar extent.
Tax expenditures represent a significant federal investment. If the  Department of the Treasury (Treasury) estimates are summed, an  estimated $1 trillion in revenue was forgone from the 169 tax  expenditures reported for fiscal year 2012, nearly the same as  discretionary spending that year. For some tax expenditures, forgone  revenue can be of the same magnitude or larger than related federal  spending for some mission areas. For example, in fiscal year 2010, tax  expenditures represented about 78 percent ($132 billion) of federal  support for housing. Since 1994, we have recommended greater scrutiny  of tax expenditures, as periodic reviews could help determine how well  specific tax expenditures work to achieve their goals and how their  benefits and costs compare to those of spending programs with similar  goals. In November 2012, we issued a guide that identifies criteria for  assessing tax expenditures and provides questions for the Congress to  ask about a tax expenditure’s effectiveness. However, OMB has not  developed a framework for reviewing tax expenditure performance, as we  recommended in June 1994 and again in September 2005. Because  OMB has not yet established such a framework, little is known about how  tax expenditures contribute to broad federal outcomes and how they are  related to spending programs seeking the same or a similar outcome.
OMB guidance has shown some progress in addressing how agencies  should incorporate tax expenditures in strategic plans and annual  performance plans and reports, as we first recommended in September  2005. GPRAMA specifically requires OMB to identify tax expenditures  among the various federal activities that contribute to each CAP goal,  when applicable. Although the act does not explicitly require agencies to  identify tax expenditures among the various federal programs and  activities that contribute to their performance goals, OMB’s guidance  directs agencies to do so for their APGs, which are a small subset of their  performance goals. However, our review of the APGs developed for 2012  to 2013 found that only one agency, for one of its APGs, identified two  relevant tax expenditures. We recently reported that OMB was missing an  opportunity to more broadly identify how tax expenditures contribute to  each agency’s overall performance.
Even among the CAP goals, OMB and agencies are missing  opportunities to identify tax expenditures as contributors. In the original  information on Performance.gov in February 2012, OMB included tax  expenditures as potential contributors for 5 of the 14 CAP goals (veteran  career readiness, entrepreneurship and small businesses, energy  efficiency, job training, and improper payments). In the December 2012  and March 2013 updates to Performance.gov, only two goals (veteran  career readiness and improper payments) discussed two tax  expenditures, which represent $2.7 billion or 0.3 percent of the $1 trillion  sum across the tax expenditures listed by Treasury. Tax expenditures  were no longer mentioned as contributing to the entrepreneurship and  small businesses, energy efficiency, and job training CAP goals. For  example, under the energy efficiency CAP goal, OMB originally listed  both spending programs and tax expenditures that contribute to the goal.  However, in the December 2012 update to Performance.gov, OMB had  deleted all of the tax expenditures even though many of these tax  expenditures remained unchanged. In one case, OMB deleted the credit  for energy efficiency improvements to existing homes (estimated at $780  million for fiscal year 2012), but highlighted the Department of Energy’s  (DOE) weatherization assistance spending program (estimated at $68  million in obligations for fiscal year 2012), even though both fund  residential energy efficiency. Overall, we identified eight tax expenditures,  totaling $2.4 billion in forgone revenue, which share the purpose of  achieving energy efficiency, but are no longer identified as potential  contributors. When asked about these changes, OMB staff shared that for  the entrepreneurship and small business CAP goal the goal leaders  narrowed the focus of the goal, which resulted in an updated list of  contributing programs and activities that no longer included tax  expenditures. For the energy efficiency and job training CAP goals, OMB  staff told us that the exclusion of tax expenditures from the December  2012 and March 2013 updates was an oversight. OMB staff told us they  planned to add the appropriate tax expenditures as contributors to those  goals in the next quarterly update to Performance.gov, which occurred in  June 2013. However, none were added to the job training CAP goal  update, and as of June 19, 2013, the energy efficiency CAP goal had not  yet been updated.
However, these examples raise concerns as to whether OMB previously  ensured all relevant tax expenditures were identified as contributors to the  14 CAP goals when they were published in February 2012, especially  since only 5 CAP goals listed tax expenditures as contributors at that  time. We have previously reported that, as with spending programs, tax  expenditures represent a substantial federal commitment to a wide range  of mission areas. Given the lack of scrutiny tax expenditures receive  compared to spending programs—especially absent a comprehensive  framework for reviewing them—it is possible that additional tax  expenditures should have been identified and included as contributors to  one or more of the other 9 CAP goals. Moreover, for the 2 CAP goals  where tax expenditures were listed as contributors and mistakenly  removed, it is unclear if OMB and the goal leaders assessed the  contributions of those tax expenditures toward the CAP goal efforts, since  they were not listed in the December 2012 and March 2013 updates.  Without information about which tax expenditures support these goals  and measures of their performance, Congress and other decision makers  will not have the needed information to assess overall federal  contributions towards desired results, and the costs and relative  effectiveness associated with those contributions.
section_title: Ensuring Performance Information Is Useful and Used by Managers to Improve Results Remains a Weakness, but Key Performance Management Practices Hold Promise
subsections: section_title: GAO Continues to Find Widespread Progress Is Needed to Use Data to Drive Performance
paragraphs: We have previously reported that data-driven decision making leads to  better results. Moreover, we have reported that if agencies do not use  performance measures and performance information to track progress  toward goals, they may be at risk of failing to achieve their goals. The  textbox illustrates this problem in the high risk area of the Department of  Defense’s (DOD) approach to business transformation.
DOD Is Not Regularly Reviewing Performance Information to Assess Progress  towards Goals in Transforming Its Business Operations In 2005, we identified DOD’s approach to business transformation as high-risk because  DOD had not established clear and specific management responsibility, accountability and  control over its business transformation and it lacked a plan with specific goals, measures,  and mechanisms to monitor progress. We subsequently reported that DOD made  improvements to strengthen its management approach, but we also identified additional  steps that are needed. For example, DOD has broadly outlined a performance  management approach, and established governance structures, such as the Defense  Business Council, to help monitor progress in its business transformation efforts.  However, we found the Council had not regularly reviewed performance data and when  reviews did occur, it did not have sufficient information to assess progress. To enhance  DOD’s ability to set strategic direction for its business transformation efforts, better assess  overall progress toward business transformation goals, and take any necessary corrective  actions, we recommended in February 2013 that DOD take a number of steps to improve  its approach to performance management. DOD agreed with this recommendation and  said it would continue to improve and institutionalize the Council’s operations.
In the first 4 months of 2013 alone, we issued numerous testimonies and  reports that illustrate how performance management weaknesses can  hinder agencies’ abilities to achieve critical results. This work also  illustrates that the scope of these problems is widespread, affecting  agencies such as DOD, Treasury, the Departments of Transportation  (DOT), Homeland Security (DHS), Health and Human Services, Housing  and Urban Development (HUD), and State. The impact of these  weaknesses is far reaching as well: These agencies are responsible for  performing functions that affect every aspect of Americans’ lives, from  education, healthcare, and housing to national security and illicit drug  use, as described in the textbox.
Office of National Drug Control Policy Has Established a Performance Monitoring  System to Address Illicit Drug Use, but Not Yet Reported on Results The public health, social, and economic consequences of illicit drug use, coupled with the  nation’s constrained fiscal environment, highlight the need for federal programs to use  resources efficiently and effectively to address this problem. However, we reported in  March 2013 that the Office of National Drug Control Policy and federal agencies have not  made progress toward achieving most of the goals in the 2010 National Drug Control  Strategy, although they reported to be on track to implement most Strategy action items in  support of these goals. In April 2012, the Office established the Performance Reporting  System, a monitoring mechanism intended to provide specific, routine information on  progress toward Strategy goals and help identify factors for performance gaps and options  for improvement. We reported that this could help increase accountability for improving  results and identify ways to bridge the gap that existed between the lack of progress  toward the Strategy’s goals and the strong progress made on implementing the Strategy’s  actions. While this was promising, the Office does not plan to report on results until later in  2013, and until then, operational information is not available to evaluate its effectiveness.
GAO, Office of National Drug Control Policy: Office Could Better Identify Opportunities to Increase  Program Coordination, GAO-13-333, (Washington, D.C.: Mar. 26, 2013).
section_title: Surveys of Federal Managers Show that Managers’ Use of Performance Information for Decision Making Has Stagnated
paragraphs: Our prior work has shown that performance information can be used  across a range of management functions to improve results, from setting  program priorities and allocating resources to taking corrective action to  solve program problems. Since our 2007 survey there was statistically  significant improvement on two survey items related to use of  performance information. More managers reported in 2013—after  GPRAMA’s enactment and initial implementation—that they used  performance information to a great or very great extent in developing  program strategy and refining program performance measures. However,  the 2013 improvement on the refining program performance measures  item followed an earlier decline and does not represent an improvement  in comparison to our 1997 survey results. While there was also a  statistically significant change between 1997 and 2013 in the percentage  of managers who reported to a great or very great extent that they used  performance information in adopting new program approaches or  changing work processes, the initial decline on this item occurred  between our 1997 and 2000 surveys with no significant changes since  then. Overall, our periodic surveys of federal managers since 1997  indicate that with the few exceptions described above, the use of  performance information has not changed significantly at the government- wide level, as shown in figure 4.
In addition, we introduced an item in the 2013 survey on streamlining  programs, a performance management activity that can help address the  overlap and duplication challenges and opportunities described earlier in  this report. Less than half of federal managers (44 percent) reported to a  great or very great extent that they used performance information for  “streamlining programs to reduce duplicative activities.”
Our prior work has identified practices that can promote the use of  performance information for management decision making, such as  leadership demonstrating commitment to using performance information,  communicating performance information frequently and effectively,  ensuring that performance information is useful, and building capacity to  use performance information. Moreover, many of the requirements put  in place by GPRAMA reinforce the importance of these practices. Our  past government-wide surveys of federal managers indicated that these  key practices were not always being employed across various agencies.  Our 2013 survey suggests that effectively adopting these practices  continues to be a substantial weakness across the government as  described below.
Demonstrating leadership commitment: Our prior work has shown that  the demonstrated commitment of leadership and management to  achieving results and using performance information can encourage the  federal workforce to apply the principles of performance management.  GPRAMA requires top leadership involvement in performance  management, such as requiring agency leadership to routinely review  performance information and progress toward APGs during the QPRs.  However, results from our 2013 survey show almost no statistically  significant changes in managers’ perceptions of their leaders’ and  supervisors’ attention and commitment to the use of performance  information since our last survey in 2007. The only statistically significant  change from 2007 to 2013 was a decline in the percentage of managers  that agreed to a great or very great extent that their agencies’ top  leadership demonstrates a strong commitment to achieving results, from  67 percent to 60 percent. Moreover, less than two-thirds of managers  agreed to a great or very great extent with other survey items related to  leadership commitment and attention to performance information, as  shown in figure 5.
Communicating performance information: Our prior work showed that  communicating performance information frequently and effectively  throughout an agency can help managers to inform staff and other  stakeholders of their commitment to achieve the agency’s goals and to  keep these goals in mind as they pursue their day-to-day activities.  Frequently reporting progress toward achieving performance targets also  allows managers to review the information in time to make  improvements. GPRAMA includes requirements for communicating  performance information, such as sharing performance information at  least quarterly and directing agencies to update performance indicators  on their websites at least annually. However, there was no statistically  significant change between 2007 and 2013 in the percentage of federal  managers agreeing to a great or very great extent that agency managers  at their level effectively communicate performance information on a  routine basis (41 percent in 2013 and 43 percent in 2007). Our analysis  suggests that easy access to performance information is related to the  effective communication of performance information. Of the 49 percent of  federal managers who agreed to a great or very great extent that  performance information is easily accessible to managers at their level,  62 percent also agreed that agency managers at their level effectively  communicate performance information on a routine basis to a great or  very great extent. Conversely, of the 19 percent that agreed to only a  small or no extent that performance information is easily accessible to  managers at their level, only 9 percent also agreed that agency managers  at their level effectively communicate performance information on a  routine basis to a great or very great extent.
Ensuring performance information is useful: As we previously  reported, to facilitate the use of performance information, agencies should  ensure that information meets various users’ needs for completeness,  accuracy, consistency, timeliness, validity, and ease of use. GPRAMA  introduced several requirements that could help to address these various  dimensions of usefulness. For example, agencies must disclose more  information about the accuracy and validity of their performance data and  actions to address limitations to the data. Without useful performance  information, it is difficult to monitor agencies’ progress toward critical  goals, such as improving veterans’ access to health care provided by the  Department of Veterans Affairs (VA), as illustrated in the textbox.
Performance Information on Veterans’ Wait Times for Medical Appointments Was  Unreliable The Veterans Health Administration (VHA), within the VA, provided nearly 80 million  outpatient medical appointments to veterans in fiscal year 2011. Although access to timely  medical appointments is important to ensuring veterans obtain needed care, long wait  times and inadequate scheduling processes have been persistent problems. VHA is  implementing a number of initiatives to improve veterans’ access to medical appointments  such as use of technology to interact with patients and provide care. However, we testified  in March 2013 that certain aspects of VHA’s policies and policy implementation  contributed to unreliable performance information on veterans’ wait times. VA concurred with our  recommendations and identified actions planned or under way to address them.
GAO, VA Health Care: Appointment Scheduling Oversight and Wait Time Measures Need  Improvement, GAO-13-372T (Washington, D.C.: Mar. 14, 2013).
Responses to four survey items on hindrances related to the usefulness  of performance information indicate some limited improvement. There  was a statistically significant improvement between the 2007 and 2013  surveys on two of these four items (shown as declines because they  concern hindrances), but no significant change otherwise, as illustrated in  figure 6.
In addition, related survey items introduced after 1997 showed no  significant change between 2007 and 2013, with about 40 percent of  managers agreeing to a great or very great extent that “agency managers  at my level take steps to ensure that performance information is useful  and appropriate” and 36 percent agreeing to the same extent that “I have  sufficient information on the validity of the performance data I use to  make decisions.”
Despite these limited improvements, the overall picture from the 2013  results—with about one-fifth to nearly one-third of managers reporting  hindrances, as indicated in figure 6, and less than half agreeing with most  of the positive statements about the format, timeliness, and accessibility  of their performance information in figure 7—remains a major concern.
Building capacity to use performance information: We have  previously reported that building the capacity to use performance  information is critical to using performance information in a meaningful  fashion, and that inadequate staff expertise, among other factors, can  hinder agencies from using performance information. GPRAMA lays out  specific requirements for OPM to identify skills and competencies for  performance management functions, among other actions, which  reinforce the importance of staff capacity to use performance  information. Managers’ survey responses and our recent work indicate  areas of weakness in agencies’ analysis and evaluation tools and staff’s  skills and competencies, both of which are critical components of  performance management capacity. About a third (36 percent) of  managers reported in 2013 that they agreed to a great or very great  extent that their agencies have sufficient analytical tools for managers at  their levels to collect, analyze, and use performance information.  Furthermore, less than a third of managers reported that their agencies  were investing resources to improve the use and quality of performance  information. Thirty percent of managers reported that they agree to a  great or very great extent that the programs they are involved with have  sufficient staff with the knowledge and skills needed to analyze  performance information.
Additionally, our recent work found gaps in performance management  competencies among agency staff. Although PIOs we surveyed at 24  agencies in 2012 for our April 2013 report on performance management  leadership roles reported that their staff generally possessed core  competencies identified by OPM for performance management staff,  certain competencies—performance measurement, information  management, organization performance analysis, and planning and  evaluating—were present to a lesser extent.
Training is one way agencies can address a lack of staff capacity to use  performance information, as illustrated in the sidebar. Between 1997 and  2013, there was a statistically significant increase in the percentage of  managers reporting that their agencies made training available in the past  3 years on most of the performance management tasks we asked about.  However, between 2007 and 2013, there was either no significant change  or a decline in the percentage of managers responding positively to the  same items, as shown in figure 8.
Our prior work has indicated that effective data-driven reviews can serve  as a leadership strategy, requiring leadership and other responsible  parties to come together to review performance information and progress  toward results and identify important opportunities to drive performance  improvements. According to our 2012 survey of PIOs at 24 agencies, the  majority (21 of 24) reported that actionable opportunities for performance  improvement are identified through the reviews at least half the time. In  addition, most officials we interviewed at DOE, Treasury, and the Small  Business Administration (SBA) attributed improvements in performance  and decision making to their QPRs. The textbox presents one such  improvement described by officials at Treasury.
Treasury Credits QPRs with Decision to Stop Minting $1 Coins for Circulation and  Saving U.S. Government Millions  Treasury’s Deputy Secretary said that it was a performance review session with the U.S.  Mint that first led him to question the direction they had been taking with the $1 coin.  Performance data he reviewed for the meeting indicated that the Mint was producing 400  million new $1 coins annually, while the Federal Reserve already had 1.4 billion existing  ones in storage. Digging deeper, he learned that the Federal Reserve had previously  estimated that there were enough $1 coins to meet demand for more than a decade.
This estimate was based on the assumption that demand would remain at 2012 levels.
While our case studies and survey of PIOs indicated the benefits of  QPRs, our 2013 government-wide federal managers’ survey indicated  that the majority of federal managers are not familiar with the QPRs at  their agencies, although a greater percentage of Senior Executive Service  (SES) managers reported that they were familiar with the QPRs, as  shown in figure 9.
Our analysis suggests that, while familiarity with QPRs may be somewhat  limited government-wide, it is positively related to managers’ perceptions  of their leadership’s demonstrated commitment to using performance  information. Of the 12 percent of all federal managers who reported they  were very familiar with QPRs, 76 percent agreed that their top leadership  demonstrates a strong commitment to using performance information to  guide decision making to a great or very great extent. In contrast, of the  66 percent who reported they were not familiar with QPRs, 36 percent  agreed to a great or very great extent with the same statement.
Similarly, our analysis suggests that being the subject of a QPR is  positively related to the extent to which managers view the QPRs as  being used to accomplish certain purposes to a great or very great extent.  For example, federal managers who reported that their programs have  been the subject of a QPR to a great or very great extent were more likely  to report that their agencies use QPRs to identify problems or  opportunities than those who reported that their programs have been the  subject of a QPR to a moderate or small or no extent. Figure 10 shows  this trend, along with a similar one for federal managers’ ratings of  agency leadership use of QPRs to help achieve performance goals.
Our analysis also suggests that being the subject of a QPR may be  positively related to managers’ perceptions of their agencies employment  of key practices that we have previously reported can promote successful  data-driven performance reviews. For example, federal managers who  reported that their programs have been the subject of a QPR to a great or  very great extent were more likely to report that the reviews included key  practices, such as leadership actively participating in reviews, than those  who reported that their programs have been the subject of QPRs to a  moderate or small or no extent. This trend and similar ones for other key  practices are shown in figure 11.
Federal managers’ responses to items about other key practices—holding  QPRs on a regular, routine basis and having a process for following up on  QPRs—were similarly related to the extent to which managers’ programs  were the subject of a QPR.
section_title: Agencies Have Taken Steps to Align Daily Operations with Agency Results, but Some Continue to Face Difficulties Measuring Performance
subsections: section_title: Agencies Have Taken Steps to Align Individual Performance with Results
paragraphs: It is important for individuals to see a connection between their daily  operations and results to help understand how individual performance  can contribute to organizational success. While our past work has shown  that agencies have encountered challenges linking individual  performance with broader organizational results, progress has been  made over the last decade in establishing this linkage and holding  individuals accountable for organizational results through performance  management systems. For example, while agencies have been required  to hold senior executives accountable for their individual and  organizational performance by linking performance expectations with  GPRA-required goals since 2000, OPM and OMB have continued to  reinforce the importance of this alignment in improvements in SES  performance management. Most recently, in January 2012, OPM and  OMB released a government-wide performance appraisal system for  senior executives that provides agencies with a standard framework for  managing the performance of its executives. While striving to provide  greater clarity and equity in the development of performance standards  and link to compensation, among other things, the Directors of OPM and  OMB stated that the new system is intended to provide agencies with the  necessary flexibility and capability to customize the system in order to  meet their needs. As part of this framework, agencies are to identify  expectations for the senior executives that focus on measurable  outcomes from the strategic plan or other measurable outputs and  outcomes clearly aligned to organizational goals and objectives.
In addition, the Goals-Engagement-Accountability-Results (GEAR) model,  established in 2011, focuses on aligning employee performance with  organizational performance, creating a culture of engagement, and  implementing accountability at all levels, among other things. The GEAR  model outlines a series of recommended actions for agencies to adopt in  order to help improve employee and organizational performance. We  reported in September 2012 that DOE’s GEAR implementation plan  includes aligning employee performance management with organizational  performance management and developing training to support these  goals, which along with initiating knowledge-sharing activities, will  promote improvement of DOE’s organizational performance, according to  DOE officials. We have ongoing work looking at GEAR implementation  in the five pilot agencies and plan to issue the results of our work later in  2013.
To further institutionalize individual accountability for achieving results,  GPRAMA established in law several mechanisms that help individuals  and agencies see this connection and hold them accountable for their  contributions to agency and government-wide goals. As we recently  reported, agency leaders should hold goal leaders and other responsible  managers accountable for knowing the progress being made in achieving  goals and, if progress is insufficient, understanding why and having a plan  for improvement including improvements in the quality of the data to help  ensure they are sufficient for decision making. For example, PIOs are  responsible for, among other things, assisting the agency head and COO  in developing and using performance measures specifically for assessing  individual performance in the agency. QPRs offer an opportunity for  organizational performance to be assessed and responsible officials to be  held accountable for addressing problems and identifying strategies for  improvement.
As agencies implement the accountability provisions of GPRAMA, they  will need to ensure managers have decision-making authority  commensurate with the responsibility to identify and address performance  problems as they arise. Since our 1997 government-wide survey of  federal managers, SES managers have reported improvements in  accountability for agency goals and results and the decision-making  authority to help achieve agency goals. However, there has been a gap  between SES managers’ perceptions of their accountability for program  performance as opposed to their decision-making authority since our  initial survey in 1997. In 2013, 80 percent of SES managers reported that  they are held accountable for the results of the programs for which they  are responsible to a great or very great extent, while 61 percent reported  that they have the decision-making authority they need to help the agency  achieve its strategic goals, a 19 percentage point difference. See figure  12.
Using performance information in employee performance management  helps individuals track their performance and progress toward achieving  organizational goals and can help emphasize the importance of individual  contributions to organizational success. However, the percentage of  federal managers reporting use of performance information in employee  performance management to a great or very great extent has stagnated  with no statistically significant change in reported use from 1997 to  2013. See figure 13.
A fundamental element in an organization’s efforts to manage for results  is its ability to set meaningful goals for performance and to measure  progress toward those goals. In our 1996 Executive Guide, we  underscored the importance of taking a balanced approach to setting  goals and measuring performance. If a balance across an organization’s  various priorities does not exist, the measures in place can  overemphasize some goals and create skewed incentives. This need for  agencies to have a balanced set of performance measures was  reinforced in GPRAMA, which calls for agencies to develop a variety of  measures, such as output, outcome, customer service, and efficiency,  across program areas.
As we have previously reported, based on our government-wide federal  managers surveys, federal managers reported a statistically significant  increase in the presence of different types of performance measures for  their programs to a great or very great extent following initial  implementation of GPRA. Despite this early progress in establishing a  variety of performance measures, since our 2003 federal managers  survey, there generally has been no statistically significant increase in the  reported presence of these measures to a great or very great extent.  More recently, as illustrated in figure 14, the only statistically significant  increase between 2007 and 2013 is in the percentage of managers  reporting the presence of quality measures.
We have further found over the years and through our more recent work  that there has been uneven development of outcome-oriented  performance measures across federal programs, even though agencies  have been responsible for measuring program outcomes, among other  things, since the passage of GPRA in 1993. As demonstrated in the  textbox, outcome-oriented performance measures help agencies  determine if the program is achieving its intended purpose. Additionally,  these performance measures are essential for assessing the vast number  of results of federal efforts that span multiple agencies and organizations.
GAO Has Reported on Agency Difficulties in Developing and Using Outcome  Measures In May 2006, we recommended that USDA and DHS adopt meaningful performance  measures for assessing the effectiveness of the Agriculture Quarantine Inspection  (AQI) program at intercepting foreign pests and disease on agricultural materials  entering the country by all pathways and posing a risk to U.S. agriculture.
We reported in March 2013 that the Federal Emergency Management Agency has  not yet established clear, objective, and quantifiable capability requirements and  performance measures to identify capability gaps in a national preparedness  assessment, as recommended in our March 2011 report.
We reported in April 2013 that the Federal Communications Commission, DHS, DOD,  and Department of Commerce had taken a variety of actions to support the security of  the nation’s communications networks, including ones related to developing cyber  policy and standards, securing Internet infrastructure, sharing information, supporting  national security and emergency preparedness, and promoting sector protection  efforts.
GAO, Homeland Security: Management and Coordination Problems Increase the Vulnerability of  U.S. Agriculture to Foreign Pests and Disease, GAO-06-644 (Washington, D.C.: May 19, 2006).  GAO, Homeland Security: Agriculture Inspection Program Has Made Some Improvements, but  Management Challenges Persist, GAO-12-885 (Washington, D.C.: Sept. 27, 2012).  GAO-11-318SP.  GAO, Communications Networks: Outcome-Based Measures Would Assist DHS in Assessing  Effectiveness of Cybersecurity Efforts, GAO-13-275 (Washington, D.C.: Apr. 3, 2013).
Our work over the last 20 years has identified difficulties agencies face in  measuring performance across various program types, such as  regulations and grants. Some commonly reported difficulties that cut  across the various program types include: accounting for factors that are both outside of an agency’s control and  impact the results of a program; developing appropriate performance measures, especially for  programs without a clearly defined purpose or that require a long time  period to achieve intended results; and obtaining complete, timely, and accurate performance information of  the program.
Illustrative examples from our recent work that show how agencies have  experienced difficulties in measuring program performance are provided  in table 2. In our 2013 annual report on fragmentation, overlap, and  duplication, we identified the need for improving the measurement of  performance and results—including program evaluation—as a theme that  cuts across our suggested actions to address fragmentation, overlap, and  duplication in federal agencies.
While some agencies have faced difficulties in measuring program  performance, some progress has been made in developing performance  measures and using the resulting performance information to measure  performance in the applicable program area. For example:
HUD has made progress in measuring grant program performance.
As we reported in November 2011, HUD measured progress toward  some green building goals by collecting energy consumption data for  participating properties receiving grants or loans under its Green  Retrofit Program for Multifamily Housing before and after the  properties are retrofitted and planned to use this data to calculate  savings and evaluate effectiveness.  In January 2011, we reported that the Federal Railroad Administration  (FRA) has created a set of performance goals and measures that  address important dimensions of program performance related to its  regulatory safety activities. In its proposed fiscal year 2011 budget,  FRA included specific safety goals to reduce the rate of train  accidents caused by various factors, including human errors and track  defects. These goals were quantitative, with a targeted accident rate  per every million train miles. Collecting such accident data equips  FRA with a clear way to measure whether or not those safety goals  are met. FRA’s budget request has also linked FRA’s performance  goals and measures with DOT’s strategic goals.
Moving forward, we will continue to examine the availability and use of  performance measures across a variety of program types and update our  work in this area. Given that we have found that agencies across the  federal government have experienced similar difficulties in measuring the  performance of different program types and have not made consistent  progress in addressing them, a comprehensive examination of these  difficulties is needed. The PIC could help facilitate this examination. As  discussed earlier, GPRAMA requires the PIC, in part, to resolve  crosscutting performance issues and facilitate the exchange of practices  that have led to performance improvements within specific programs or  agencies or across agencies. Although measuring the performance of  different program types is a significant and long-standing challenge, the  PIC has not yet addressed this issue in a systematic way, such as  through a working group to identify common difficulties in developing and  using performance measures to assess program performance and share  best practices from instances in which agencies have overcome these  difficulties. Without a comprehensive examination, it will be difficult for the  PIC and agencies to fully understand these measurement issues and  develop a crosscutting approach to help address them, which will likely  result in agencies experiencing difficulties in measuring program  performance in the future.
section_title: Communication of Performance Information Could Better Meet Users’ Needs
subsections: section_title: Federal Managers Reported that Performance Information Is Not Always Available or Easily Accessible to Federal Managers or the Public
paragraphs: According to our 2013 survey of federal managers, 34 percent reported  that performance information is easily accessible to agency employees to  a great or very great extent, while 17 percent reported that their agency’s  performance information is easily accessible to the public to a great or  very great extent. Survey data also indicate that agencies are not  communicating to their employees about contributions to CAP goals or  their progress toward achieving APGs. In fact, of the 58 percent of federal  managers who indicated they were familiar with CAP goals, 22 percent  reported that their agency has communicated to its employees on those  goals to a great or very great extent. Of the 82 percent of federal  managers who indicated familiarity with APGs, 40 percent reported that  their agency has communicated on progress toward achieving them to  great or very great extent.
We recently reported that Performance.gov, as the central repository for  federal government performance information, can assist in oversight and  lead to a greater focus within government on the activities and efforts  necessary to improve performance. OMB’s stated goals for  Performance.gov include, among others, providing both a public view into  government performance to support transparency as well as providing  executive branch management capabilities to enhance senior leadership  decision making. According to OMB staff, OMB will maintain responsibility  for the website, but going forward, the plans are that the effort will be  driven more by the General Services Administration (GSA) and the PIC,  with GSA continuing to provide technological support. For future  development of Performance.gov, OMB, the PIC, and GSA are working  with federal agencies to develop the Performance Management Line of  Business that, according to OMB staff, will standardize the collection and  reporting of performance information by agencies.
Performance.gov has the potential to increase the accessibility of  performance information for users both inside and outside the federal  government. An analysis of statements from OMB and GSA staff, agency  officials, and feedback we obtained from potential users, however,  indicates that there are varying expectations regarding the primary uses  of Performance.gov. For example, OMB and GSA staff emphasized that  they have viewed Performance.gov as a tool for agencies to support  cross-agency coordination and efforts to achieve agency goals.  Consistent with this, OMB staff said that Performance.gov has been used  to facilitate conversations between OMB examiners and agency  managers about progress on APGs. While most officials we interviewed  said that OMB had collected feedback from the agencies in the  development of Performance.gov, officials from most of these agencies  also said that Performance.gov is not being used as a resource by  agency leadership or other staff, as they have information sources  tailored to meet their needs, and Performance.gov does not contain  critical indicators or the ability to display some visualizations used for  internal agency performance reviews. In addition, a performance  management practitioner and other potential users of the website noted  that the detailed, technical nature of Performance.gov seemed primarily  oriented toward a government rather than a public audience.
According to OMB staff, the specific legal requirements of GPRAMA have  been the primary framework used to guide efforts to develop  Performance.gov thus far. They noted that they have been focused on  working to comply with these requirements by providing information on  CAP goals and APGs, and by establishing a phased development plan for  the integration of additional information from agency strategic plans,  performance plans, and performance reports. OMB and GSA staff  members have said, however, that the leading practices for developing  federal websites will be helpful in guiding the future development of  Performance.gov. OMB and GSA staff have also noted that as the  phased development of Performance.gov unfolds, they expect to use  broader outreach to, and usability testing with, a wider audience,  including members of the public, to make Performance.gov more “public- facing” and “citizen-centric.”
In accordance with this transition, we recommended in June 2013 that  OMB work with GSA and the PIC to clarify the specific ways that intended  audiences could use the information on Performance.gov. HowTo.gov, a  leading source of best practices and guidance on the development of  federal government websites, recommends identifying the purposes of a  website, and the ways in which specific audiences could use a website to  accomplish various tasks, and then structuring information and providing  tools to help visitors quickly complete these tasks. With greater clarity  about the intended uses of Performance.gov, OMB and GSA should have  sufficient direction to design Performance.gov to make it a relevant and  accessible source of information for a variety of potential users including  those specified under GPRAMA—members and committees of Congress  and the public.
In the same report, we also recommended that OMB should work with  GSA and the PIC to systematically collect information on the needs of  intended audiences and collect recommended performance metrics that  help identify improvements to the website. For example, HowTo.gov  practices recommend that a website use consistent navigation. Although  users we interviewed had mixed opinions on the organization and  navigation of Performance.gov, simplifying the website’s navigation,  adding an effective internal search engine, and providing an appropriate  level of detail and information for intended audiences could increase the  overall usability of Performance.gov. Outreach and testing on the ease of  navigation and searching would help OMB systematically collect  information on the needs of various audiences and how these could be  addressed through Performance.gov. With performance goals and  measures for the website, it would also be possible for the developers of  Performance.gov to identify the gap between current capabilities and  what is needed to fulfill stated goals and to identify and set priorities for  improvements. OMB staff agreed with these recommendations.
section_title: Agency Performance Information Is Not Always Useful for Congressional Decision Making
paragraphs: Congressional support has played a critical role in sustaining interest in  management improvement initiatives over time. As we have previously  reported, Congress has served as an institutional champion for many  government-wide management reform initiatives over the years, such as  the CFO Act and GPRA in the 1990s and more recently GPRAMA.  Further, Congress has often played an important role in performance  improvement and management reforms at individual agencies. Congress  has also provided a consistent focus on oversight and has reinforced  important policies.
As we have previously reported, having pertinent and reliable  performance information available is necessary for Congress to  adequately assess agencies’ progress in making performance and  management improvements and ensure accountability for results.  However, our work has found that the performance information that  agencies provided to Congress was not always useful for congressional  decision making because the information was not clear, directly relevant,  or sufficiently detailed. As stated earlier, in order for performance  information to be useful, it should meet the needs of different users— including Congress—in terms of completeness, accuracy, consistency,  timeliness, validity, and ease of use.
GPRA required agencies to consult with Congress and obtain the views  of interested stakeholders as a part of developing their strategic plans.  However, according to the Senate committee report that accompanied the  bill that ultimately became GPRAMA, agencies did not adequately  consider the input of Congress in developing strategic plans, often  because the agencies waited until strategic plans were substantially  drafted and reviewed within the executive branch before consulting with  Congress. In doing so, agencies limited the opportunities for Congress to  provide input on their strategic plans and related goals, as well as the  performance information that would be most useful for congressional  oversight.
To help ensure agency performance information is useful for  congressional decision making, GPRAMA strengthens the consultation  requirement. The act requires agencies to consult at least once every two  years with relevant appropriations, authorization and oversight  committees, obtaining majority and minority views, when developing or  updating strategic plans—which include APGs. Subsequently, agencies  are to describe how congressional input was incorporated into those  plans and goals. Similarly, OMB is required to consult with relevant  committees with broad jurisdiction at least once every two years when  developing or updating CAP goals, and describe how that input was  incorporated into those goals. At the request of Congress, in June 2012,  we developed a guide to assist Members of Congress and their staffs in  ensuring the consultations required under GPRAMA are useful to the  Congress. The guide outlines general approaches for successful  consultations, including creating shared expectations and engaging the  right people in the process at the right time. The guide also provides key  questions that Members and congressional staff can ask as part of the  consultation process to ensure that agency performance information  reflects congressional priorities.
However, it is unclear if agencies incorporated congressional input on  their updated strategic plans and APGs published in 2012, and therefore  if this information will be useful for congressional decision making. In our  recent review of APGs, we found that agencies reported engaging  Congress during the development of their strategic plans and goals to  varying degrees, and only 1 of the 24 agencies we reviewed explained  how congressional input was incorporated into its APGs, as required by  GPRAMA. We recommended in April 2013 that OMB ensure that  agencies adhere to OMB’s guidance for website updates by providing a  description of how input from congressional consultations was  incorporated into each goal. OMB staff concurred with our  recommendation.
In addition, our recent work indicated that the performance information  provided on Performance.gov also may not be meeting congressional  needs. We found that outreach from OMB to congressional staff was  limited, as were opportunities for staff to provide input on the  development of Performance.gov. According to OMB staff, they met  several times with staff from the Senate Homeland Security and  Governmental Affairs Committee, House Oversight and Government  Reform Committee, and the Senate Budget Committee to discuss the  development of Performance.gov, and used this outreach to identify  several specific website modifications. Of the three congressional staff  that we spoke to that said they had received briefings on the development  of Performance.gov, however, only one told us she had been consulted  on website input. In addition, since 2010, OMB staff has not held  meetings on the development of Performance.gov with staff from other  committees in the House or Senate that might use the website to inform  their oversight of federal agencies. As previously mentioned, we also  found that OMB has not articulated how various intended audiences,  including Congress, can use the site to accomplish specific tasks, such as  supporting coordination and decision making to advance shared goals.
At the request of the Congress, in December 2011 and June 2012, we  highlighted several instances in which Congress has used agency  performance information in various oversight and legislative activities,  including (1) identifying issues that the federal government should  address; (2) measuring the federal government’s progress toward  addressing those issues; and (3) identifying better strategies to address  the issues when necessary. For example, to help promote the use of e- filing of tax returns with the IRS, Congress used performance information  to set clear expectations for agency performance, support oversight  activities, and inform the development of additional legislation to help IRS  achieve its goals. For further information, see the textbox.
Congressional Use of Performance Information to Promote E-filing of Tax Returns  Congress sought to promote the use of e-filing, which allows taxpayers to receive refunds  faster, is less prone to errors, and provides IRS significant cost savings. Congress took  the following actions to increase the use of e-filing:   Setting Expectations: As part of the Internal Revenue Service Restructuring and  Reform Act of 1998, Congress established a performance goal of having 80 percent of  individual tax returns e-filed by 2007.   Oversight: Congress monitored IRS’s progress in meeting the established goal for e- filings; held 22 hearings related to IRS filing seasons and e-filings; and requested annual  GAO reports to Congress on filing season performance, including e-filing.   Additional Legislation: Congress saw the need for further actions to help IRS achieve  the goal, and subsequently passed legislation to require tax return preparers who file  more than 10 returns per year to do so electronically.  Although IRS did not meet the 80 percent e-filing target by 2007 (58 percent were e-filed  that year), increased use of e-filing has substantially reduced IRS’s cost to process  returns. IRS subsequently met this goal for individual tax returns as of the 2012 tax filing  season, with 82 percent of individual returns e-filed.
IRS has yet to reach the 80 percent e-file goal for some types of returns other than individual income  tax returns. See GAO, 2012 Tax Filing: IRS Faces Challenges Providing Service to Taxpayers and  Could Collect Balances Due More Effectively, GAO-13-156 (Washington, D.C.: Dec. 18, 2012).
section_title: Conclusions
paragraphs: Moving forward, the federal government will need to make tough choices  in setting priorities as well as reforming programs and management  practices to address the pressing and complex economic, social, security,  sustainability, and other issues the nation confronts. GPRAMA provides a  number of tools that could help address these challenges. Since  enactment in 2011, the executive branch has taken a number of important  steps to implement key provisions of the act, by developing interim CAP  goals and APGs, conducting quarterly reviews, assigning key  performance management roles and responsibilities, and communicating  results more frequently and transparently through Performance.gov.
However, the executive branch needs to do more to fully implement and  leverage the act’s provisions to address these challenges. Our recent  work reviewing federal performance issues and implementation of the act  has pointed to several areas where improvements are needed and,  accordingly, we recommended a number of actions. In addition, examples  from our past work along with the most recent results from our survey of  federal managers show that the executive branch has made little  progress addressing long-standing governance challenges related to  improving coordination and collaboration to address crosscutting issues,  using performance information to drive decision making, measuring the  performance of certain types of federal programs, and engaging  Congress in a meaningful way in agency performance management  efforts to ensure the resulting information is useful for congressional  decision making.
Of particular concern, OMB has yet to develop a framework for reviewing  the performance of tax expenditures, which represented approximately $1  trillion in forgone revenue in fiscal year 2012. In some areas, forgone  revenue due to tax expenditures is nearly equal to or greater than  spending for federal outlay programs. Since 1994 we have recommended  OMB take this action, and the act puts into place explicit requirements for  the CAP goals that OMB identify related tax expenditures and measure  their contributions to broader federal outcomes. While early  implementation of CAP goals showed some promise, with tax  expenditures being identified as contributing to 5 of the 14 goals, many of  those tax expenditures were subsequently removed. For example, our  work shows that eight tax expenditures, representing about $2.4 billion in  forgone revenue, should be listed as contributing to the energy efficiency  CAP goal. The few tax expenditures that continue to be listed as  contributors to a CAP goal only represent about $2.7 billion in forgone  revenue—approximately 0.3 percent of the total estimate of forgone  revenue from tax expenditures. While OMB staff told us the removal of  these tax expenditures was an oversight and that they will be added as  contributors in the near future, it raises concerns as to whether OMB  previously ensured all relevant tax expenditures were identified as  contributors to the 14 CAP goals when they were published in February  2012. Tax expenditures represent a substantial federal commitment to a  wide range of mission areas, but do not receive the same scrutiny as  spending programs. Therefore, it is possible that additional tax  expenditures should have been identified and included as contributors to  one or more of the other 9 CAP goals. Moreover, for the 2 CAP goals  where tax expenditures were mistakenly removed, it is unclear if OMB  and the goal leaders assessed the contributions of those tax expenditures  toward the CAP goal efforts, since they were not listed in the December  2012 and March 2013 updates. Without information about which tax  expenditures support these goals and measures of their performance,  Congress and other decision makers will not have the needed information  to assess overall federal contributions towards desired results and the  costs and relative effectiveness associated with those contributions.
OMB took another promising action in 2012 by directing agencies to  identify tax expenditures among the various federal programs and  activities that contribute to their APGs—above and beyond what the act  requires for all performance goals, which include APGs. However, the  103 APGs developed for 2012 to 2013 at 24 agencies represent only a  small subset of all performance goals across the government. In addition,  our review of the APGs for 2012 to 2013 found that only one agency, for  one of its APGs, identified two relevant tax expenditures. OMB and  agencies are missing important opportunities to more broadly identify how  tax expenditures contribute to each agency’s overall performance.
In addition to measuring the contributions of tax expenditures to their  goals, our work has found that agencies have experienced common  issues in measuring the performance of various other types of programs  and have not made consistent progress in addressing them in the last 20  years. As such, a comprehensive and concerted effort to address these  long-standing difficulties needs to be taken. With responsibilities to  resolve crosscutting performance issues and facilitate the exchange of  proven practices, the PIC should lead such an assessment. The PIC has  not yet addressed this issue in a systematic way, and without a  comprehensive examination, it will be difficult for the PIC and agencies to  fully understand these measurement issues and develop a crosscutting  strategy to address them. That would likely result in agencies continuing  to experience difficulties in measuring program performance in the future.  The PIC’s upcoming strategic planning effort provides a venue for  developing an approach for tackling this issue by putting in place the  necessary plans and accountability. The PIC’s strategy should detail  specific actors and actions to be made within set time frames to ensure  that these persistent measurement challenges are adequately addressed.
section_title: Recommendations for Executive Action
paragraphs: To improve implementation of GPRAMA and help address pressing  governance issues, we make the following four recommendations.
To help ensure that the contributions made by tax expenditures toward  the achievement of agency goals and broader federal outcomes are  properly recognized, we recommend that the Director of OMB take the  following three actions:
Revise relevant OMB guidance to direct agencies to identify relevant  tax expenditures among the list of federal contributors for each  appropriate agency goal.
Review whether all relevant tax expenditures that contribute to a CAP  goal have been identified, and as necessary, include any additional  tax expenditures in the list of federal contributors for each goal.
Assess the contributions relevant tax expenditures are making toward  the achievement of each CAP goal.
Given the common, long-standing difficulties agencies continue to face in  measuring the performance of various types of federal programs and  activities—contracts, direct services, grants, regulations, research and  development, and tax expenditures—we also recommend the Director of  OMB work with the PIC to develop a detailed approach to examine these  difficulties across agencies, including identifying and sharing any  promising practices from agencies that have overcome difficulties in  measuring the performance of these program types. This approach  should include goals, planned actions, and deliverables along with  specific time frames for their completion, as well as the identification of  the parties responsible for each action and deliverable.
section_title: Agency Comments
paragraphs: We provided a draft of this report for review and comment to the Director  of OMB. Via e-mail, staff from OMB’s Office of Performance and  Personnel Management agreed with the recommendations in this report.  The staff also provided technical comments, which we incorporated as  appropriate.
We are sending copies of this report to the Director of OMB as well as  interested congressional committees and other interested parties. This  report will also be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-6806, or mihmj@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of our report. Key contributors to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The GPRA Modernization Act of 2010 (GPRAMA) lays out a schedule for  gradual implementation of its provisions during a period of interim  implementation—from its enactment in January 2011 to February 2014  when a new planning and reporting cycle begins. GPRAMA also includes  provisions requiring us to review implementation of the act at several  critical junctures and provide recommendations for improvements to its  implementation. This report is the final in a series responding to the  mandate to assess initial implementation of the act by June 2013, and  pulls together findings from our recent work related to the act, the results  of our periodic survey of federal managers, and our related recent work  on federal performance and coordination issues.
Our specific objectives for this report were to assess the executive  branch’s (1) progress in implementing the act and (2) effectiveness in  using tools provided by the act to address challenges the federal  government faces. To address both objectives, we reviewed GPRAMA,  related congressional documents and Office of Management and Budget  (OMB) guidance, and our past and recent work related to managing for  results and the act. We also interviewed OMB staff.
In addition, to further address the second objective, we administered a  web-based questionnaire on organizational performance and  management issues to a stratified random sample of 4,391 persons from  a population of approximately 148,300 mid-level and upper-level civilian  managers and supervisors working in the 24 executive branch agencies  covered by the Chief Financial Officers (CFO) Act of 1990, as amended.  The survey results provided information about the extent to which key  performance management practices are in place to help address  challenges. The sample was drawn from the Office of Personnel  Management’s (OPM) Central Personnel Data File (CPDF) as of March  2012, using file designators indicating performance of managerial and  supervisory functions. In reporting the questionnaire data, when we use  the term “government-wide” and the phrases “across the government” or  “overall” we are referring to these 24 CFO Act executive branch agencies,  and when we use the terms “federal managers” and “managers” we are  referring to both managers and supervisors.
The questionnaire was designed to obtain the observations and  perceptions of respondents on various aspects of results-oriented  management topics such as the presence and use of performance  measures, hindrances to measuring performance and using performance  information, agency climate, and program evaluation use. In addition, to  address implementation of GPRAMA, the questionnaire included a  section requesting respondents’ views on various provisions of GPRAMA,  such as cross-agency priority goals, agency priority goals, and quarterly  performance reviews. For the agency priority goal questions, we directed  the federal managers from the Nuclear Regulatory Commission to not  answer these questions since OMB did not require the agency to develop  agency priority goals for 2012 to 2013.
This survey is comparable to surveys we have conducted four times  previously at the 24 CFO Act agencies—1997, 2000, 2003, and 2007.  The 1997 survey was conducted as part of the work we did in response to  a GPRA requirement that we report on implementation of the act. The  2000, 2003, and 2007 surveys were designed to update the results from  each of the previous surveys. The 2007 survey also included a section  requesting the respondent’s view on OMB’s Program Assessment Rating  Tool and the priority that should be placed on various potential  improvements to it. The 2000 and 2007 surveys, unlike the other two  surveys, were designed to support analysis of the data at the department  and agency level as well as government-wide. For this report, we focus  on comparing the 2013 survey results with those from the 1997 baseline  survey; and with the results of the 2007 survey, which is the most recent  survey conducted before GPRAMA was enacted in 2011. We noted the  results from the other two surveys—2000 and 2003—when statistically  significant trends compared to 2013 occurred.
Similar to the four previous surveys, the sample was stratified by agency  and by whether the manager or supervisor was a member of the Senior  Executive Service (SES) or non-SES. The management levels covered  general schedule (GS) or equivalent schedules at levels comparable to  GS-13 through GS-15 and career SES or equivalent. Similar to our 2000,  2003, and 2007 surveys, we also incorporated managers or supervisors  in other pay plans at levels generally equivalent to the GS-13 through  career SES levels into the population and the selected sample to ensure  at least a 90 percent coverage of all mid- to upper-level managers and  supervisors at the departments and agencies we surveyed.
Most of the items on the questionnaire were closed-ended, meaning that  depending on the particular item, respondents could choose one or more  response categories or rate the strength of their perception on a 5-point  extent scale ranging from “to no extent” at the low end of the scale to “to a  very great extent” at the high end. On most items, respondents also had  an option of choosing the response category “no basis to judge/not  applicable.” A few items had yes, no, or do not know options for  respondents.
Many of the items on the questionnaire were asked in our earlier surveys;  the sections of the questionnaire asking about GPRAMA, program  evaluations, and availability of performance information are new. For  these new questions, we conducted pretests with federal managers in  several of the 24 CFO Act agencies. For the 2013 survey, based on  feedback we obtained from our pretests with managers, we moved the  placement of question 8 in the survey to accommodate the insertion of a  new question. In previous surveys, only those respondents who  answered yes to question 5—that they had performance measures  available for their programs—were asked to answer question 8—a series  of items about the extent to which they used information obtained from  performance measurement when participating in certain activities.  Respondents answering “no” or “do not know” to question 5 could skip  past the question 8 items. For the 2013 survey, all respondents were  asked to answer question 8 given the new question added. To maintain  the consistency and comparability with how we have previously analyzed  and reported question 8 results, we applied the skip pattern used in prior  surveys to question 8 by removing those individuals who did not answer  yes to question 5 (and in the past would have been directed to skip out of  answering the question). However, in the e-supplement we report the  results as the federal managers answered the questionnaire, regardless  of how they had answered question 5.
To administer the survey, an e-mail was sent to managers in the sample  that notified them of the survey’s availability on the GAO website and  included instructions on how to access and complete the survey. With the  exception of the managers at the Department of Justice (DOJ), which is  discussed below, managers in the sample who did not respond to the  initial notice were sent up to four subsequent e-mail reminders and follow- up phone calls asking them to participate in the survey.
In our prior surveys, we worked with OPM to obtain the names of the  managers and supervisors in our sample as selected through the CPDF.
However, since our last survey in 2007, some agencies had requested  from OPM that the names of individuals within selected subcomponents  be withheld from the CPDF. We worked with officials at these agencies to  attempt to gain access to these individuals to maintain continuity of the  population of managers surveyed from previous years. Due to DOJ’s  national security concerns about providing identifying information (e.g.,  names, e-mail addresses, phone numbers) of federal agents to us, we  administered the current survey to all DOJ managers in our sample  through a DOJ official. To identify the sample of managers whose names  were withheld from the CPDF, we provided DOJ with the last four digits of  Social Security numbers, the subcomponent, duty location, and pay grade  information. To ensure that DOJ managers received the same survey  administration process as the rest of the managers in our sample to the  extent possible, we provided DOJ with copies of the notification,  activation (including the web link to our survey), and follow-up e-mails that  managers at other agencies received from us. DOJ administered the  survey to its managers and conducted follow-up with the nonrespondents.  We administered the survey to all 24 agencies from November 2012  through February 2013.
To help determine the reliability and accuracy of the CPDF data elements  used to draw our sample of federal managers, we checked the data for  reasonableness and the presence of any obvious or potential errors in  accuracy and completeness. For example, we identified cases where the  managers’ names were withheld and contacted OPM to determine the  reason and extent of this issue. We also checked the names of the  managers in our selected sample provided from OPM with the applicable  agency contacts to verify these managers were still employed with the  agency in the role. We noted discrepancies when they occurred and  excluded them from our population of interest, as applicable. We also  reviewed our past analyses of the reliability of the CPDF data. On the  basis of these procedures, we believe the data we used from the CPDF  are sufficiently reliable for the purpose of this report.
Of the 4,391 managers selected for this survey, we found that 266 of the  sampled managers had retired, separated, died, or otherwise left the  agency or had some other reason that excluded them from the population  of interest. We received usable questionnaires from 2,762 sample  respondents, or about 69 percent of the remaining eligible sample. In  addition, there were 29 persons that we were unable to locate and  therefore unable to request that they participate in the survey. The  response rate across the 24 agencies ranged from 57 percent to 88  percent.
The overall survey results are generalizable to the population of  managers as described above at each of the 24 agencies and  government-wide. The responses of each eligible sample member who  provided a usable questionnaire were weighted in the analyses to  account statistically for all members of the population. All results are  subject to some uncertainty or sampling error as well as nonsampling  error. Because we followed a probability procedure based on random  selections, our sample is only one of a large number of samples that we  might have drawn. Since each sample could have provided different  estimates, we express our confidence in the precision of our particular  sample’s results as a 95 percent confidence interval. This is the interval  that would contain the actual population value for 95 percent of the  samples we could have drawn.
The percentage estimates presented in this report based on our sample  for the 2013 survey have 95 percent confidence intervals within plus or  minus 5 percentage points of the estimate itself, unless otherwise noted.  An online e-supplement shows the questions asked on the survey along  with the percentage estimates and associated 95 percent confidence  intervals for each item for each agency and government-wide.
Because a complex survey design was used in the current survey as well  as the four previous surveys, and different types of statistical analyses  are being done, the magnitude of sampling error will vary across the  particular surveys, groups, or items being compared due to differences in  the underlying sample sizes, usable sample respondents, and associated  variances of estimates. For example, the 2000 and 2007 surveys were  designed to produce agency-level estimates and had effective sample  sizes of 2,510 and 2,943, respectively. However, the 1997 and 2003  surveys were designed to obtain government-wide estimates only, and  their sample sizes were 905 and 503, respectively. Consequently, in  some instances, a difference of a certain magnitude may be statistically  significant. In other instances, depending on the nature of the comparison  being made, a difference of equal or even greater magnitude may not  achieve statistical significance. We note in this report when we are 95  percent confident that the difference is statistically significant. Also, as  part of any interpretation of observed shifts in individual agency  responses between the 2013 and the 2000 surveys, it should be kept in  mind that components of some agencies and all of the Federal  Emergency Management Agency became part of the Department of  Homeland Security.
In addition to sampling errors, the practical difficulties of conducting any  survey may also introduce other types of errors, commonly referred to as  nonsampling errors. For example, difficulties in how a particular question  is interpreted, in the sources of information available to respondents, or in  how the data were entered into a database or were analyzed can  introduce unwanted variability into the survey results. With this survey, we  took a number of steps to minimize these nonsampling errors. For  example, our staff with subject matter expertise designed the  questionnaire in collaboration with our survey specialists. As noted  earlier, the new questions added to the survey were pretested to ensure  they were relevant and clearly stated. When the data were analyzed, a  second independent GAO analyst independently verified the analysis  programs to ensure the accuracy of the code and the appropriateness of  the methods used for the computer-generated analysis. Since this was a  web-based survey, respondents entered their answers directly into the  electronic questionnaire, thereby eliminating the need to have the data  keyed into a database, thus avoiding a source of data entry error.
We conducted this performance audit from August 2012 to June 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Status of Key Recommendations Related to GPRAMA
paragraphs: Status  OMB staff agreed with our  recommendations. clarify the ways that intended audiences could use  the information on the Performance.gov website to  accomplish specific tasks and specify the design  changes that would be required to facilitate that use;  seek to more systematically collect information on  the needs of a broader audience, including through  the use of customer satisfaction surveys and other  approaches recommended by HowTo.gov; and  seek to ensure that all performance, search, and  customer satisfaction metrics, consistent with  leading practices outlined in HowTo.gov, are tracked  for the website, and, where appropriate, create goals  for those metrics to help identify and prioritize  potential improvements to Performance.gov.
OMB staff agreed with our  recommendations. provide a definition of what constitutes “data of  significant value;”  direct agencies to develop and publish on  Performance.gov interim quarterly performance  targets for their agency priority goal performance  measures when the above definition applies;  direct agencies to provide and publish on  Performance.gov completion dates, both in the near- term and longer-term for their milestones; and  direct agencies to describe in their performance  plans how the agency’s performance goals— including priority goals—contribute to any of the  cross-agency priority goals.
When such revisions are made, the Director of OMB  should work with the PIC to test and implement these  provisions.
subsections: section_title: Title and GAO product number
paragraphs: Status  OMB staff agreed with our  recommendations. complete information about the organizations,  program activities, regulations, policies, tax  expenditures, and other activities—both within and  external to the agency—that contribute to each goal;  and  a description of how input from congressional  consultations was incorporated into each goal.
To improve performance management staff capacity to  support performance management in federal agencies,  the Director of OPM should, in coordination with the PIC  and the Chief Learning Officer Council, work with  agencies to: identify competency areas needing improvement  within agencies;  identify agency training that focuses on needed  performance management competencies; and  share information about available agency training on  competency areas needing improvement.
OPM agreed with our  recommendations, and  explained that it will work with  agencies, and in particular with  PIOs, to assess the  competencies of the  performance management  workforce. OPM also stated that  it will support the use of the  PIC’s performance learning  website to facilitate the  identification and sharing of  training related to competencies  in need of improvement.
OMB staff agreed with our  recommendations. conduct formal feedback on the performance of the  PIC from member agencies, on an ongoing basis;  and  update its strategic plan and review the PIC’s goals,  measures, and strategies for achieving performance,  and revise them if appropriate.
To better leverage agency quarterly performance reviews  as a mechanism to manage performance toward agency  priority and other agency-level performance goals, the  Director of OMB should—working with the PIC and other  relevant groups—identify and share promising practices  to help agencies extend their quarterly performance  reviews to include, as relevant, representatives from  outside organizations that contribute to achieving their  agency performance goals.
OMB staff agreed with our  recommendation.
Summary of related recommendations  The Director of OMB, in considering additional programs  with the potential to contribute to the crosscutting goals,  should review the additional departments, agencies, and  programs that we have identified, and consider including  them in the federal government performance plan, as  appropriate.
Status  OMB staff agreed with our  recommendation. In December  2012 and March 2013, OMB  updated information on  Performance.gov on the CAP  goals. OMB included some of  the agencies and programs we  identified for select goals, but in  other instances eliminated key  contributors that were previously  listed.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the above contact, Elizabeth Curda (Assistant Director) and  Benjamin T. Licht supervised this review and the development of the  resulting report. Tom Beall, Peter Beck, Mallory Barg Bulman, Virginia  Chanley, Laura Miller Craig, Sara Daleski, Karin Fangman, Stuart  Kaufman, Don Kiggins, Judith Kordahl, Jill Lacey, Janice Latimer, Adam  Miles, Kathleen Padulchick, Mark Ramage, Daniel Ramsey, Marylynn  Sergent, Megan Taylor, Sarah Veale, Kate Hudson Walker, and Dan  Webb made significant contributions to this report. Pawnee Davis,  Shannon Finnegan, Quindi Franco, Ellen Grady, Robert Gebhart, Tom  James, Donna Miller, Michael O’Neill, Robert Robinson, and Stephanie  Shipman also made key contributions.