Results-Oriented Government: GPRA Has Established a Solid Foundation for Achieving Greater Results
Published: 2004-03-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Now that the Government Performance and Results Act (GPRA) has been in effect for 10 years, GAO was asked to address (1) the effect of GPRA in creating a governmentwide focus on results and the government's ability to deliver results to the American public, (2) the challenges agencies face in measuring performance and using performance information in management decisions, and (3) how the federal government can continue to shift toward a more results-oriented focus.
section_title: What GAO Found
paragraphs: GPRA's requirements have established a solid foundation of results-oriented performance planning, measurement, and reporting in the federal government. Federal managers surveyed by GAO reported having significantly more of the types of performance measures called for by GPRA. GPRA has also begun to facilitate the linking of resources to results, although much remains to be done in this area to increase the use of performance information to make decisions about resources. We also found agency strategic and annual performance plans and reports we reviewed have improved over initial efforts. Although a foundation has been established, numerous significant challenges to GPRA implementation still exist. Inconsistent top leadership commitment to achieving results within agencies and Office of Management and Budget (OMB) can hinder the development of results-oriented cultures in agencies. Furthermore, in certain areas, federal managers continue to have difficulty setting outcome-oriented goals, collecting useful data on results, and linking institutional, program, unit, and individual performance measurement and reward systems. Finally, there is an inadequate focus on addressing issues that cut across federal agencies. OMB, as the focal point for management in the federal government, is responsible for overall leadership and direction in addressing these challenges. OMB has clearly placed greater emphasis on management issues during the past several years. However, it has showed less commitment to GPRA implementation in its guidance to agencies and in using the governmentwide performance plan requirement of GPRA to develop an integrated approach to crosscutting issues. In our view, governmentwide strategic planning could better facilitate the integration of federal activities to achieve national goals.

section_title: Letter
paragraphs: As you requested, we have assessed the effectiveness of the Government  Performance and Results Act (GPRA), in light of its 10-year anniversary in  2003.  Our review focused on GPRA’s accomplishments, challenges to its  continued implementation, and an agenda for achieving a sustainable,  governmentwide focus on results.
Upon issuance, we will send copies to the Director of the Office of  Management and Budget and executive branch agencies (see appendix VII  for a list).  We will also make copies available to others upon request.  In  addition, this report will be available at no charge on the GAO web site at  http://www.gao.gov.
If you have any questions concerning this report, please contact Patricia A.  Dalton at (202) 512-6806 or daltonp@gao.gov.  The major contributors to  this report are listed in appendix XII.
subsections: section_title: Introduction
paragraphs: From defending the homeland against terrorists, to preventing the spread  of infectious diseases, to providing a reliable stream of social security  income to retirees and supporting the transition from welfare to work, the  federal government provides funding and services to the American public  that can affect their lives in critical ways every day.  However, the federal  government is in a period of profound transition and faces an array of  challenges and opportunities to enhance performance, ensure  accountability, and position the nation for the future.  A number of  overarching trends, such as diffuse security threats and homeland security  needs, increasing global interdependency, the shift to knowledge-based  economies, and the looming fiscal challenges facing our nation, drive the  need to reconsider the proper role for the federal government in the 21st  century, how the government should do business (including how it should  be structured), and in some instances, who should do the government’s  business.
Without effective short- and long-term planning, which takes into account  the changing environment and needs of the American public and the  challenges they face and establishes goals to be achieved, federal agencies  risk delivering programs and services that may or may not meet society’s  most critical needs.  At a cost to taxpayers of over $2 trillion annually, the  federal government should be able to demonstrate to the American public  that it can anticipate emerging issues, develop sound strategies and plans  to address them, and be accountable for the results that have been  achieved.
Concerned that the federal government was more focused on program  activities and processes than the results to be achieved, Congress passed  the Government Performance and Results Act of 1993 (GPRA).  The act  required federal agencies to develop strategic plans with long-term  strategic goals, annual goals linked to achieving the long-term goals, and  annual reports on the results achieved.  Now that GPRA has been in effect  for 10 years, you asked us to assess the effectiveness of GPRA in creating a  focus on results in the federal government.  Specifically, this report  discusses (1) the effect of GPRA over the last 10 years in creating a  governmentwide focus on results and the government’s ability to deliver  results to the American public, including an assessment of the changes in  the overall quality of agencies’ strategic plans, annual performance plans,  and annual performance reports; (2) the challenges agencies face in  measuring performance and using performance information in  management decisions; and (3) how the federal government can continue  to shift toward a more results-oriented focus.
subsections: section_title: Impact of Emerging Trends and Fiscal Challenges
paragraphs: With the 21st century challenges we are facing, it is more vital than ever to  maximize the performance of federal agencies in achieving their long-term  goals.  The federal government must address and adapt to major trends in  our country and around the world.  At the same time, our nation faces  serious long-term fiscal challenges.  Increased pressure also comes from  world events: both from the recognition that we cannot consider ourselves  “safe” between two oceans—which has increased demands for spending on  homeland security—and from the United States (U.S.) role in combating  terrorism in an increasingly interdependent world.  To be able to assess  federal agency performance and hold agency managers accountable for  achieving their long-term goals, we need to know what the level of  performance is.  GPRA planning and reporting requirements can provide  this essential information.
Our country’s transition into the 21st century is characterized by a number  of key trends, including  the national and global response to terrorism and other threats to our  personal and national security;  the increasing interdependence of enterprises, economies, markets, civil  societies, and national governments, commonly referred to as  globalization;  the shift to market-oriented, knowledge-based economies;  an aging and more diverse U.S. population;  rapid advances in science and technology and the opportunities and  challenges created by these changes;  challenges and opportunities to maintain and improve the quality of life  for the nation, communities, families, and individuals; and  the changing and increasingly diverse nature of governance structures  and tools.
As the nation and government policymakers grapple with the challenges  presented by these evolving trends, they do so in the context of rapidly  building fiscal pressures.  GAO’s long-range budget simulations show that  this nation faces a large and growing structural deficit due primarily to  known demographic trends and rising health care costs.  The fiscal  pressures created by the retirement of the baby boom generation and rising  health costs threaten to overwhelm the nation’s fiscal future.  As figure 1  shows, by 2040, absent reform or other major tax or spending policy  changes, projected federal revenues will likely be insufficient to pay more  than interest on publicly held debt.  Further, our recent shift from surpluses  to deficits means the nation is moving into the future in a more constrained  fiscal position.
The United States has had a long-range budget deficit problem for a  number of years, even during recent years in which we had significant  annual budget surpluses.  Unfortunately, the days of surpluses are gone,  and our current and projected budget situation has worsened significantly.   The bottom line is that our projected budget deficits are not manageable  without significant changes in “status quo” programs, policies, processes,  and operations.
section_title: GPRA Background
paragraphs: GPRA is the centerpiece of a statutory framework that Congress put in  place during the 1990s to help resolve the long-standing management  problems that have undermined the federal government’s efficiency and  effectiveness and to provide greater accountability for results.  In addition  to GPRA, the framework comprises the Chief Financial Officers Act of  1990, as amended by the Government Management Reform Act of 1994, and  information technology reform legislation, including the Paperwork  Reduction Act of 1995 and the Clinger-Cohen Act of 1996.  Together, these  laws provide a powerful framework for developing and integrating  information about agencies’ missions and strategic priorities, the results- oriented performance goals that flow from those priorities, performance  data to show the level of achievement of those goals, and the relationship  of reliable and audited financial information and information technology  investments to the achievement of those goals.
GPRA was intended to address several broad purposes, including  strengthening the confidence of the American people in their government;  improving federal program effectiveness, accountability, and service  delivery; and enhancing congressional decision making by providing more  objective information on program performance.
2.  The agency must develop annual performance plans covering each program    activity set forth in the agencies’ budgets*  Building on the decisions made as part of the strategic planning process, GPRA requires  executive agencies to develop annual performance plans covering each program activity set  forth in the agencies’ budgets. Annual performance plans, covering the upcoming fiscal year,  are to be submitted to Congress after the President’s Budget is submitted, which generally  occurs in February. Each plan is to contain an agency’s annual performance goals and  associated measures, which the agency is to use in order to gauge its progress toward  accomplishing its strategic goals. OMB is to use the agencies’ performance plans to develop  an overall federal government performance plan that is to be submitted with the President’s  Budget. The performance plan for the federal government is to present to Congress a single  cohesive picture of the federal government’s annual performance goals for a given fiscal  year. 3.  The agency must prepare annual reports on program performance for the previous    fiscal year, to be issued by March 31 each year GPRA requires executive agencies to prepare annual reports on program performance for  the previous fiscal year, to be issued by March 31 each year. In each report, an agency is to  compare its performance against its goals, summarize the findings of program evaluations  completed during the year, and describe the actions needed to address any unmet goals.  Recent OMB guidance states that executive agencies must combine their program  performance report with their accountability report and transmit the combined report for  fiscal year 2003 by January 30, 2004, and the combined report for fiscal year 2004 by  November 15, 2004. *Program activity refers to the lists of projects and activities in the appendix portion of the Budget of the United States Government. Program    activity structures are intended to provide a meaningful representation of the operations financed by a specific budget account.
The Office of Management and Budget (OMB) plays an important role in  the management of the federal government’s performance, and specifically  GPRA implementation.  Part of OMB’s overall mission is to ensure that  agency plans and reports are consistent with the President’s Budget and  administration policies.  OMB is responsible for receiving and reviewing  agencies’ strategic plans, annual performance plans, and annual  performance reports.  To improve the quality and consistency of these  documents, OMB issues annual guidance to agencies for their preparation,  including guidelines on format, required elements, and submission  deadlines.  GPRA requires OMB to prepare the overall governmentwide  performance plan, based on agencies’ annual performance plan  submissions.  OMB also played an important role in the pilot phase of  GPRA implementation by designating agencies for pilot projects in  performance measurement, managerial accountability and flexibility, and  performance budgeting, and assessing the results of the pilots.  Finally,  GPRA provides OMB with authority to grant agencies waivers to certain  administrative procedures and controls.
Recent OMB guidance requires agencies to submit “performance budgets”  in lieu of annual performance plans for their budget submission to OMB  and Congress.  Performance budgets are to meet all the statutory  requirements of GPRA for annual performance plans.  In addition, agencies  are to include all performance goals used in the assessment of program  performance done under OMB’s Program Assessment Rating Tool (PART)  process.  Moreover, the guidance states that until all programs have been  assessed by PART, the performance budget will also for a time include  performance goals for agency programs that have not yet been assessed  using PART.  The expectation is that agencies are to substitute new or  revised performance goals resulting from OMB’s review for goals it deemed  unacceptable.
In crafting GPRA, Congress recognized that managerial accountability for  results is linked to managers having sufficient flexibility, discretion, and  authority to accomplish desired results.  GPRA authorizes agencies to  apply for managerial flexibility waivers in their annual performance plans  beginning with fiscal year 1999.  The authority of agencies to request  waivers of administrative procedural requirements and controls is intended  to provide federal managers with more flexibility to structure agency  systems to better support program goals.  The nonstatutory requirements  that OMB can waive under GPRA generally involve the allocation and use  of resources, such as restrictions on shifting funds among items within a  budget account.  Agencies must report in their annual performance reports  on the use and effectiveness of any GPRA managerial flexibility waivers  that they receive.
OMB was to designate at least five agencies from the first set of pilot  projects to test managerial accountability and flexibility during fiscal years  1995 and 1996.  We previously reported on the results of the pilot project to  implement managerial flexibility waivers and found that the pilot did not  work as intended.  OMB did not designate any of the seven departments  and one independent agency that submitted a total of 61 waiver proposals  as GPRA managerial accountability and flexibility pilots.  For about three- quarters of the waiver proposals, OMB or other central management  agencies determined that the waivers were not allowable for statutory or  other reasons or that the requirement for which the waivers were proposed  no longer existed.  For the remaining proposals, OMB or other central  management agencies approved waivers or developed compromises by  using authorities that were already available independent of GPRA.
Under GPRA, another set of pilot projects, which were scheduled for fiscal  years 1998 and 1999, were to test performance budgeting—i.e., the  presentation of the varying levels of performance that would result from  different budget levels.  We previously reported that OMB initially deferred  these pilots—originally to be designated in fiscal years 1998 and 1999—to  give federal agencies time to develop the capability of calculating the  effects of marginal changes in cost or funding on performance.  When the  pilots began in August 1999, OMB designed them as case studies prepared  by OMB staff to demonstrate how performance information could be used  to compare alternatives and to develop funding recommendations for  incorporation into the President’s fiscal year 2001 budget submission.
On January 18, 2001, OMB reported the results of five performance  budgeting pilots that explored agencies’ capabilities of more formally  assessing the effects of different funding levels on performance goals.   OMB selected the pilots to reflect a cross section of federal functions and  capabilities so that a representative range of measurement and reporting  issues could be explored.  In its report, OMB concluded that legislative  changes were not needed.  OMB reported that the pilots demonstrated that  assuring further performance measurement improvements and steadily  expanding the scope and quality of performance measures is paramount,  and that the existing statute provides sufficient latitude for such  improvement.
Overall, OMB concluded that the pilots raised several key challenges about  performance budgeting at the federal level including, for example, the  following:   In many instances, measuring the effects of marginal, annual budget  changes on performance is not precise or meaningful.
While continuing to change from an almost total reliance on output  measures to outcome measures, it will be much more difficult to  associate specific resource levels with those outcomes, particularly over  short periods of time.
Establishing clear linkages between funding and outcomes will vary by  the nature of the program and the number of external factors.
Delays in the availability of performance data, sometimes caused by  agencies’ reliance on nonfederal program partners for data collection,  will continue to present synchronization problems during budget  formulation.
section_title: Scope and Methodology
paragraphs: To meet the three objectives stated earlier, we reviewed our extensive prior  work on GPRA best practices and implementation and collected  governmentwide data to assess the government’s overall focus on results.   We conducted a random, stratified, governmentwide survey of federal  managers comparable to surveys we conducted in 1997 and 2000.  We also  held eight in-depth focus groups—seven comprised of federal managers  from 23 federal agencies and one with GPRA experts.  We also interviewed  top appointed officials from the current and previous administrations.   Finally, we judgmentally selected a sample of six agencies to review for  changes in the quality of their strategic plans, performance plans, and  performance reports since their initial efforts.  The agencies we selected  were the Departments of Education (Education), Energy (DOE), Housing  and Urban Development (HUD), and Transportation (DOT) and the Small  Business (SBA) and Social Security Administrations (SSA).  In making this  selection, we chose agencies that collectively represented the full range of  characteristics in the following four areas:  (1) agency size (small, medium,  large); (2) primary program types (direct service, research, regulatory,  transfer payments, and contracts or grants); (3) quality of fiscal year 2000  performance plan based on our previous review (low, medium, high); and  (4) type of agency (cabinet department and independent agency).   Appendix I contains a more detailed discussion of our scope and  methodology.
We performed our work in Washington, D.C., from January through  November 2003 in accordance with generally accepted government  auditing standards.  Major contributors to this report are listed in appendix  XII.
section_title: GPRA Established the Foundation for a More Results-Oriented Federal Government
paragraphs: Among the purposes of GPRA cited by Congress was to improve federal  program effectiveness and service delivery by promoting a new focus on  results, service quality, and customer satisfaction by setting program goals,  measuring performance against goals, and reporting publicly on progress.   Furthermore, GPRA was to improve congressional decision making by  providing better information on achieving objectives, and on the relative  effectiveness and efficiency of federal programs and spending.  Ten years  after enactment, GPRA’s requirements have laid a foundation of results- oriented agency planning, measurement, and reporting that have begun to  address these purposes.  Focus group participants, high-level political  appointees, and OMB officials we interviewed cited positive effects of  GPRA that they generally attributed to GPRA’s statutory requirements for  planning and reporting.  Our survey results indicate that since GPRA went  into effect governmentwide in 1997, federal managers reported having  significantly more of the types of performance measures called for by  GPRA—particularly outcome-oriented performance measures.  GPRA has  also begun to facilitate the linking of resources to results, although much  remains to be done in this area.
subsections: section_title: GPRA Statutory Requirements Laid a Foundation for Agencywide Results- Oriented Management
paragraphs: Prior to enactment of GPRA, our 1992 review of the collection and use of  performance data by federal agencies revealed that, although many  agencies collected performance information at the program level, few  agencies had results-oriented performance information to manage or make  strategic policy decisions for the agency as a whole.  Federal agencies  surveyed indicated that many had a single, long-term plan that contained  goals, standards, or objectives for the entire agency or program.  Many of  these agencies also reported they collected a wide variety of performance  measures.  However, in validating the survey responses with a sample of  agencies, we found that measures were typically generated and used by  program-level units within an agency and focused on measuring work  activity levels and outputs or compliance with statutes.  Little of this  performance information was transparent to Congress, OMB, or the public  and few of the agencies we visited used performance measures to manage  toward long-term objectives.  Few of the agencies surveyed had the  infrastructure in place, such as a unified strategic plan with measurable  goals, an office that collected performance measures, and regular  consolidated reports, to tie plans and measures.
GPRA addressed these shortcomings by creating a comprehensive and  consistent statutory foundation of required agencywide strategic plans,  annual performance plans, and annual performance reports.  In contrast to  prior federal government efforts to measure performance, GPRA explicitly  emphasized that, in addition to performance indicators that agencies may  need to manage programs on a day-to-day basis, such as quantity, quality,  timeliness, and cost, agencies also needed outcome-oriented goals and  measures that assess the actual results, effects, or impact of a program or  activity compared to its intended purpose.
Expert and agency focus group participants cited the creation of this  statutory foundation as one of the key accomplishments of GPRA.   Participants agreed that GPRA created a framework in statute for federal  agencies to plan their activities in order to become more results oriented  and provided a managerial tool for program accountability.  Using this  framework, agencies could develop and focus on strategies to carry out the  programs they administer; set goals and identify performance indicators  that will inform them whether or not they achieved the performance they  expected; and determine what impact, if any, their programs have had on  the American public.  According to the experts in one of our focus groups,  comparing federal agencies’ current mission statements contained in their  strategic plans to what they were in the past demonstrates that the  agencies have done some “soul searching” to get a better sense of what  their role is (or should be) and how they can achieve it.  Given that GPRA is  in statute, participants indicated that the use of this planning framework is  likely to be sustained within agencies.
One of the premises of GPRA is that both congressional and executive  branch oversight of federal agency performance were seriously hampered  by a lack of adequate results-oriented goals and performance information.   As noted above, prior to the enactment of GPRA few agencies reported  their performance information externally.  OMB officials we interviewed as  part of our current review suggested that OMB has been a key consumer of  the performance information produced under GPRA and that it has  provided a foundation for their efforts to oversee agency performance.
For example, during the development of the fiscal year 2004 budget, OMB  used PART to review and rate 234 federal programs. We recently reported  that one of PART’s major impacts was its ability to highlight OMB’s  recommended changes in program management and design.  PART  reviews look at four elements—program purpose and design, strategic  planning, program management, and program results/accountability—and  rate the program on how well each of these elements is executed.   However, without the foundation of missions, goals, strategies,  performance measures, and performance information generated under  GPRA, such oversight would be difficult to carry out.
Participants in most of our focus groups also agreed that GPRA has been a  driving force behind many cultural changes that have occurred within  federal agencies.  Highlighting the focus on results, participants stated that  GPRA had stimulated a problem-solving approach within federal agencies  and encouraged agency managers to think creatively when developing  performance indicators for their programs.  GPRA has also changed the  dialogue within federal agencies; front-line managers and staff at lower  levels of the organization now discuss budget issues in connection with  performance.  Similarly, experts noted that information about performance  management and resource investments are more frequently communicated  between agency officials and Congress than in the past.  Within agencies,  GPRA documents can provide a context of missions, goals, and strategies  that political appointees can use to articulate agencies’ priorities.
subsections: section_title: Views on GPRA’s Effect on the Federal Government’s Ability to Deliver Results to the American Public Were Mixed
paragraphs: A key purpose of GPRA was “to improve the confidence of the American  people in the capability of the Federal Government, by systematically  holding Federal agencies accountable for achieving program results.”   When asked about the direct effects of GPRA on the public in our 2003  survey, an estimated 23 percent of federal managers agreed to a moderate  or greater extent that GPRA improved their agency’s ability to deliver  results to the American public; a larger percentage—38 percent—chose a  “no basis to judge/not applicable” category.
When a similar question was posed in our focus groups with experts and  federal managers, participants’ views were generally mixed.  Some federal  managers in our focus groups agreed that GPRA has had a positive effect  on raising awareness on many performance issues, and that in and of itself  is a way of delivering results.  The information gathered and reported for  GPRA allows agencies to make better-informed decisions, which improves  their ability to achieve results.  Other participants stated that while certain  aspects of GPRA-related work have been positive, agencies’ ability to  deliver results and public awareness of their activities cannot always be  exclusively attributed to GPRA.  For example, some participants stated that  many agencies rely on grant recipients to carry out their work, and  delivering results to the American public depends, to a large extent, on the  diligence of these organizations to implement their programs; such results  would not change dramatically if GPRA were no longer a requirement.
A number of the political appointees we interviewed cited examples of  outcomes they believe would not have occurred without the structure of  GPRA.  For example, a former deputy secretary of the Department of  Veterans Affairs (VA) stated that “the Results Act brought about a  fundamental rethinking of how we managed our programs and  processes. . . . We developed a strategic plan that was veteran-  focused. . . . We made every effort to define program successes from the  veteran’s perspective.”  A former Chief Financial Officer (CFO) cited  Customs Service goals to reduce the quantity of illegal drugs flowing into  the United States and the Food and Drug Administration’s focus on  speeding up the approval of new drugs as examples of outcomes that can  make a big difference in people’s lives.
Another major accomplishment of GPRA cited by our focus group  participants is that GPRA improved the transparency of government results  to the American public.  As noted above, prior to GPRA, few agencies  reported performance results outside of their agencies.  Focus group  participants indicated a key accomplishment of GPRA was its value as a  communication tool by increasing the transparency to the public of what  their agencies did in terms the public could understand.  For example,  information on agencies’ strategic plans, performance goals, measures, and  results are easily obtainable from agency Web sites.  One focus group  participant commented that GPRA helps bureaucrats explain to  nonbureaucrats what the federal government does in terms they can better  understand.  Other comments indicated that because of GPRA agencies  could now tell Congress and the American public what they are getting for  their money.
section_title: More Managers Reported Having Performance Measures
paragraphs: A fundamental element in an organization’s efforts to manage for results is  its ability to set meaningful goals for performance and to measure  performance against those goals.  From our 2003 survey we estimate that  89 percent of federal managers overall said there were performance  measures for the programs they were involved with.  This is a statistically  significantly higher percentage than the 76 percent of managers who  answered yes to this item on our 1997 survey.  (See fig. 2.)
Moreover, when we asked managers who said they had performance  measures which of five types of measures they had to a great or very great  extent, they reported increases in all five types of measures between 1997  and 2003, all of which were statistically significant.  (See fig. 3.)  Notably,  managers indicated the existence of outcome measures, defined as  “performance measures that demonstrate to someone outside the  organization whether or not intended results are being achieved,” grew  from a low of 32 percent in 1997 to the current estimate of 55 percent, a  level that is on par with output measures for the first time since we began  our survey.
Similarly, focus group participants commented on certain cultural changes  that had taken place within their agencies since the passage of GPRA in  which the “vocabulary” of performance planning and measurement—e.g., a  greater focus on performance management; orientation toward outcomes  over inputs and outputs; and an increased focus on program evaluation— had become more pervasive.  This perception is partly born out by our  survey results.  Since 1997 those reporting a moderate to extensive  knowledge of GPRA and its requirements shifted significantly from 26  percent to 41 percent in 2003, while those reporting no knowledge of GPRA  declined significantly from 27 percent to 20 percent.  (See fig. 4.)
Consistent with our survey results indicating increases in results-oriented  performance measures and increasing GPRA knowledge, we also observed  a significant decline in the percentage of federal managers who agreed that  certain factors hindered measuring performance or using the performance  information.  For example, as shown in figure 5, of those who expressed an  opinion, the percentage of managers who noted that determining  meaningful measures was a hindrance to a great or very great extent was  down significantly from 47 percent in 1997 to 36 percent in 2003.  Likewise,  the percentage that agreed to a great or very great extent that different  parties are using different definitions to measure performance was a  hindrance also declined significantly from 49 percent in 1997 to 36 percent  in 2003.
Finally, our survey data suggested that more federal managers, especially at  the Senior Executive Service (SES) level, believed that OMB was paying  attention to their agencies’ efforts under GPRA.  Moreover, there was no  corresponding increase in their concern that OMB would micromanage the  programs in their agencies.  In our survey, we asked respondents to assess  the extent to which OMB pays attention to their agencies’ efforts under  GPRA.  As seen in figure 6, in 2003, the percentage of respondents who  responded “Great” or “Very Great” to this question (31 percent) was  significantly higher than in 2000 (22 percent).  Of those, SES respondents  showed an even more dramatic increase, from 33 to 51 percent.  We also  asked respondents the extent to which their concern that OMB would  micromanage programs in their agencies was a hindrance to measuring  performance or using performance information.  The percentage among  those expressing an opinion that it was a hindrance to a great or very great  extent was low—around 24 percent in 2003—with no significant difference  between 2000 and 2003.
section_title: GPRA Has Begun to Establish a Link between Resources and Results
paragraphs: Among its major purposes, GPRA aims for a closer and clearer linkage  between requested resources and expected results.  The general concept of  linking performance information with budget requests is commonly known  as performance budgeting.  Budgeting is and will remain an exercise in  political choice, in which performance can be one, but not necessarily the  only, factor underlying decisions.  However, efforts to infuse performance  information into resource allocation decisions can more explicitly inform  budget discussions and focus them—both in Congress and in agencies—on  expected results, rather than on inputs.
GPRA established a basic foundation for performance budgeting by  requiring that an agency’s annual performance plan cover each program  activity in the President’s budget request for that agency.  GPRA does not  specify any level of detail or required components needed to achieve this  coverage.  Further, GPRA recognizes that agencies’ program activity  structures are often inconsistent across budget accounts and thus gives  agencies the flexibility to consolidate, aggregate, or disaggregate program  activities, so long as no major function or operation of the agency is  omitted or minimized.  In addition, OMB guidance has traditionally  required agencies to display, by budget program activity, the funding level  being applied to achieve performance goals.  OMB’s guidance on  developing fiscal year 2005 performance budgets also encourages a greater  link between performance and funding levels, however, it places greater  emphasis on linking agencies’ long-term and annual performance goals to  individual programs.  At a minimum, agencies are to align resources at the  program level, but they are encouraged to align resources at the  performance goal level.  Resources requested for each program are to be  the amounts needed to achieve program performance goal targets.
Our 1998 assessment of fiscal year 1999 performance plans found that  agencies generally covered the program activities in their budgets, but  most plans did not identify how the funding for those program activities  would be allocated to performance goals.  However, our subsequent  reviews of performance plans indicate that agencies have made progress in  demonstrating how their performance goals and objectives relate to  program activities in the budget.
Over the first 4 years of agency efforts to implement GPRA, we observed  that agencies continued to tighten the required link between their  performance plans and budget requests.  Of the agencies we reviewed over  this period, all but three met the basic requirement of GPRA to define a link  between their performance plans and the program activities in their budget  requests, and most of the agencies in our review had moved beyond this  basic requirement to indicate some level of funding associated with  expected performance described in the plan.  Most importantly, more of the  agencies we reviewed each year—almost 75 percent in fiscal year 2002  compared to 40 percent in fiscal year 1999—were able to show a direct link  between expected performance and requested program activity funding  levels—the first step in defining the performance consequences of  budgetary decisions.  However, we have also observed that the nature of  these linkages varied considerably.  Most of the agencies in our review of  fiscal year 2002 performance plans associated funding requests with higher,  more general levels of expected performance, rather than the more  detailed “performance goals or sets of performance goals” suggested in  OMB guidance.
Although not cited by our group of experts, participants at six of our seven  focus groups with federal managers cited progress in this area as a key  accomplishment of GPRA.  However, the participants also commented that  much remains to be done in this area.  The comments ranged from the  general—GPRA provides a framework for planning and budgeting, to the  more specific—GPRA created a definition of programs and how they will  help the agency achieve its goals/objectives and the amount of money that  will be required to achieve said goals/objectives.  One of the comments  implied that GPRA has helped to prioritize agency efforts by helping  agencies align their efforts with programs or activities that make a  difference.  A political appointee we interviewed echoed this comment,  stating that GPRA was pushing the department to think about what it gets  out of the budget, not just what it puts into it—12 to 15 years ago the “so  what” was missing from the budget process.  Another political appointee  we interviewed stated that the department was in the process of tying its  goals to its budget formulation and execution processes and linking  program costs to departmental goals.  A former political appointee  discussed how his department used program performance information to  inform a major information systems investment decision.
Furthermore, GAO case studies on the integration of performance  information in budget decision making found that performance information  has been used to inform the allocation of resources and for other  management purposes at selected agencies.  For example, the Veterans  Health Administration provides its health care networks with performance  information on patterns of patient care and patient health outcomes, which  can be used to analyze resource allocation and costs and reallocate  resources as appropriate.  Officials at the Administration for Children and  Families said that training and technical assistance and salaries and  expense funds are often allocated based on program and performance  needs.  The Nuclear Regulatory Commission monitors performance  against targets and makes resource adjustments, if needed, to achieve  those targets.
Although there has been progress in formally establishing the linkages  between budgets and plans, our survey results are somewhat conflicting  and have not reflected any notable changes either in managers’ perceptions  governmentwide as to their personal use of plans or performance  information when allocating resources, or in their perceptions about the  use of performance information when funding decisions are made about  their programs.  Our 2003 survey data show that a large majority of federal  managers reported that they consider their agency’s strategic goals when  they are allocating resources.  As shown in figure 7, on our 2003 survey, an  estimated 70 percent of all federal managers agreed to a great or very great  extent that they considered their agency’s strategic goals when allocating  resources.  However, using our 1997 survey responses as a baseline, it was  not a statistically significant increase over 64 percent of the managers who  responded comparably then.  As shown in figure 8, a similar, but somewhat  smaller, majority (60 percent) of managers who expressed an opinion on  our 2003 survey agreed to a great or very great extent that they used  information from performance measurement when they were involved in  allocating resources.  In 1997, the comparable response was about the  same at 62 percent.  When we asked managers on another item, however,  about the extent to which they perceived funding decisions for their  programs being based on results or outcome-oriented performance  information, only 25 percent of federal managers in 2003 endorsed this  view to a great or very great extent.  In 1997, 20 percent of managers  expressed a comparable view, again not a significant increase.  (See fig. 9.)
section_title: Agencies Have Addressed Many Critical Performance Planning and Reporting Challenges, but Weaknesses Persist
paragraphs: Beginning with federal agencies’ initial efforts to develop effective strategic  plans in 1997 and annual performance plans and reports for fiscal year  1999, Congress, GAO, and others have commented on the quality of those  efforts and provided constructive feedback on how agency plans and  reports could be improved.  On the basis of our current review of the  strategic plans, annual performance plans, and annual performance reports  of six selected agencies—Education, DOE, HUD, DOT, SBA, and SSA—we  found that these documents reflect much of the feedback that was  provided.  For example, goals were more quantifiable and results oriented,  and agencies were providing more information about goals and strategies  to address performance and accountability challenges and the limitations  to their performance data.  However, certain weaknesses, such as lack of  detail on how annual performance goals relate to strategic goals and how  agencies are coordinating with other entities to achieve common  objectives, persist.  A detailed discussion of our scope and methodology  and the results of our reviews of the six agencies’ most recent strategic  plans, annual performance plans, and annual performance reports  compared to initial efforts are contained in appendixes III, IV, and V,  respectively.
subsections: section_title: Quality of Selected Strategic Plans Reflects Improvements over Initial Drafts
paragraphs: Under GPRA, strategic plans are the starting point and basic underpinning  for results-oriented management.  GPRA requires that an agency’s strategic  plan contain six key elements:  (1) a comprehensive agency mission  statement; (2) agencywide long-term goals and objectives for all major  functions and operations; (3) approaches (or strategies) and the various  resources needed to achieve the goals and objectives; (4) a description of  the relationship between the long-term goals and objectives and the annual  performance goals; (5) an identification of key factors, external to the  agency and beyond its control, that could significantly affect the  achievement of the strategic goals; and (6) a description of how program  evaluations were used to establish or revise strategic goals and a schedule  for future program evaluations.
Our 1997 review of agencies’ draft strategic plans found that a significant  amount of work remained to be done by executive branch agencies if their  strategic plans were to fulfill the requirements of GPRA, serve as a basis for  guiding agencies, and help congressional and other policymakers make  decisions about activities and programs.  Our assessment of 27 agencies’  initial draft strategic plans revealed several critical strategic planning  issues that needed to be addressed.  These planning issues were as follows:  Most of the draft plans did not adequately link required elements in the  plans, such as strategic goals to annual performance goals.
Long-term strategic goals often tended to have weaknesses.
Many agencies did not fully develop strategies explaining how their  long-term strategic goals would be achieved.
Most agencies did not reflect in their draft plans the identification and  planned coordination of activities and programs that cut across multiple  agencies.
The draft strategic plans did not adequately address program  evaluations.
We noted that Congress anticipated that it may take several planning cycles  to perfect the process and that strategic plans would be continually refined  as various planning cycles occur.  We also recognized that developing a  strategic plan is a dynamic process and that agencies, with input from OMB  and Congress, were continuing to improve their plans.
Agencies have now had 6 years to refine their strategic planning processes.   Although the six strategic plans we looked at for this review reflected many  new and continuing strengths as well as improvements over the 1997 initial  drafts, we continued to find certain persistent weaknesses.  As depicted in  table 1, of the six elements required by GPRA, the plans generally discussed  all but one—program evaluation, an area in which we have found capacity  is often lacking in federal agencies.  Although the strategic plans generally  listed the program evaluations agencies planned to complete over the  planning period, they generally did not address how the agencies planned  to use their evaluations to establish new or revise existing strategic goals,  as envisioned by GPRA.  Finally, although not required by GPRA, the  strategic plans would have benefited from more complete discussions of  how agencies planned to coordinate with other entities to address common  challenges or achieve common or complementary goals.  Appendix III  provides a more detailed discussion of (1) the required and other useful  elements we reviewed to assess strategic plan strengths and weaknesses  and (2) changes in the quality of the six agencies’ strategic plans we  reviewed.
subsections: section_title: Strategic Planning Strengths and Improvements from Initial Draft Plans
paragraphs: Consistent with our review of agencies’ 1997 strategic plans, the recent  strategic plans we reviewed generally contained mission statements that  were results oriented, distinct from other agencies, and covered the  agencies’ major activities.  DOT’s mission statement had improved by  reflecting additional language from its enabling legislation that we  recommended adding during our 1997 review.  Still improvement could be  made in this area as is shown by DOE’s mission statement.  DOE’s mission  was results oriented but did not address the department’s activities related  to energy supply and conservation.
Our review of the current strategic plans also revealed improvements in the  development of agencies’ long-term, strategic goals—essential for results- oriented management.  Although GPRA does not require that all of an  agency’s long-term, strategic goals be results oriented, the intent of GPRA  is to have agencies focus their strategic goals on results to the extent  feasible.  In addition, as required by GPRA, the goals should be expressed  in a manner that could be used to gauge success in the future and should  cover an agency’s major functions or activities.  All of the strategic plans  we reviewed contained long-term, strategic goals that demonstrated  improvements in the quality of their 1997 goals.  Agencies’ long-term  strategic goals generally covered their missions, were results oriented, and  were expressed in a manner that could be used to gauge future success.   For example, SBA improved the quality of its long-term goals by focusing  more on key outcomes to be achieved and less on process improvements,  as was the case in its 1997 plan.  In some cases, we observed strategic goals  that addressed the agency’s organizational capacity to achieve results, such  as SSA’s long-term goal to strategically manage and align staff to support its  mission.
We also found improvements in how agencies’ current plans addressed  performance and accountability challenges we had identified, a key  weakness we identified in our earlier review.  Each of the agency plans we  reviewed discussed the long-term goals and strategies to address the  challenges that we had identified.  For example, Education’s strategic plan  contained a long-term strategic goal to modernize the Federal Student  Assistance programs and address identified problems in this area, which  we have designated as high risk since 1990.  SSA noted that it considered  GAO-identified performance and accountability challenges when it  determined its strategic goals and objectives, however not all of the  challenges are clearly addressed in the plan.
A third area of improvement we observed was in the description of the  strategies agencies planned to use to achieve their long-term strategic  goals.  In our review of agencies’ 1997 draft strategic plans, we found that  many agencies did not fully develop strategies explaining how their long- term strategic goals would be achieved.  In contrast, all six of the current  strategic plans we reviewed contained strategies that appeared logically  linked to achieving the agencies’ long-term goals.
Other strengths and improvements we observed in meeting GPRA’s basic  requirements involved the reporting of external factors that could affect  the achievement of the long-term goals and the identification of  crosscutting activities, although as indicated below these discussions could  be improved.  The six agencies reviewed for this report each reported on  external factors in current strategic plans.  For example, for each of the  strategic objectives in DOT’s strategic plan, DOT lists factors external to its  control and how those factors could affect the achievement of its  objectives.  Although not a requirement, some of the better plans we  reviewed discussed strategies to ameliorate the effect of external factors.   For example, for an external factor on teacher certification under a goal on  reading, Education’s plan states that the agency “will work with the states  and national accreditation bodies to encourage the incorporation of  research-based reading instruction into teacher certification  requirements.”
We have frequently reported that a focus on results, as envisioned by  GPRA, implies that federal programs contributing to the same or similar  results should be closely coordinated to ensure that goals are consistent  and, as appropriate, program efforts are mutually reinforcing.  This means  that federal agencies are to look beyond their organizational boundaries  and coordinate with other agencies to ensure that their efforts are aligned.   During our 1997 review, we found that most agencies did not reflect in their  draft plans the identification and planned coordination of activities and  programs that cut across multiple agencies.  In contrast, each of the six  current agency strategic plans that we reviewed identified at least some  activities and programs that cut across multiple agencies.  For example,  SBA’s 1997 plan contained no evidence of how the agency coordinated with  other agencies, but the current plan contained a separate section  describing crosscutting issues in the areas of innovation and research  assistance, international trade assistance, business development  assistance, veterans affairs, and disaster assistance.
section_title: Critical Strategic Planning Issues Needing Further Improvement
paragraphs: First, consistent with our 1997 review, the strategic plans we reviewed did  not adequately link required elements in the plans.  Although all of the  agencies we reviewed provided some information on the relationship  between their long-term and annual goals, the extent of information  provided on how annual goals would be used to measure progress in  achieving the long-term goals varied greatly.  In the case of DOE, the plan  provides a very brief description of the overall relationship between its  long-term and annual goals with examples, but does not demonstrate how  it will assess progress for each of its long-term goals and objectives.   Another plan, DOT’s, refers the reader to the annual performance plan for  information about annual goals.  We have reported that this linkage is  critical for determining whether an agency has a clear sense of how it will  assess progress toward achieving its intended results.
Second, although the agencies’ descriptions of their strategies had  improved since our initial reviews, with few exceptions, their strategies  generally did not include information on how the agencies plan to align  their activities, core processes, human capital, and other resources to  support their mission-critical outcomes and whether they have the right  mix of activities, skills, and resources to achieve their goals.  Such  information is critical to understanding the viability of the strategies.   Furthermore, none of the agencies discussed alternative strategies they  had considered in developing their plans.  Without such discussions, it is  unclear whether agency planning processes were truly strategic or simply a  recasting of existing activities, processes, etc.
HUD was the only agency that provided any details of how it intended to  coordinate with other agencies to achieve common or complementary  goals for its crosscutting programs or activities.  For example, to support  its goal of “Equal Opportunity in Housing,” HUD’s plan states that HUD and  the Department of Justice continue to coordinate their fair housing  enforcement activities, especially with respect to responding quickly and  effectively to Fair Housing Act complaints that involve criminal activity  (e.g., hate crimes), a pattern and practice of housing discrimination, or the  legality of state and local zoning or other land use laws or ordinances.  We  have reported that mission fragmentation and program overlap are  widespread throughout the federal government.  As such, interagency  coordination is important for ensuring that crosscutting programs are  mutually reinforcing and efficiently implemented.
Finally, the draft strategic plans did not adequately address program  evaluations.  In combination with an agency’s performance measurement  system, program evaluations can provide feedback to the agency on how  well its activities and programs contributed to achieving strategic goals.   For example, evaluations can be a potentially critical source of information  for Congress and others in assessing (1) the appropriateness and  reasonableness of goals; (2) the effectiveness of strategies by  supplementing performance measurement data with impact evaluation  studies; and (3) the implementation of programs, such as identifying the  need for corrective action.  Evaluations are important because they  potentially can be critical sources of information for ensuring that goals are  reasonable, strategies for achieving goals are effective, and that corrective  actions are taken in program implementation.  Five out of the six current  plans that we reviewed included a discussion of program evaluations,  however for most of these plans the discussions lacked critical information  required by GPRA, such as a discussion of how evaluations were used to  establish strategic goals or a schedule of future evaluations.  For example,  DOE’s plan stated that internal, GAO, and Inspector General (IG)  evaluations were used as resources to develop its draft strategic plan, but  specific program evaluations were not identified.
section_title: Fiscal Year 2004 Annual Performance Plans Addressed Some Weaknesses of Earlier Plans, but Still Have Room for Significant Improvement
paragraphs: According to our review of agencies’ first annual performance plans, which  presented agencies’ annual performance goals for fiscal year 1999, we  found that substantial further development was needed for these plans to  be useful in a significant way to congressional and other decision makers.   Most of the fiscal year 1999 plans that we reviewed contained major  weaknesses that undermined their usefulness in that they (1) did not  consistently provide clear pictures of agencies’ intended performance,   (2) generally did not relate strategies and resources to performance, and  (3) provided limited confidence that agencies’ performance data will be  sufficiently credible.  Although all of the fiscal year 1999 plans contained  valuable information for decision makers, their weaknesses caused their  usefulness to vary considerably within and among plans.
As shown in table 2, our current review of agencies’ fiscal year 2004  performance plans found that five agencies—Education, HUD, SBA, SSA,  and DOT—improved their efforts to provide a clear picture of intended  performance, with SSA and DOT being the clearest.  Furthermore, the same  five agencies improved the specificity of the strategies and resources they  intended to use to achieve their performance goals, with DOT being the  most specific.  Finally, the same five agencies—Education, HUD, SBA, SSA,  and DOT—made improvements in the area of greatest weakness— reporting on how they will ensure performance data will be credible.   However, only DOT’s plan provided a full level of confidence that the  performance data the agency intended to collect would be credible.   Appendix IV provides a more detailed discussion of (1) the required and  other useful elements we reviewed to assess the clarity of the picture of  intended performance, the specificity of the strategies and resources, and  the level of confidence in the performance data and (2) changes in the  quality of the six agencies’ annual performance plans we reviewed.
subsections: section_title: Plans Generally Provided a Clearer Picture of Intended Performance, Except for Crosscutting Areas
paragraphs: At the most basic level, an annual performance plan is to provide a clear  picture of intended performance across the agency.  Such information is  important to Congress, agency managers, and others for understanding  what the agency is trying to achieve, identifying subsequent opportunities  for improvement, and assigning accountability.  Our current review of  agencies’ fiscal year 2004 performance plans found that five of the six  agencies provided a clearer picture of intended performance than their  fiscal year 1999 plans did, although only two of the 2004 plans—DOT’s and  SSA’s—received the highest rating possible.  As shown in table 2, except for  DOT, the six agencies we reviewed for this report initially provided a  limited picture of intended performance.  Most of the fiscal year 1999  performance plans we previously reviewed had at least some objective,  quantifiable, and measurable goals, but few plans consistently included a  comprehensive set of goals that focused on the results that programs were  intended to achieve.  Moreover, agencies did not consistently follow OMB’s  guidance that goals for performance and accountability challenges be  included in the plans.  Agencies’ plans generally showed how their missions  and strategic goals were related to their annual performance goals and  covered all of the program activities in the agencies’ budget requests. In  addition, many agencies took the needed first step of identifying their  crosscutting efforts, with some including helpful lists of other agencies  with which they shared a responsibility for addressing similar national  issues.  However, the plans generally did not go further to describe how  agencies expected to coordinate their efforts with other agencies.
The fiscal year 2004 plans improved the picture of performance by making  annual goals and performance measures more results oriented, objective,  and quantifiable.  For example, Education’s plan included a measure for the  number of states meeting their eighth grade mathematics achievement  targets under the long-term goal to improve mathematics and science  achievement for all students.  We previously criticized Education’s 1999  plan for lacking such outcome-oriented measures.  Another overall  improvement we observed was that all of the plans described intended  efforts to address performance and accountability challenges we and  others had previously identified.  For instance, to address the  governmentwide high-risk area of strategic human capital management,  HUD states that to develop its staff capacity, it will complete a  comprehensive workforce analysis in 2004 to serve as the basis to fill  mission critical skill gaps through succession planning, hiring, and training  initiatives in a 5-year human capital management strategy.  The clarity of  DOE’s plan remained limited because its annual goals were not clearly  linked to its mission, the long-term goals in its strategic plan, or the  program activities in its budget request.
Although five of the six agencies improved the clarity of the picture of  intended performance, improvement is still needed in reporting on  crosscutting efforts.  In both the 1999 and 2004 plans, many agencies  identified their crosscutting efforts, with some including helpful lists of  other agencies with which they shared a responsibility for addressing  similar national issues.  Our review of fiscal year 2004 plans shows that the  six agencies we reviewed still did not discuss how they expected to  coordinate with other agencies to address common challenges or to  achieve common or complementary performance goals.  As we have  reported previously, improved reporting on crosscutting efforts can help  Congress use the annual performance plan to evaluate whether the annual  goals will put the agency on a path toward achieving its mission and long- term strategic goals.  In addition, the plans can aid in determining efforts to  reduce significant program overlap and fragmentation that can waste  scarce resources, confuse and frustrate program customers, and limit  overall program effectiveness.
None of the six agencies’ plans indicated an intention to request waivers of  specific administrative procedural requirements and controls that may be  impeding an agencies’ ability to achieve results.  This provision of GPRA  allows agencies greater managerial flexibility in exchange for  accountability for results.  We previously reported on the results of the  pilot project to implement this provision of GPRA and found that the pilot  did not work as intended.  OMB did not designate any of the seven  departments and one independent agency that submitted a total of 61  waiver proposals as GPRA managerial accountability and flexibility pilots.   For about three-quarters of the waiver proposals, OMB or other central  management agencies determined that the waivers were not allowable for  statutory or other reasons or that the requirement for which the waivers  were proposed no longer existed.  For the remaining proposals, OMB or  other central management agencies approved waivers or developed  compromises by using authorities that were already available independent  of GPRA.
section_title: Plans More Specifically Related Strategies and Resources to Performance Goals
paragraphs: To judge the reasonableness of an agency’s proposed strategies and  resources, congressional and other decision makers need complete  information on how the proposed strategies and resources will contribute  to the achievement of agency goals.  Agencies generally improved their  plans by better relating strategies and resources to performance.   Education’s, HUD’s, SBA’s, and SSA’s 1999 plans had a limited discussion,  while DOE’s and DOT’s 1999 plans had a general discussion.  In 2004, five of  the six plans—Education’s, DOE’s, HUD’s, SBA’s, and SSA’s—provided  general discussions of how their strategies and resources would contribute  to achieving their performance goals.  DOT’s 2004 plan improved to include  a specific discussion.
Our review of the 1999 plans found that most agencies’ performance plans  did not provide clear strategies that described how performance goals  would be achieved.  In contrast, the 2004 performance plans we reviewed  generally provided lists of the agencies’ current array of programs and  initiatives.  Several plans provided a perspective on how these programs  and initiatives were necessary or helpful for achieving results.  For  example, DOE and HUD included in their plans a “means and strategies”  section for each of their goals that described how the goal would be  achieved.  One strategy DOE identified to meet its goal of contributing  unique, vital facilities to the biological environmental sciences was to  conduct peer reviews of the facilities to assess the scientific output, user  satisfaction, the overall cost-effectiveness of each facility’s operations, and  their ability to deliver the most advanced scientific capability.
In addition, each of the agencies’ plans identified the external factors that  could influence the degree to which goals are achieved.  Some of the better  plans, such as DOT’s and SBA’s, provided strategies to mitigate the negative  factors or take advantage of positive factors, as appropriate.  For example,  for its transportation accessibility goals, DOT’s plan states that as the  population ages, more people will require accessible public transit, for  which states and local agencies decide how best to allocate federally  provided resources.  One of the strategies DOT intends to employ to  address this external factor is the “Special Needs of Elderly Individuals and  Individuals with Disabilities” grant program.  The plan states the grant  program will help meet transportation needs of the elderly and persons  with disabilities when regular transportation services are unavailable,  insufficient, or inappropriate to meet their needs.
Agencies’ 2004 plans did not consistently describe all the resources needed  and how they would be used to achieve agency goals.  Our review of  agencies’ fiscal year 1999 plans found that most did not adequately  describe—or reference other appropriate documents that describe—the  capital, human, information, and financial resources needed to achieve  their agencies’ performance goals.  The 2004 plans we reviewed generally  described the funding levels needed to achieve their performance goals  overall and in some cases broke out funding needs by high-level  performance goal.  For example, SSA’s plan provides a general perspective  on the financial resources needed to achieve its performance goals because  it provides budget information by account and program activity.  However,  the plan is neither structured by budget program activity or account, nor  does it provide a crosswalk between the strategic goals and budget  program accounts.  In contrast, HUD’s plan presented its requested funding  and staffing levels at the strategic goal level, but did not present budget  information at the level of its annual goals.  In addition, although the plans  make brief mention of nonfinancial resources, such as human capital,  information technology, or other capital investments, little information is  provided on how such resources would be used to achieve performance  goals.
section_title: Plans Continue to Provide Less Than Full Confidence That Performance Data Will Be Credible
paragraphs: Credible performance information is essential for accurately assessing  agencies’ progress towards the achievement of their goals and, in cases  where goals are not met, identifying opportunities for improvement or  whether goals need to be adjusted.  Under GPRA, agencies’ annual  performance plans are to describe the means that will be used to verify and  validate performance data.  To help improve the quality of agencies’  performance data, Congress amended GPRA through the Reports  Consolidation Act of 2000 to require that agencies assess the completeness  and reliability of the performance data in their performance reports.   Agencies were also required to discuss in their report any material  inadequacies in the completeness and reliability of their performance data  and discuss actions to address these inadequacies.  Meeting these new  requirements suggests the need for careful planning to ensure that agencies  can comment accurately on the quality of the performance data they report  to the public.
As shown in table 2, although five of the six agencies we reviewed  improved in reporting how they plan to ensure that performance data will  be credible, only one agency—DOT—improved enough over its 1999 plan  to provide a full level of confidence in the credibility of its performance  data.  Four agencies—Education, HUD, SBA, and SSA—improved enough  to provide a general level of confidence.  However, DOE provided the same  limited level of confidence in the credibility of the performance data as in  its 1999 plan.  Regarding all 24 of the fiscal year 1999 performance plans we  reviewed, we found most provided only superficial descriptions of  procedures that agencies intended to use to verify and validate  performance data.  Moreover, in general, agencies’ performance plans did  not include discussions of documented limitations in financial and other  information systems that may undermine efforts to produce high-quality  data.  As we have previously noted, without such information, and  strategies to address those limitations, Congress and other decision makers  cannot assess the validity and reliability of performance information.
We found that each of the 2004 plans we reviewed contained some  discussion of the procedures the agencies would use to verify and validate  performance information, although in some cases the discussion was  inconsistent or limited.  For example, the discussions of SBA’s verification  and validation processes for its indicators in the 2004 plan were generally  one- or two-sentence statements.  SBA also noted that it does not  independently verify some of the external data it gathers or that it does not  have access to the data for this purpose.  In contrast, the DOT plan referred  to a separate compendium available on-line that provides source and  accuracy statements, which give more detail on the methods used to  collect performance data, sources of variation and bias in the data, and  methods used to verify and validate the data.
In addition, all of the agencies except DOE discussed known limitations to  performance data in their plans.  These agencies’ plans generally provided  information about the quality of each performance measure, including any  limitations.  According to DOE officials, DOE’s plan generally does not  discuss data limitations because the department selected goals for which  data are expected to be available and therefore did not anticipate finding  any limitations.  However, in our 2003 Performance and Accountability  Series report on DOE, we identified several performance and  accountability challenges where data were a concern, such as the need for  additional information on the results of contractors’ performance to keep  projects on schedule and within budget.  DOE’s contract management  continues to be a significant challenge for the department and remains at  high risk.
Finally, the remaining five agencies also discussed plans to address  limitations to the performance data.  For example, DOT’s plan provided a  general discussion of the limitations to the internal and external sources of  data used to measure performance.  Detailed discussions were contained in  an appendix to the plan and separate source and accuracy statements.  This  information had been lacking in its 1999 plan.  Education, HUD, SBA, and  SSA also provided information on limitations to their performance data and  plans for improvement.
section_title: Strengths and Weaknesses of Selected Agencies’ Fiscal Year 2002 Annual Performance Reports
paragraphs: Key to improving accountability for results as Congress intended under  GPRA, annual performance reports are to document the results agencies  have achieved compared to the goals they established.  To be useful for  oversight and accountability purposes, the reports should clearly  communicate performance results, provide explanations for any unmet  goals as well as actions needed to address them, and discuss known data  limitations as well as how the limitations are to be addressed in the future.   Compared to our reviews of the six agencies’ fiscal year 1999 performance  reports, we identified a number of strengths and improvements as well as  areas that continued to need improvement.  Because the scope of our  review of the fiscal year 2002 reports was broader than that for the fiscal  year 1999 reports we previously reviewed, we were unable to make specific  comparisons for the three characteristics we used to assess the fiscal year  2002 reports.  However, we discuss comparative information on aspects of  the reports where available.  Table 3 shows the results of our assessment of  the six agencies’ annual performance reports for fiscal year 2002.   Appendix V provides a more detailed discussion of (1) the required and  other useful elements we reviewed to assess the clarity of the picture of  performance, the clarity of the linkage between costs and performance,  and the level of confidence in the performance data and (2) changes in the  quality of the six agencies’ annual performance plans we reviewed.
subsections: section_title: Progress in Providing a Clear Picture of Performance
paragraphs: The six agency reports that we reviewed contained a number of strengths,  some of which we can describe as improvements over the reports on fiscal  year 1999 performance.  A key strength of four of the 2002 reports  (Education, HUD, DOT, SSA) was a discussion of the relationship between  the strategic plan, performance plan, and performance report.  For  example, SSA’s report identified relevant results that were linked to its  strategic objective to deliver “citizen-centered, world-class service,” such  as maintaining the accuracy, timeliness, and efficiency of service to people  applying for its benefit programs.  The clarity of the DOE and SBA reports  was limited by not clearly relating agency performance results to strategic  and annual performance goals.  For example, the structure of SBA’s report  reflected the objectives in its draft 2003 to 2008 strategic plan rather than  those in its 2002 performance plan, making it difficult to assess progress  against the original 2002 objectives.  Furthermore, although there is no  “right” number of performance measures to be used to assess progress, a  number of the plans allowed for an easier review of results by limiting the  overall number of measures presented or by highlighting key performance  measures of greatest significance to their programs.  For example, SBA  discussed a total of 19 performance goals and DOT discussed a total of 40.   Although SSA discussed a total of 69 performance goals, the report  highlighted its progress in achieving 14 key goals.  In contrast, Education,  HUD, and DOE presented a total of 120, 184, and 260 measures,  respectively.  Furthermore, while Education and SSA each provided a table  showing progress across all its measures, the other agencies did not  provide such summary information.
As we found in our earlier reviews, the six agencies’ fiscal year 2002 reports  generally allowed for an assessment of progress made in achieving agency  goals.  Some of the reports made this assessment easier than others by  providing easy-to-read summary information.  For example, SSA provided a  table at the beginning of the report that summarized the results for each of  its 69 indicators with the following dispositions:  met, not met, almost met,  and data not yet available.  Other reports, such as HUD’s, required an  extensive review to make this assessment.  In addition, to place current  performance in context, each of the agencies’ reports contained trend  information, as required by GPRA, which allowed for comparisons between  current year and prior year performance.
In addition, the majority of agencies maintained, or demonstrated  improvements over, the quality of their 1999 reports in discussing the  progress achieved in addressing performance and accountability  challenges identified by agency IGs and GAO.  For example, SBA’s report  contained two broad overviews and an appendix describing the status of  GAO audits and recommendations, as well as a description of the most  serious management challenges SBA faces as identified by the agency’s IG.
Unfortunately, many of the weaknesses we identified in the agencies’ fiscal  year 2002 reports were similar to those we found in their fiscal year 1999  reports related to the significant number of performance goals (1) which  were not achieved and lacked explanations or plans for achieving the goal  in the future and (2) for which performance data were unavailable.  Three  of the six agencies we reviewed—HUD, SSA, and Transportation—did not  consistently report the reasons for not meeting their goals.  For example,  Transportation provided explanations for only 5 of the 14 goals it did not  meet.  In addition, similar to our 1999 report findings, three of the six  agencies we reviewed—HUD, SBA, and DOT—did not discuss their plans  or strategies to achieve unmet goals in the future.  For example, HUD  reported “substantially meeting” only 47 percent of the performance targets  in fiscal year 2002.  However, although HUD provides various reasons for  not meeting all its targets, it offers no information on plans or time frames  to achieve the goals in the future.  Finally, we continued to observe a  significant number of goals for which performance data were unavailable.   For example, performance data for 10 of SBA’s 19 performance goals were  unavailable.
In addition, the majority of the reports we reviewed did not include other  GPRA requirements.  The reports generally did not evaluate the  performance plan for the current year relative to the performance achieved  toward the performance goals in the fiscal year covered by the report.  The  reports also did not discuss the use or effectiveness of any waivers in  achieving performance goals.  In addition, for two of the agencies—DOE  and SBA—program evaluation findings completed during the fiscal year  were not summarized.  As we have previously noted, such evaluations  could help agencies understand the relationship between their activities  and the results they hope to achieve.
section_title: Progress in Linking Resources to Results
paragraphs: Although linking costs to performance goals is not a requirement of GPRA,  both GPRA and the CFO Act emphasized the importance of linking  program performance information with financial information as a key  feature of sound management and an important element in presenting to  the public a useful and informative perspective on federal spending.  The  committee report for GPRA suggested that developing the capacity to  relate the level of program activity with program costs, such as cost per  unit of result, cost per unit of service, or cost per unit of output, should be a  high priority.  In our survey of federal managers, this year we asked for the  first time the extent to which federal managers had measures of cost- effectiveness for the programs they were involved with.  Only 31 percent of  federal managers we surveyed reported having such measures to a great or  very great extent, lower than any of the other types of measures associated  with GPRA we asked about by at least 12 percent (see fig. 3 in ch. 2).  Under  the PMA, the current administration has set an ambitious agenda for  performance budgeting, calling for agencies to better align budgets with  performance goals and focus on capturing full budgetary costs and  matching those costs with output and outcome goals.  All this suggests that  agencies will need to develop integrated financial and performance  management systems that will enable the reporting of the actual costs  associated with performance goals and objectives along with presentations  designed to meet other budgetary or financial purposes, such as the  accounts and program activities found in the President’s Budget and  responsibility segments found in financial statements.
Of the six agencies we reviewed, only Education’s report clearly linked its  budgetary information to the achievement of its performance goals or  objectives.  Education’s report laid out, using both graphics and text, the  estimated appropriations associated with achieving each of its 24  objectives.  In addition the report provided the staffing in full-time  equivalent employment (FTEs) and an estimate of the funds from salaries  and expenses contributing to the support of each of these objectives.  SBA’s  report contained crosswalks that showed the relationship between SBA’s  strategic goals, outcome goals, performance goals, and programs.  Because  SBA shows the resources for each program, a reader can infer a  relationship between SBA’s resources and performance goals.  However,  the linkage between resources and results would be clearer if results and  resources were presented by performance goal as well.  SSA provided a  limited view of the costs of achieving its performance goals by providing  the costs associated with four out of five of its strategic goals.  However,  as reported by the IG, SSA needs to further develop its cost accounting  system, which would help link costs to performance.  DOE also provided  a limited view of the costs of achieving its performance goals by organizing  its performance information by budget program activity and associated net  costs.  According to DOE officials, the department plans to link its  individual performance measures to the costs of program activities in  future reports.  Neither HUD nor DOT provided information on the cost of  achieving individual performance goals or objectives.
section_title: Progress in Providing Confidence in the Credibility of Performance Data
paragraphs: To assess the degree to which an agency’s report provided full confidence  that the agency’s performance information was credible, we examined the  extent to which the reports discussed the quality of the data presented.  As  shown in table 3, only DOT’s report provided a full level of confidence in  the quality of the data.  The other agencies provided general or limited  confidence in their data.
All six agencies in our current review complied with the Reports  Consolidation Act of 2000 by including assessments of the completeness  and reliability of their performance data in their transmittal letters.  In  contrast, we found that only 5 of the 24 CFO Act agencies included this  information in their fiscal year 2000 performance reports.  Of the six  agencies in our current review, only DOE provided this assessment in its  fiscal year 2000 report.  For example, the Secretary of DOT stated in the  transmittal letter that the 2002 report “contains performance and financial  data that are substantially complete and reliable.”  However, only two of  the six agencies also disclosed material inadequacies in the completeness  and reliability of their performance data and discussed actions to address  the inadequacies in their transmittal letters.  For example, SBA stated in its  transmittal letter that it is “working to improve the completeness and  reliability of the performance data for the advice provided to small  business through SBA’s resource partners.”  SBA explained that data for  this aspect of its performance are collected through surveys, which are  inconsistent and not comparable, and for which client responses are  difficult to obtain.  SBA stated that it is working to improve the survey  instruments it uses to obtain performance data.
In addition to the requirements of the Reports Consolidation Act, we have  previously reported on other practices that enhance the credibility of  performance data that are not specifically required by GPRA.  For instance,  discussions of standards and methods used by agencies to assess the  quality of their performance data in their performance reports provide  decision makers greater insight into the quality and value of the  performance data.  None of the reports explicitly referred to a specific  standard they used, however, DOE described its method for assuring data  quality.  The report states that the heads of DOE’s organizational elements  certified the accuracy of their performance data.  DOE subsequently  reviewed the data for quality and completeness.
Other useful practices that help foster transparency to the public and assist  decision makers in understanding the quality of an agency’s data include:  (1) discussion of data quality, including known data limitations and actions  to address the limitations, and (2) discussion of data verification and  validation procedures, including proposals to review data collection and  verification and validation procedures.
All six agencies’ reports described data limitations, although discussions  were mostly brief and very high level.  One exception was DOT, which  directed readers to the DOT Web site to obtain an assessment of the  completeness and reliability of its performance data and detailed  information on the source, scope, and limitations of the performance data.   HUD and SBA also discussed plans for addressing the limitations.  For  example, HUD stated that to address problems with its indicator on the  number of homeowners who have been assisted with the Home Investment  Partnership Program (HOME), HUD has established a team of managers,  technical staff, and contractors to make a series of improvements to the  Integrated Disbursement and Information System beginning in fiscal year  2003 that should reduce the need to clean up the data.
Each of the six agencies’ reports also discussed the procedures they used  to verify and validate their performance data.  However, these discussions  ranged from the very general description of the DOE method (noted  previously), to the very detailed discussions provided by DOT.  DOT  provides an on-line compendium that discusses the source and accuracy of  its data.  Furthermore, DOT’s 2002 report also describes strategies being  undertaken to address the quality of its data.  The report states that a DOT  intermodal working group addressed data quality issues by developing  departmental statistical standards and by updating source and accuracy  statements for all of DOT’s data programs.  The working group also worked  to improve quality assurance procedures, evaluate sampling and  nonsampling errors, and develop common definitions for data across  modes.
section_title: Challenges to GPRA Implementation Persist
paragraphs: While a great deal of progress has been made in making federal agencies  more results oriented, numerous challenges still exist to effective  implementation of GPRA.  The success of GPRA depends on the  commitment of top leadership within agencies, OMB, and Congress.   However, according to federal managers surveyed, top leadership  commitment to achieving results has not grown significantly since our 1997  survey.  Furthermore, although OMB has recently shown an increased  commitment to management issues, it significantly reduced its guidance to  agencies on GPRA implementation compared to prior years, and it is not  clear how the program goals developed through its PART initiative will  complement and integrate with the long-term, strategic focus of GPRA.   Obtaining leadership commitment to implement a strategic plan depends in  part on the usefulness and relevance of agency goals and strategies to  agency leaders, Congress, and OMB.  However, GPRA’s requirement to  update agency strategic plans every 3 years is out of sync with presidential  and congressional terms and can result in updated plans that do not have  the support of top administration leadership and key congressional  stakeholders.
As noted in chapter 2, more federal managers surveyed reported having  results-oriented performance measures for their programs and we would  expect to have seen similar increases in the use of this information for  program management.  However, we did not observe any growth in their  reported use of this information for key management activities, such as  adopting new program approaches or changing work processes.   Additionally, managers noted human capital-related challenges that impede  results-oriented management, including a lack of authority and training to  carry out GPRA requirements, as well as a lack of recognition for the  results achieved.
Consistent with our previous work, federal managers in our focus groups  reported that significant challenges persist in setting outcome-oriented  goals, measuring performance, and collecting useful data.  However, our  survey data suggested that federal managers do not perceive issues, such  as “difficulty distinguishing between the results produced by the program  and results caused by other factors” and “difficulty obtaining data in time to  be useful,” to be substantial hindrances to measuring performance or using  performance information.
Additionally, mission fragmentation and overlap contribute to difficulties in  addressing crosscutting issues, particularly when those issues require a  national focus, such as homeland security, drug control, and the  environment.  GAO has previously reported on a variety of barriers to  interagency cooperation, such as conflicting agency missions, jurisdiction  issues, and incompatible procedures, data, and processes.  We have also  reported that OMB could use the provision of GPRA that calls for OMB to  develop a governmentwide performance plan to integrate expected agency- level performance.  Unfortunately, this provision has not been fully  implemented and the federal government lacks a tool, such as a strategic  plan, that could provide a framework for a governmentwide reexamination  of existing programs, as well as proposals for new programs.  Finally,  federal managers in our focus groups and political appointees we  interviewed believed that Congress does not use performance information  to the fullest extent to conduct oversight and to inform appropriations  decisions.  While there is concern regarding Congress’ use of performance  information, it is important to make sure that this information is initially  useful.  As a key user of performance information, Congress needs to be  considered a partner in shaping agency goals at the outset.  GPRA provides  Congress opportunities to influence agency performance goals through the  consultation requirement for strategic plans and through Congress’  traditional oversight role.
subsections: section_title: Top Leadership Does Not Consistently Show Commitment to Achieving Results
paragraphs: We have previously testified that perhaps the single most important  element of successful management improvement initiatives is the  demonstrated commitment of top leaders to change. This commitment is  most prominently shown through the personal involvement of top leaders  in developing and directing reform efforts. Organizations that successfully  address their long-standing management weaknesses do not “staff out”  responsibility for leading change.  Top leadership involvement and clear  lines of accountability for making management improvements are critical  to overcoming organizations’ natural resistance to change, marshalling the  resources needed in many cases to improve management, and building and  maintaining the organizationwide commitment to new ways of doing  business.
Results from our surveys show that while the majority of managers  continue to indicate top leadership demonstrates a strong commitment to  achieving results, we have not seen a noteworthy improvement in the  percentage of managers expressing this view.  From our 1997 survey, we  estimated about 57 percent of managers overall reported such commitment  to a great or very great extent.  On our 2003 survey, 62 percent of managers  expressed a comparable view—a higher but not statistically significant  increase.  (See fig. 10.)
As shown in figure 11, however, we continued to see a significant  difference between the perceptions of SES and non-SES managers on this  issue.  That is, the percentage of SES managers reporting that top  leadership demonstrated strong commitment to a great or very great extent  in 2003 was 22 percent higher than for non-SES managers.
We observed in our 1997 and 2000 reports on governmentwide  implementation of GPRA that we would expect to see managers’ positive  perceptions on items, such as the extent to which top leadership is  committed to achieving results, become more prevalent and the gap  between SES and non-SES managers begin to narrow as GPRA and related  reforms are implemented; however, these changes do not appear to be  happening as expected.
Demonstrating the willingness and ability to make decisions and manage  programs based on results and the ability to inspire others to embrace such  a model are important indicators of leadership commitment to results- oriented management.  However, in both our 1997 and 2000 surveys, only  about 16 percent of managers reported that changes by management above  their levels to the programs for which they were responsible were based on  results or outcome-oriented performance information to a great or very  great extent.  In our 2003 survey, this indicator increased to 23 percent, a  statistically significant increase from prior surveys.  Twenty-eight percent  of federal managers surveyed who expressed an opinion reported that the  lack of ongoing top executive commitment or support for using  performance information to make program/funding decisions hindered  measuring performance or using performance information to a great or  very great extent.
Our interviews with 10 top political appointees from the Clinton and  current Bush administrations indicated a high level of support and  enthusiasm for effectively implementing the principles embodied in GPRA.   For example, one appointee noted that GPRA focused senior management  on a set of goals and objectives to allow the organization to understand  what is important and how to deal with accomplishment at a macro-level,  as well as provided a structure for problem solving.  Another political  appointee noted that GPRA has made it important to look at what you get  out of the budget, not just what you put into it, while another concluded  that GPRA brought about a fundamental rethinking of how they managed  their programs and processes.  Such indications of support for GPRA are  promising.  However, to support the transition to more results-oriented  agency cultures, top agency management will need to make a more  concerted effort to translate their enthusiasm for GPRA into actions that  communicate to employees that top management cares about performance  results and uses the information in its decision making.
The need for strong, committed leadership extends to OMB as well.  OMB  has shown a commitment to improving the management of federal  programs, both through its leadership in reviewing agency program  performance using the PART tool as well as through the PMA, which calls  for improved financial performance, strategic human capital management,  competitive sourcing, expanded electronic government, and performance  budget integration.  Using the foundation of information generated by  agencies in their strategic plans, annual performance plans, and program  performance reports, OMB has used the PART tool to exercise oversight of  selected federal programs by assessing program purpose and design, the  quality of strategic planning, the quality of program management, and the  extent to which programs can demonstrate results.  PART provides OMB a  lens through which to view performance information for use in the budget  formulation process.  PART, and OMB’s use of performance data in the  budget formulation process, potentially can complement GPRA’s focus on  increasing the supply of credible performance information by promoting  the demand for this information in the budget formulation process.  As we  reported in chapter 2, more federal managers noted that OMB was paying  attention to their agencies’ efforts under GPRA.  (See fig. 6.)  Additionally,  OMB convened a performance measurement workshop in April 2003 to  identify practical strategies for addressing common performance  measurement challenges.  As a result of this workshop, it produced a paper  in June 2003 that included basic performance measurement definitions and  concepts and common performance measurement problems that were  discussed at the workshop.  This was part of OMB’s continuing efforts to  improve PART as an evaluation tool.
However, there are areas where OMB could further enhance its leadership.   OMB has stated that the PART exercise presents an opportunity to inform  and improve on agency GPRA plans and reports and establish a  meaningful, systematic link between GPRA and the budget process.  OMB  has instructed agencies that, in lieu of a performance plan, they are to  submit a performance budget that includes information from the PART  assessments, including all performance goals used in the assessment of  program performance done under the PART process.  The result is that  program-specific performance measures developed through the PART  review are to substitute for other measures developed by the agency  through its strategic planning process.  GPRA is a broad legislative  framework that was designed to be consultative with Congress and other  stakeholders and address the needs of many users of performance  information—Congress to provide oversight and inform funding decisions,  agency managers to manage programs and make internal resource  decisions, and the public to provide greater accountability.  Changing  agency plans and reports for use in the budget formulation process may not  satisfy the needs of these other users.  Users other than OMB are not likely  to find the information useful unless it is credible and valid for their  purposes.  PART’s program-specific focus may fit with OMB’s agency-by- agency budget reviews, but it is not well suited to achieving one of the key  purposes of strategic plans—to convey agencywide, long-term goals and  objectives for all major functions and operations.  PART’s focus on  program-specific measures does not substitute for the strategic, long-term  focus of GPRA on thematic goals and department- and governmentwide  crosscutting comparisons.
To reach the full potential of performance management, agency planning  and reporting documents need to reflect the full array of uses of  performance information, which may extend beyond those needed for  formulating the President’s Budget.  However, it is not yet clear whether the  results of those reviews, such as changes to agencies’ program  performance measures, will complement and be integrated with the long- term, strategic goals and objectives agencies have established in  consultation with Congress and other stakeholders under GPRA.  OMB has  not yet clearly articulated how PART is to complement GPRA.  Focus group  participants suggested that the administration and OMB needed to  reinforce GPRA’s usefulness as a management tool for agencies.  They also  emphasized the need for OMB to help agencies understand how to  integrate GPRA with other management initiatives, such as PART.
As we noted in chapter 3, agencies’ plans and reports still suffer from  persistent weaknesses and could improve in a number of areas, such as  attention to issues that cut across agency lines, and better information  about the quality of the data that underlie agency performance goals.   However, OMB’s July 2003 guidance for the preparation and submission of  strategic plans, annual performance plans, and annual performance reports  is significantly shorter and less detailed than its 2002 guidance.  For  example, OMB no longer provides detailed guidance to agencies for the  development of performance plan components.  OMB’s 2002 guidance on  the preparation and submission of annual performance plans is  approximately 39 pages long; in its 2003 guidance, that discussion spans  only 2 pages.  The 2003 guidance in this area does not include entire  sections found in the 2002 guidance, such as principles for choosing  performance goals and indicators for inclusion in the annual plan, types of  performance goals, crosscutting programs, and requirements for verifying  and validating data.
OMB needs to maintain and strengthen its leadership role in working with  agencies to help them produce the highest quality GPRA documents  through its formal guidance and reviews of strategic plan and report  submissions.  Focus group participants discussed the need for consistent  guidance on how to implement GPRA.  Furthermore, there is no evidence  that agencies have institutional knowledge of GPRA requirements that  would obviate the need for OMB’s guidance.  New managers will need a  consistent resource that provides practical guidance on what agencies  need to include in their planning and reporting documents to comply with  GPRA and reflect best practices.  Consistent, explicit OMB guidance on  preparing GPRA documents can help ensure that gains in the quality of  GPRA documents are maintained and provide a resource for agencies to  make further improvements in those documents.  For example, guidance  on how to discuss coordination of crosscutting programs or improvements  to the credibility of performance data in agency performance plans goes  hand-in-hand with OMB’s enhanced oversight of agency performance  through the PART exercise.
The success of GPRA depends on the commitment of top leadership within  agencies, OMB, and Congress.  Obtaining such leadership commitment  depends in part on the usefulness and relevance of agency goals and  strategies to these parties.  GPRA requires an agency to develop a strategic  plan at least every 3 years to cover the following 5-year period.  Thus, there  have been two required updates of strategic plans since the initial strategic  plans were submitted for fiscal year 1997—fiscal year 2000 and fiscal year  2003.  The fiscal year 2000 update occurred the year before a new  presidential term began.  According to our focus group participants—both  the experts and federal managers—it makes little sense to require an  update of a strategic plan shortly before a new administration is scheduled  to take office.  For example, changes in political leadership generally result  in a new agenda with new objectives.  Such changes force agencies to  revise their plans, management initiatives, and strategies, which translates  into additional GPRA-related work.  A strategic plan that does not reflect  the participation and buy-in of top administration leadership and key  congressional stakeholders is unlikely to be successfully implemented.   Therefore, GPRA’s requirement to update agency strategic plans according  to a schedule that is out of sync with presidential and congressional terms  means that effort may be wasted on plans that lack the support of top  leadership.
section_title: Managers Report Mixed Results in Use of Performance Information
paragraphs: GPRA’s usefulness to agency leaders and managers as a tool for  management and accountability was cited as a key accomplishment  numerous times by focus group participants.  However, a number of  alternative views indicated use of performance information for key  management decisions has been mixed.  For example, one participant said  they did not believe GPRA has been used as a tool yet, while another  participant commented that only better managers take advantage of GPRA  as a management tool.  According to focus group participants, although  many federal managers understand and use results-oriented management  concepts in their day-to-day activities, such as strategic planning and  performance measurement, they do not always connect these concepts to  the requirements of GPRA.
This view was strongly supported by our survey results.  Prior to  mentioning GPRA in our survey, we asked federal managers the extent to  which they consider their agency’s strategic goals when engaging in key  management tasks such as setting program activities, allocating resources,  or considering changes in their programs.  A relatively high percentage of  managers—ranging from 66 to 79 percent—responded to a great or very  great extent.  However, when we asked similar questions about the extent  to which they considered their agency’s annual performance goals as set  forth in the agency’s GPRA annual performance plan for the same  activities, the comparable responses were considerably lower, ranging  from 22 to 27 percent.
Because the benefit of collecting performance information is only fully  realized when this information is actually used by managers, we asked  them about the extent to which they used the information obtained from  measuring performance for various program management activities.  As  shown in figure 12, for seven of the nine activities we asked about, the  majority of managers who expressed an opinion reported using  performance information to a great or very great extent in 2003.  Across all  nine activities, the percentage of managers saying they used performance  information to a great or very great extent ranged from 41 percent for  developing and managing contracts to 60 percent for allocating resources,  setting individual job expectations, and rewarding staff.  While we had  observed a decline in the reported use of performance information to this  extent for many of these activities between 1997 and 2000, our 2003 results  increased to levels not significantly different from 1997 for all but one  category—adopting new program approaches or changing work processes.   This category of use continued to be significantly lower at 56 percent in  2003 than it was in 1997 at 66 percent.  Although another category,  coordinating program efforts with other internal or external organizations,  shows a similar pattern of limited recovery, the difference between the  1997 and 2003 results is not statistically significant.
We have reported that involving program managers in the development of  performance goals and measures is critical to increasing the relevance and  usefulness of this information to their day-to-day activities.  Yet, our survey  data indicate that participation in activities related to the development and  use of performance information has also been mixed.  In 2003, only 14  percent of managers believed to a great or very great extent that their  agencies considered their contributions to or comments on their agency’s  GPRA plans or reports.  However, significantly more SES managers (43  percent) than non-SES managers (12 percent) expressed this view.  Also,  when compared to our 2000 survey when we first asked this question, the  percentage of SES managers expressing this view in 2003 was significantly  higher than in 2000 (32 percent).  The percentage of non-SES managers was  essentially unchanged from 2000 (10 percent).
Furthermore, as shown in figure 13, overall around half or fewer of  managers responded “yes” on our 2003 survey to questions about being  involved in developing ways to measure whether program performance  goals are being achieved (46 percent), gathering and analyzing data to  measure whether programs were meeting their specific performance goals  (51 percent), or using measures for program performance goals to  determine if the agency’s strategic goals were being achieved (43 percent).   None of these overall results were significantly different from our 1997  results.  We did find, however, that significantly more SES managers  responded “yes” on the 2003 survey (72 percent) than the 1997 survey (55  percent) with regard to being involved in using performance measurement  information to determine if the agency’s strategic goals were being  achieved when compared to our 1997 results.
section_title: Managers Continue to Confront a Range of Human Capital Management Challenges
paragraphs: Managing people strategically and maintaining a highly skilled and  energized workforce that is empowered to focus on results are critically  important.  Such human capital management practices are essential to the  success of the federal government in the 21st century and to maximizing  the value of its greatest asset—its people.  Our survey results showed  continuing challenges related to the adequacy of managerial decision  making authority, training, and incentives.
subsections: section_title: Federal Managers Report That They Are Held Accountable for Program Results but Do Not Have the Decision-Making Authority They Need to Accomplish Agency Goals
paragraphs: High-performing organizations seek to shift the focus of management and  accountability from activities and processes to contributions and achieving  results.  In each of our three surveys, we asked managers about the amount  of decision-making authority they had and the degree to which they were  held accountable for results.
As shown in figure 14, for 2003, an estimated 40 percent of federal  managers overall reported that they had the decision-making authority they  needed to help the agency accomplish its strategic goals to a great or very  great extent.  This was a statistically significant increase over our 1997  estimate of 31 percent.  While there were more SES and non-SES managers  expressing this view on our 2003 survey than the 1997 survey, it was the  non-SES managers that showed the significant increase.  Despite this  promising trend, however, there continued to be substantial differences in  2003, as well as on the two previous surveys, between the responses of SES  and lower-level managers on this question.  Compared to the 57 percent of  SES managers who reported having such authority to a great or very great  extent in 2003, only 38 percent of non-SES managers reported having such  authority to a great or very great extent.
However, when asked the extent to which managers or supervisors at their  levels were held accountable for the accomplishment of agency strategic  goals, 57 percent responded to a great or very great extent in 2003.  Unlike  in other areas, where SES managers had significantly different views from  non-SES managers, there was little difference in the area of accountability.   (See fig. 15.)
This 57 percent is significantly higher than the 40 percent of managers  overall who indicated that they had comparable decision-making authority.   However, in contrast to the question on authority, as shown in figure 14,  where more SES managers than non-SES managers expressed the view that  they had the authority, there was little difference, as shown in figure 15,  between the two groups in their views about being held accountable for  achieving agency strategic goals to a great or very great extent.  As figures  14 and 15 further illustrate, roughly the same percentage of SES managers  perceived to a great or very great extent that managers at their level had  decision-making authority and accountability for achieving agency  strategic goals.  This result suggests that their authority was perceived to  be on par with their accountability.  In contrast, only 38 percent of non-SES  managers perceived that managers at their levels had the decision-making  authority they needed to a great or very great extent, while 57 percent  perceived that they were held accountable to a comparable extent.
Managers are hard-pressed to achieve results when they do not have  sufficient authority to act.  In our report containing the results of our 1997  survey, we noted that agencies needed to concentrate their efforts on areas  where managers were not perceiving or experiencing progress, such as that  concerning devolving decision-making authority to managers throughout  their organizations.  While authority for achieving results appears to be in a  modestly upward trend, the balance between authority and accountability  that fosters decision making to achieve results could be further improved,  particularly among non-SES managers.
section_title: Fewer Than Half of Managers Reported Training on Key Tasks
paragraphs: We previously reported on the need for agencies to expend resources on  effective training and professional development to equip federal employees  to work effectively. Among the resources focus group participants cited as  lacking included federal managers and staff with competencies and skills  needed to plan strategically, develop robust measures of performance, and  analyze what the performance data mean.  Our 2003 Guide calls for training  and development efforts to be strategically focused on improving  performance toward the agency’s goals and put forward with the agency’s  organizational culture firmly in mind.  Throughout this process it is  important that top leaders in the agencies communicate that investments in  training and development are expected to produce clearly identified  results.  By incorporating valid measures of effectiveness into the training  and development programs they offer, agencies can better ensure that they  will adequately address training objectives and thereby increase the  likelihood that desired changes will occur in the target population’s skills,  knowledge, abilities, attitudes, or behaviors.  Furthermore, if managers  understand and support the objectives of training and development efforts,  they can provide opportunities to successfully use the new skills and  competencies on the job and model the behavior they expect to see in their  employees.
In response to our 2003 survey, fewer than half of managers answered “yes”  when we asked them whether, during the past 3 years, their agencies had  provided, arranged, or paid for training that would help them accomplish  any of seven critical results-oriented management-related tasks.  However,  progress is indicated in our survey results.  As shown in figure 16, more  managers answered “yes” in 2003 on all seven training areas than in  previous surveys.  These increases were statistically significant for five of  the tasks—setting program performance goals, using program performance  information to make management decisions, linking program performance  to the achievement of agency strategic goals, and implementing the  requirements of GPRA.
As with our 2000 survey results, the 2003 survey results continued to  demonstrate that there is a positive relationship between agencies  providing training and development on setting program performance goals  and the use of performance information when setting or revising  performance goals.  For those managers who responded “yes” to training  on setting performance goals, 60 percent also reported that they used  information obtained from performance measurement when setting new or  revising existing performance goals to a great or very great extent.  In  contrast, for those managers who responded “no” to training on setting  performance goals, only 38 percent reported that they used information  obtained from performance measurement for setting new or revising  existing performance goals to a great or very great extent.  The difference  between these percentages is statistically significant.  Effective training  and development programs are an integral part of a learning environment  that can enhance the federal government’s ability to attract and retain  employees with the skills and competencies needed to achieve results.   Training and developing new and current staff to fill new roles and work in  different ways will be a crucial part of the federal government’s endeavors  to meet its transformation challenges.  Ways that employees learn and  achieve results will also continue to transform how agencies do business  and engage employees in further innovation and improvements.
section_title: Managers Perceive a Lack of Positive Recognition for Helping Agencies Achieve Results
paragraphs: Another fundamental aspect of the human capital management challenge  agencies face is providing the incentives to their employees to encourage  results-oriented management.  Monetary and nonmonetary incentives can  be used as a method for federal agencies to reward employees and to  motivate them to focus on results.
Overall, an increasing but still small percentage of managers reported in  1997, 2000, and 2003 that employees in their agencies received positive  recognition to a great or very great extent for helping agencies accomplish  their strategic goals.  In 1997, 26 percent of federal managers reported such  an extent of positive recognition as compared to 37 percent in 2003, a  statistically significant increase.  Interestingly, this improvement is seen in  the responses of non-SES managers.  As shown in figure 17, the percentage  of SES managers expressing this view stayed at about the same level over  the three surveys, while the percentage of non-SES managers holding this  view was significantly higher in 2003 than in 1997.  Even with this  improvement on the part of the responses from non-SES managers,  significantly more SES managers (47 percent) than non-SES managers (36  percent) expressed this perception to a comparable extent in 2003.
Unfortunately, most existing federal performance appraisal systems are not  designed to support a meaningful performance-based pay system in that  they fail to link institutional, program, unit, and individual performance  measurement and reward systems.  In our view, one key need is to  modernize performance management systems in executive agencies so that  they link to the agency’s strategic plan, related goals, and desired outcomes  and are therefore capable of adequately supporting more performance- based pay and other personnel decisions.
We have reported federal agencies can develop effective performance  management systems by implementing a selected, generally consistent set  of key practices.  These key practices helped public sector organizations  both in the United States and abroad create a clear linkage—“line of  sight”—between individual performance and organizational success and,  thus, transform their cultures to be more results oriented, customer- focused, and collaborative in nature.  Examples of such practices include  aligning individual performance expectations with organizational goals,  connecting performance expectations to crosscutting goals,  linking pay to individual and organizational performance, and  making meaningful distinctions in performance.
Beyond implementing these key practices, high-performing organizations  understand that their employees are assets whose value to the organization  must be recognized, understood, and enhanced.  They view an effective  performance management system as an investment to maximize the  effectiveness of people by developing individual potential to contribute to  organizational goals.  To maximize this investment, an organization’s  performance management system is designed, implemented, and  continuously assessed by the standard of how well it helps the employees  help the organization achieve results and pursue its mission.
section_title: Persistent Challenges in Setting Outcome- Oriented Goals, Measuring Performance, and Collecting Useful Data
paragraphs: In prior reports, we have described difficulties faced by federal managers  in developing useful, outcome-oriented measures of performance and  collecting data indicating progress achieved.  One of the most persistent  challenges has been the development of outcome-oriented performance  measures.  Additionally, it is difficult to distinguish the impact of a  particular federal program from the impact of other programs and factors,  thus making it difficult to attribute specific program performance to  results.  The lack of timely and useful performance information can also  hinder GPRA implementation.
subsections: section_title: Meaningful, Outcome- Oriented Performance Measures Are Sometimes Hard to Develop
paragraphs: In the past, we have noted that federal managers found meaningful  performance measures difficult to develop.  Focus group participants and  survey respondents noted that outcome-oriented performance measures  were especially difficult to establish when the program or line of effort was  not easily quantifiable.  The challenge of the “complexity of establishing  outcome-oriented goals and measuring performance” was cited by six of  the eight focus groups as one of the key challenges that managers face in  implementing GPRA.  Focus group participants agreed that they often felt  as if they were trying to measure the immeasurable, not having a clear  understanding of which performance indicators could accurately inform  the agency how it is carrying out a specific activity.  Managers from  agencies engaged in basic science research and development and grant- making functions noted that this effort was particularly difficult for them  because federal programs, especially those that are research-based, often  take years to achieve the full scope of their goals.  On our most recent  survey, we estimated that 36 percent of federal managers who had an  opinion indicated that the determination of meaningful measures hindered  the use of performance information or performance measurement to a  great or very great extent.  While this number was significantly lower than  the percentage of managers expressing the comparable view on the 1997 or  2000 survey and may reflect some lessening of this as a hindrance to some  managers, it nonetheless continues to be among those items having the  largest percentage of managers citing it as a substantial hindrance.
section_title: Impact of Federal Programs Difficult to Discern
paragraphs: In our June 1997 report on GPRA, we noted that “the often limited or  indirect influence that the federal government has in determining whether  a desired result is achieved complicates the effort to identify and measure  the discrete contribution of the federal initiative to a specific program  result.” This occurs primarily because many federal programs’ objectives  are the result of complex systems or phenomena outside the program’s  control.  In such cases, it is particularly challenging for agencies to  confidently attribute changes in outcomes to their program—the central  task of program impact evaluation.  This is particularly challenging for  regulatory programs, scientific research programs, and programs that  deliver services to taxpayers through third parties, such as state and local  governments.
We have reported that determining the specific outcomes resulting from  federal research and development has been a challenge that will not be  easily resolved.  Due to the difficulties in identifying outcomes, research  and development agencies typically have chosen to measure a variety of  proxies for outcomes, such as the number of patents resulting from  federally funded research, expert review and judgments of the quality and  importance of research findings, the number of project-related publications  or citations, and contributions to expanding the number of research  scientists.
We have also reported that implementing GPRA in a regulatory  environment is particularly challenging.  Although federal agencies are  generally required to assess the potential benefits and costs of proposed  major regulatory actions, they generally do not monitor the benefits and  costs of how these and other federal programs have actually performed.   For example, in the case of the Environmental Protection Agency (EPA), to  determine if existing environmental regulations need to be retained or  improved, we previously recommended that EPA study the actual costs and  benefits of such regulations.
In the past, regulatory agencies have cited numerous barriers to their  efforts to establish results-oriented goals and measures.  These barriers  included problems in obtaining data to demonstrate results, accounting for  factors outside of the agency’s control that affect results, and dealing with  the long time periods often needed to see results.  Our prior work  discussed best practices for addressing challenges to measuring the results  of regulatory programs.  In particular, to address the challenge of  discerning the impact of a federal program, when other factors also affect  results, we suggested agencies “establish a rationale of how the program  delivers results.”  Establishing such a rationale involves three related  practices:  (1) taking a holistic or “systems” approach to the problem being  addressed, (2) building a program logic model that described how activities  translated to outcomes, and (3) expanding program assessments and  evaluations to validate the model linkages and rationale.
We have also reported on the difficulties encountered in meeting GPRA  reporting requirements for intergovernmental grant programs.  Programs  that do not deliver a readily measurable product or service are likely to  have difficulty meeting GPRA performance measurement and reporting  requirements.  Intergovernmental grant programs, particularly those with  the flexibility inherent in classic block grant design, may be more likely to  have difficulty producing performance measures at the national level and  raise delicate issues of accountability.  Although most flexible grant  programs we reviewed reported simple activity or client counts, relatively  few of them collected uniform data on the outcomes of state or local  service activities.  Collecting such data requires conditions (such as  uniformity of activities, objectives, and measures) that do not exist under  many flexible program designs, and even where overall performance of a  state or local program can be measured, the amount attributable to federal  funding often cannot be separated out.
Focus group participants also suggested that they faced challenges in  obtaining timely performance data from relevant partner organizations and  in identifying what the federal government’s contribution has been to a  specific outcome.  Furthermore, survey respondents provided some  corroboration for these views.  Across all three of our surveys, we estimate  that roughly a quarter of all federal managers reported this difficulty— distinguishing between the results produced by the program they were  involved with and results caused by other factors—as a substantial  hindrance.  In response to a survey question about what the federal  government could do to improve its overall focus on managing for results,  one respondent noted:  “Defining meaningful measures for the work we do  is extremely difficult; and even if they could be defined, performance and  accomplishment is (sic) dependent on so many factors outside our control  that it is difficult, if not impossible, to make valid conclusions.”
section_title: Timely, Useful Performance Information Not Always Available
paragraphs: In February 2000, we reported that intergovernmental programs pose  potential difficulties in collecting timely and consistent national data.  We  also noted that agencies had limited program evaluation capabilities and  weaknesses in agencies’ financial management capabilities make it difficult  for decision makers to effectively assess and improve many agencies’  financial performance.  On the basis of our current findings, these issues  still exist.  Federal managers who participated in our focus groups cited  difficulties in gathering data from state or local entities, as well as statutory  limitations regarding the nature and breadth of data that they were  permitted to collect.  However, in our 2003 survey, only 27 percent of  federal managers indicated that obtaining data in time to be useful was a  substantial hindrance; 31 percent expressed a comparable view with regard  to obtaining valid or reliable data.
Focus group participants also noted that OMB’s accelerated time frames  for reporting performance information will contribute to the challenge of  producing complete, timely information in their agencies’ performance  reports.  Over the past 2 fiscal years, OMB has moved the deadline for  submission of agencies’ performance reports (now performance and  accountability reports) back from the statutory requirement of March 31;  for fiscal year 2003 data, the deadline is January 30, 2004.  In fiscal year  2004, these reports will be due on November 15, 2004.  According to the  managers, individual agencies may work on different time frames based  partially on the population they serve or the stakeholders they must work  with, such as state or local agencies.  This “one size fits all” approach does  not take such differences into account.
Additionally, OMB requires agencies to report on their performance data  quarterly; managers noted that this was particularly difficult for outcomes  that may be achieved over extended periods of time, such as outcomes  associated with basic science.  As we have previously reported, measuring  the performance of science-related projects can be difficult because a wide  range of factors determine if and how a particular research and  development project will result in a commercial application or have other  benefits.  Efforts to cure diseases or pursue space exploration are difficult  to quantify and break down into meaningful quarterly performance  measures.
section_title: Crosscutting Issues Hinder Successful GPRA Implementation
paragraphs: Crosscutting issues continue to be a challenge to GPRA implementation.   Mission fragmentation and program overlap are widespread across the  federal government.  Moreover, addressing this challenge is essential to the  success of national strategies in areas such as homeland security, drug  control, and the environment.
We have reported that agencies could use the annual performance planning  cycle and subsequent annual performance reports to highlight crosscutting  program efforts and to provide evidence of the coordination of those  efforts.  Our review of six agencies’ strategic and annual performance plans  showed some improvement in addressing their crosscutting program  efforts, but a great deal of improvement is still necessary.  Few of the plans  we reviewed attempted the more challenging task of discussing planned  strategies for coordination and establishing complementary performance  goals and complementary or common performance measures.  For  example, SSA’s 2004 performance plan makes some mention of the agency’s  efforts to coordinate with other agencies to preserve the integrity of the  Social Security number as a personal identifier, but there are very few  details about this important component of its mission.
Previous GAO reports and agency managers identified several barriers to  interagency coordination.  First, missions may not be mutually reinforcing  or may even conflict, making reaching a consensus on strategies and  priorities difficult.  In 1998 and 1999, we found that mission fragmentation  and program overlap existed in 12 federal mission areas, ranging from  agriculture to natural resources and the environment.  Implementation of  federal crosscutting programs is often characterized by numerous  individual agency efforts that are implemented with little apparent regard  for the presence of related activities.  Second, we reported on agencies’  interest in protecting jurisdiction over missions and control over  resources.  Focus group participants echoed this concern, noting that  there can be “turf battles” between agencies, where jurisdictional  boundaries, as well as control over resources, are hotly contested.  Finally,  incompatible procedures, processes, data, and computer systems pose  difficulties for agencies to work across agency boundaries.  For example,  we reported how the lack of consistent data on federal wetlands programs  implemented by different agencies prevented the government from  measuring progress toward achieving the governmentwide goal of no net  loss of the nation’s wetlands.
We have previously reported and testified that GPRA could provide OMB,  agencies, and Congress with a structured framework for addressing  crosscutting program efforts.  OMB, for example, could use the provision  of GPRA that calls for OMB to develop a governmentwide performance  plan to integrate expected agency-level performance.  Unfortunately, this  provision has not been fully implemented.  OMB issued the first and only  such plan in February 1998 for fiscal year 1999.  In our review of the plan,  we found that it included a broad range of governmentwide management  objectives and a mission-based presentation of key performance goals  based on agency performance plans and the plan’s framework should  ultimately allow for a cohesive presentation of governmentwide  performance.  However, the specific contents of this initial plan did not  always deliver an integrated, consistent, and results-oriented picture of  fiscal year 1999 federal government performance goals.
OMB officials we interviewed at the time stressed that developing the  governmentwide plan was viewed as an essential and integral component  of the President’s budget and planning process.  From OMB’s perspective,  both the plan and the budget submission were intended to serve as  communication tools for a range of possible users.  In their opinion, the  plan added value by reflecting a governmentwide perspective on policy  choices made throughout the budget formulation process.  OMB  acknowledged that the plan itself did not serve to change the process  through which decisions on government priorities were made, but  enhanced it by placing a greater emphasis on results.  As one official  described it, the governmentwide performance plan was a derivative  document, reflecting the budget and management decisions made  throughout the process of formulating the President’s budget submission.   However, we found that focusing broadly on governmentwide outcomes  should be a central and distinguishing feature of the federal government  performance plan.  To be most effective and supportive of the purposes of  GPRA, the governmentwide plan must be more than a compilation of  agency-level plans; integration, rather than repetition, must be its guiding  principle.
OMB has not issued a distinct governmentwide performance plan since  fiscal year 1999.  Most recently, the President’s fiscal year 2004 budget  focused on describing agencies’ progress in addressing the PMA and the  results of PART reviews of agency programs.  Although such information is  important and useful, it does not provide a broader and more integrated  perspective of planned performance on governmentwide outcomes.   Additionally, the fiscal year 2004 budget identified budget requests and  performance objectives by agency, such as the U.S. Department of Defense,  as opposed to crosscutting governmentwide themes.  From this  presentation, one could assume that the only activities the U.S. government  planned to carry out in support of national defense were those listed under  the chapter “Department of Defense.”  However, the chapter of the fiscal  year 2004 budget discussing “the Department of State and International  Assistance Programs,” contains a heading titled, “Countering the Threat  from Weapons of Mass Destruction.”  And while OMB may have a technical  reason for not classifying this task as being related to national defense or  homeland security, it is unclear that a lay reader could make that  distinction.  The fiscal year 2005 budget also identified budget requests by  agency, not by crosscutting theme.  Without such a governmentwide focus,  OMB is missing an opportunity to assess and communicate the relationship  between individual agency goals and outcomes that cut across federal  agencies and more clearly relate and address the contributions of  alternative federal strategies.  The governmentwide performance plan also  could help Congress and the executive branch address critical federal  performance and management issues, including redundancy and other  inefficiencies in how we do business.  It could also provide a framework for  any restructuring efforts.
A strategic plan for the federal government, supported by key national  indicators to assess the government’s performance, position, and progress,  could provide an additional tool for governmentwide reexamination of  existing programs, as well as proposals for new programs.  If fully  developed, a governmentwide strategic plan could potentially provide a  cohesive perspective on the long-term goals of the federal government and  provide a much needed basis for fully integrating, rather than merely  coordinating, a wide array of federal activities.  Successful strategic  planning requires the involvement of key stakeholders.  Thus, it could serve  as a mechanism for building consensus.  Further, it could provide a vehicle  for the President to articulate long-term goals and a road map for achieving  them.  In addition, a strategic plan could provide a more comprehensive  framework for considering organizational changes and making resource  decisions.
Developing a strategic plan for the federal government would be an  important first step in articulating the role, goals, and objectives of the  federal government.  It could help provide critical horizontal and vertical  linkages.  Horizontally, it could integrate and foster synergies among  components of the federal government as well as help to clarify the role of  the federal government vis-à-vis other sectors of our society.  Vertically, it  could provide a framework of federal missions and goals within which  individual federal agencies could align their own missions and goals that  would cascade down to individual employees.  The development of a set of  key national indicators could be used as a basis to inform the development  of the governmentwide strategic and annual performance plans.  The  indicators could also link to and provide information to support outcome- oriented goals and objectives in agency-level strategic and annual  performance plans.
section_title: Managers View Congress’ Use of Performance Information as Limited
paragraphs: Focus group members believed that one of the main challenges to GPRA  implementation was the reluctance of Congress to use that information  when making decisions, especially appropriations decisions.  This concern  was cited as a significant challenge in each of the focus groups, and was  one of the top three “challenges” in five of the eight focus groups.  In some  cases, managers in our focus groups noted that this lack of usage was a  significant disincentive to doing a good job in preparing GPRA plans and  reports.  Agency managers made the following criticisms regarding the  perceived lack of congressional use of performance information:  appropriators have not bought into GPRA, so there is no incentive to do   failure of congressional leadership in developing and using performance   appropriators do not use performance data or tools to make decisions,   GPRA does not drive public policy decisions.
Results from our survey provide some further information in support of  this view.  On our 2003 survey, when we asked federal managers about the  extent to which they thought congressional committees paid attention to  agency efforts under GPRA, only 22 percent of federal managers responded  in the great to very great categories.  This result was not significantly  different from the results we observed on our 2000 survey when we asked  this question about three specific types of congressional committees— authorization, appropriation, and oversight.  On the 2000 survey, only 18  percent of federal managers held a similar view concerning authorizing  committees, 19 percent for appropriations committees, and 20 percent for  oversight committees.  As we noted earlier, when this item was asked in  relation to OMB, there was a significant increase in the percentage of  managers responding to a great or very great extent from 2000 to 2003.  The  31 percent of managers who viewed OMB as paying attention to a great or  very great extent in 2003 was significantly higher than the 22 percent  holding a comparable view of congressional committees.
Although managers expressed these concerns about the use of this  information, a recent review by the CRS suggested that Congress uses  performance information to some extent, as evidenced by citations in  legislation and committee reports.  For example, in the 106th Congress  (1999-2000), 42 public laws contained statutory language relating to GPRA  and performance measures, and 118 legislative reports contained GPRA- associated passages.  As shown in figure 18, across all three of our surveys,  only a minority of federal managers governmentwide viewed the lack of  ongoing congressional commitment for using performance information as a  hindrance to a great or very great extent.
While there is concern regarding Congress’ use of performance  information, it is important to make sure that this information is initially  useful.  One of GPRA’s purposes is to respond to a need for accurate,  reliable information for congressional decision making.  In 2000, we  reported that congressional staffs stated that they were looking for  recurring information on spending priorities within programs; the quality,  quantity, and efficiency of program operations; the populations served or  regulated; as well as programs’ progress in meeting their objectives.  For  example, learning who benefits from a program can help in addressing  questions about how well services are targeted to those most in need.   Some of these recurring needs were met through formal agency  documents, such as annual performance plans.  However, some  information the agencies provided did not fully meet the congressional  staffs’ needs because the presentation was not clear, directly relevant, or  sufficiently detailed.  For example, congressional staffs wanted to see more  direct linkages among the agencies’ resources, strategies, and goals.  In  other cases, the information was not readily available to the congressional  staffs, either because it had not been requested or reported, or because  staff were not informed that it was available.
As a key user of performance information, Congress also needs to be  considered a partner in shaping agency goals at the outset.  For example,  through the strategic planning requirement, GPRA requires federal  agencies to consult with Congress and key stakeholders to reassess their  missions and long-term goals as well as the strategies and resources they  will need to achieve their goals.  GPRA also provides a vehicle for Congress  to explicitly state its performance expectations in outcome-oriented terms  when establishing new programs or in exercising oversight of existing  programs that are not achieving desired results.  Congress could use  authorizing and appropriations hearings to determine if agency programs  have clear performance goals, measures, and data with which to track  progress and whether the programs are achieving their goals.  If goals and  objectives are unclear or not results oriented, Congress could use  legislation to articulate the program outcomes it expects agencies to  achieve.  This would provide important guidance to agencies that could  then be incorporated in agency strategic and annual performance plans.
section_title: Conclusions and Recommendations
subsections: section_title: Agenda for Achieving a Sustainable, Governmentwide Focus on Results
paragraphs: As we have shown in this report, in the 10 years since the enactment of  GPRA, significant progress has been made in instilling a focus on results in  the federal government.  First, GPRA statutory requirements laid a  foundation for results-oriented management in federal agencies.  Expert  and agency focus group participants cited the creation of this statutory  foundation as one of the key accomplishments of GPRA.  Since GPRA  began to be implemented governmentwide in fiscal year 1997, we have  observed significant increases in the percentage of federal managers who  reported having results-oriented performance measures for their programs.   Focus group participants’ views on whether GPRA has had a positive effect  on the federal government’s ability to deliver results to the American public  were mixed.  For example, the information gathered and reported for  GPRA allows agencies to make better-informed decisions, which improves  their ability to achieve results.  In addition, GPRA has made the results of  federal programs more transparent to the public.  Other participants stated  that while certain aspects of GPRA-related work have been positive,  agencies’ ability to deliver results and public awareness of their activities  cannot be exclusively attributed to GPRA.
Second, GPRA has increased the connection between resources and results  by creating more formal linkages between agency performance goals and  objectives and the program activities in the budget.  Over the first 4 years of  agency efforts to implement GPRA, we observed that agencies continued to  tighten the required linkage between their performance plans and budget  requests.  However, much remains to be done in this area.  For example, we  have not observed notable increases in federal managers’ perceptions  about their personal use of plans or performance information when  allocating resources, or about the use of performance information when  funding decisions are made about their programs.  However, it should be  noted that we estimate a majority have positive perceptions about the use  of performance information to allocate resources.
Third, GPRA has provided a foundation for examining agency missions,  performance goals and objectives, and the results achieved.  We have seen  improvements in the quality of agency strategic plans, annual performance  plans, and performance reports since initial efforts.  However, few of the  six agencies we reviewed in this report produced GPRA planning and  reporting documents that met all of our criteria for the highest level of  quality.  Most of these agencies continued to miss opportunities to present  clear pictures of their intended and actual performance results in their  GPRA plans and reports and to show how resources are aligned with actual  performance results.  Furthermore, most of the agencies we reviewed did  not provide a full level of confidence in the credibility of their performance  data.
Performance-based management, as envisioned by GPRA, requires  transforming organizational cultures to improve decision making,  maximize performance, and assure accountability.  This transformation is  not an easy one and requires investments of time and resources as well as  sustained leadership commitment and attention.  Challenges to successful  implementation of GPRA include inconsistent top leadership commitment  to creating a focus on results; an approach to setting goals and developing  strategies for achieving critical outcomes that creates individual agency  stovepipes rather than an integrated, holistic governmentwide approach;  getting federal managers to make greater use of performance information  to manage their programs and providing them authority to act that is  commensurate with their accountability for results; difficulty in  establishing meaningful measures of outcomes and assessing results of  federal programs that are carried out by nonfederal entities; and untimely  performance data.
The challenges identified in this report are not new—most have not  changed significantly since we first reported on governmentwide  implementation of GPRA.  However, we have frequently reported on  approaches that agencies, OMB, and Congress could use to address the  challenges.  These approaches include strengthening the commitment of  top leadership to creating and sustaining a focus on results; taking a  governmentwide approach to achieving outcomes that are crosscutting in  nature; improving the usefulness of performance information to managers,  Congress, and the public; and improving the quality of performance  measures and data.  Collectively, these approaches form the agenda that  federal agencies, OMB, and Congress will need to follow to bring about a  more sustainable, governmentwide focus on results.
subsections: section_title: Strengthening Top Leadership Commitment to Creating and Sustaining Results-Oriented Cultures
paragraphs: Successfully addressing the challenges that federal agencies face requires  leaders who are committed to achieving results, who recognize the  importance of using results-oriented goals and quantifiable measures, and  who integrate performance-based management into the culture and day-to- day activities of their organizations.  Top leadership must play a critical  role in creating and sustaining high-performing organizations.  Without the  clear and demonstrated commitment of agency top leadership—both  political and career—organizational cultures will not be transformed, and  new visions and ways of doing business will not take root.
To be positioned to address the array of challenges faced by our national  government, federal agencies will need to transform their organizational  cultures so that they are more results oriented, customer-focused, and  collaborative.  Leading public organizations here in the United States and  abroad have found that strategic human capital management must be the  centerpiece of any serious change management initiative and efforts to  transform the cultures of government agencies.  Performance management  systems are integral to strategic human capital management.  Such systems  can be key tools to maximizing performance by aligning institutional  performance measures with individual performance and creating a “line of  sight” between individual and organizational goals.  Leading organizations  use their performance management systems as a key tool for aligning  institutional, unit, and employee performance; achieving results;  accelerating change; managing the organization day to day; and facilitating  communication throughout the year so that discussions about individual  and organizational performance are integrated and ongoing.
Furthermore, achieving this cultural transformation requires people to  have the knowledge and skills to develop and use performance information  to improve program performance.  Our survey data indicated a significant  relationship between those managers who reported they received training  on setting performance goals and those who used performance information  when setting or revising performance goals.  However, federal agencies  have not consistently showed a commitment to investing in needed training  and development opportunities to help ensure that managers and  employees have the requisite skills and competencies to achieve agency  goals.
The commitment to focusing on and using performance information needs  to extend to OMB and Congress as well.  Through the administration’s PMA  and PART initiatives, OMB has clearly placed greater emphasis on  management issues over the past several years.  However, the focus of such  oversight needs to extend beyond the emphasis on formulating the  President’s Budget to include an examination of the many challenges  agencies face that may be contributing to poor performance.  In spite of the  persistent weaknesses we found in agencies’ strategic plans and annual  performance plans and reports, OMB significantly reduced the scope of its  guidance to agencies on how to prepare these documents.  By emphasizing  a focus on resource allocation through its PART exercise and providing less  information on how to comply with GPRA, OMB may be sending a message  to agencies that compliance with GPRA is not important.  Without strong  leadership from OMB, the foundation of performance information that has  been built could deteriorate.
OMB leadership is critical to addressing the continuing challenges  presented in GPRA implementation and the transformation of the federal  government to an increasingly results-oriented culture.  OMB, as the  primary focal point for overall management in the federal government, can  provide the needed impetus by providing guidance, fostering  communication among agencies, and forming intragovernmental councils  and work groups tasked with identifying potential approaches and  solutions to overcoming the persistent challenges to results-oriented  management.
Congress can also play a decisive role in fostering results-oriented cultures  in the federal government by using information on agency goals and results  at confirmation, oversight, authorization, and appropriation hearings.   Consistent congressional interest in the status of an agency’s GPRA efforts,  performance measures, and uses of performance information to make  decisions, will send an unmistakable message to agencies that Congress  expects GPRA to be thoroughly implemented.
We also found that timing issues may affect the development of agency  strategic plans that are meaningful and useful to top leadership.  The  commitment of top leadership within agencies, OMB, and Congress is  critical to the success of strategic planning efforts.  A strategic plan should  reflect the policy priorities of an organization’s leaders and the input of key  stakeholders if it is to be an effective management tool.  However, GPRA  specifies time frames for updating strategic plans that do not correspond to  presidential or congressional terms.  As a result, an agency may be required  to update its strategic plan a year before a presidential election and without  input from a new Congress.  If a new president is elected, the updated plan  is essentially moot and agencies must spend additional time and effort  revising it to reflect new priorities.  Our focus group participants, including  GPRA experts, strongly agreed that this timing issue should be addressed  by adjusting time frames to correspond better with presidential and  congressional terms.
section_title: Addressing Governmentwide Needs
paragraphs: Mission fragmentation and program overlap are widespread throughout the  federal government.  We have noted that interagency coordination is  important for ensuring that crosscutting program efforts are mutually  reinforcing and efficiently implemented.  Our review of six agencies’  strategic and annual performance plans along with our previous work on  crosscutting issues has demonstrated that agencies’ still present their goals  and strategies in a mostly stovepiped manner.  They have generally not  used their plans to communicate the nature of their coordination with  other agencies, in terms of the development of common or complementary  goals and objectives or strategies jointly undertaken to achieve those goals.
We have also reported that GPRA could provide a tool to reexamine federal  government roles and structures governmentwide.  GPRA requires the  President to include in his annual budget submission a federal government  performance plan.  Congress intended that this plan provide a “single  cohesive picture of the annual performance goals for the fiscal year.”  The  governmentwide performance plan could help Congress and the executive  branch address critical federal performance and management issues,  including redundancy and other inefficiencies in how we do business.  It  could also provide a framework for any restructuring efforts.   Unfortunately, this provision has not been fully implemented.  Instead,  OMB has used the President’s Budget to present high-level information  about agencies and certain program performance issues.  The agency-by- agency focus of the budget does not provide the integrated perspective of  government performance envisioned by GPRA.
If the governmentwide performance plan were fully implemented, it could  provide a framework for such congressional oversight.  For example, in  recent years, OMB has begun to develop common measures for similar  programs, such as job training.  By focusing on broad goals and objectives,  oversight could more effectively cut across organization, program, and  other traditional boundaries.  Such oversight might also cut across existing  committee boundaries, which suggests that Congress may benefit from  using specialized mechanisms to perform oversight (i.e., joint hearings and  special committees).
A strategic plan for the federal government, supported by key national  indicators to assess the government’s performance, position, and progress,  could provide an additional tool for governmentwide reexamination of  existing programs, as well as proposals for new programs.  If fully  developed, a governmentwide strategic plan can potentially provide a  cohesive perspective on the long-term goals of the federal government and  provide a much needed basis for fully integrating, rather than merely  coordinating, a wide array of federal activities.  Successful strategic  planning requires the involvement of key stakeholders.  Thus, it could serve  as a mechanism for building consensus.  Further, it could provide a vehicle  for the President to articulate long-term goals and a road map for achieving  them.  In addition, a strategic plan can provide a more comprehensive  framework for considering organizational changes and making resource  decisions.
Developing a strategic plan for the federal government would be an  important first step in articulating the role, goals, and objectives of the  federal government.  It could help provide critical horizontal and vertical  linkages.  Horizontally, it could integrate and foster synergies among  components of the federal government as well as help to clarify the role of  the federal government vis-à-vis other sectors of our society.  Vertically, it  could provide a framework of federal missions and goals within which  individual federal agencies could align their own missions and goals that  would cascade down to individual employees.  The development of a set of  key national indicators could be used as a basis to inform the development  of governmentwide strategic and annual performance plans.  The  indicators could also link to and provide information to support outcome- oriented goals and objectives in agency-level strategic and annual  performance plans.
section_title: Improving Usefulness of Performance Information
paragraphs: We have found that leading organizations that progressed the farthest to  results-oriented management did not stop after strategic planning and  performance measurement.  They applied their acquired knowledge and  data to identify gaps in their performance, report on that performance, and  finally use that information to improve their performance to better support  their missions.
Performance data can have real value only if they are used to identify the  gap between an organization’s actual performance level and the  performance level it has identified as its goal.  Once the performance gaps  are identified for different program areas, managers can determine where  to target their resources to improve overall mission accomplishment.   When managers are forced to reduce their resources, the same analysis can  help them target reductions to keep to a minimum the impact on their  organization’s overall mission.
Under GPRA, agencies produce a single strategic plan, annual performance  plan, and annual performance report.  However, there are many potential  consumers of agencies’ performance information—Congress, the public,  and the agency itself.  One size need not fit all.  Clearly, an agency will need  more detailed information on its programs for operational purposes than  would be suitable for external audiences.  Of the six agencies’ performance  reports we reviewed, some of them provided useful summary tables or  information that provided overall snapshots of performance or highlighted  progress in achieving key goals.  Other reports that lacked such a summary  made it difficult to assess the progress achieved.
To improve the prospect that agency performance information will be  useful to and used by these different users, agencies need to consider the  different information needs and how to best tailor their performance  information to meet those needs.  For example, we have reported that,  although many information needs were met, congressional staff also  identified gaps in meeting their information needs.  Key to addressing  these information gaps was improving communication between  congressional staff and agency officials to help ensure that congressional  information needs are understood, and that, where feasible, arrangements  are made to meet them.  Improved two-way communication might also  make clear what information is and is not available, as well as what is  needed and what is not needed.  This might entail the preparation of  simplified and streamlined plans and reports for Congress and other  external users.
Another challenge that limits the usefulness of agency performance reports  is the lack of timely data on performance.  For the six performance reports  we reviewed we continued to observe a significant number of goals for  which performance data were unavailable.  Policy decisions made when  designing federal programs, particularly intergovernmental programs, may  make it difficult to collect timely and consistent national data.  In  administering programs that are the joint responsibility of state and local  governments, Congress and the executive branch continually balance the  competing objectives of collecting uniform program information to assess  performance with giving states and localities the flexibility needed to  effectively implement intergovernmental programs.
section_title: Improving Performance Measures and Data Quality
paragraphs: Another key challenge to achieving a governmentwide focus on results is  that of developing meaningful, outcome-oriented performance goals and  collecting performance data that can be used to assess results.   Performance measurement under GPRA is the ongoing monitoring and  reporting of program accomplishments, particularly progress toward  preestablished goals.  It tends to focus on regularly collected data on the  level and type of program activities, the direct products and services  delivered by the programs, and the results of those activities.  For programs  that have readily observable results or outcomes, performance  measurement may provide sufficient information to demonstrate program  results.  In some programs, however, outcomes are not quickly achieved or  readily observed, or their relationship to the program is uncertain.  In such  cases, more in-depth program evaluations may be needed, in addition to  performance measurement, to examine the extent to which a program is  achieving its objectives.
Given the difficult measurement challenges we have identified, it is all the  more important that agency strategic planning efforts include the  identification of the most critical evaluations that need to take place to  address those challenges.  However, our previous work has raised concerns  about the capacity of federal agencies to produce evaluations of program  effectiveness.  Few of the agencies we reviewed deployed the rigorous  research methods required to attribute changes underlying outcomes to  program activities.  Yet we have also seen how some agencies have  profitably drawn on systematic program evaluations to improve their  measurement of program performance or understanding of performance  and how it might be improved.  Our review of six agencies’ strategic plans  and performance reports in this report revealed weaknesses in their  discussions of program evaluation.  Most of the strategic plans lacked  critical information required by GPRA, such as a discussion of how  evaluations were used to establish strategic goals or a schedule of future  evaluations.  Furthermore, two of the six performance reports did not  summarize the results of program evaluations completed that year, as  required.
Our work has also identified substantial, long-standing limitations in  agencies’ abilities to produce credible data and identify performance  improvement opportunities that will not be quickly or easily resolved.   According to our review, five of six agencies’ annual performance plans  showed meaningful improvements in how they discussed the quality of  performance data.  However, only DOT’s performance plan and report  contained information that provided a full level of confidence in the  credibility of its performance data.  In particular, the plans and reports did  not always provide detailed information on how the agencies verified and  validated their performance data.
section_title: Recommendations for Executive Action
paragraphs: To provide a broader perspective and more cohesive picture of the federal  government’s goals and strategies to address issues that cut across  executive branch agencies, we recommend that the Director of OMB fully  implement GPRA’s requirement to develop a governmentwide performance  plan.
To achieve the greatest benefit from both GPRA and PART, we recommend  that the Director of OMB articulate and implement an integrated and  complementary relationship between the two.  GPRA is a broad legislative  framework that was designed to be consultative with Congress and other  stakeholders, and allows for varying uses of performance information.   PART looks through a particular lens for a particular use—the executive  budget formulation process.
To improve the quality of agencies’ strategic plans, annual performance  plans, and performance reports and help agencies meet the requirements of  GPRA, we recommend that the Director of OMB provide clearer and more  consistent guidance to executive branch agencies on how to implement  GPRA.  Such guidance should include standards for communicating key  performance information in concise as well as longer formats to better  meet the needs of external users who lack the time or expertise to analyze  lengthy, detailed documents.
To help address agencies’ performance measurement challenges, we  recommend that the Director of OMB engage in a continuing dialogue with  agencies about their performance measurement practices with a particular  focus on grant-making, research and development, and regulatory  functions to identify and replicate successful approaches agencies are  using to measure and report on their outcomes, including the use of  program evaluation tools.  Additionally, we recommend that the Director of  OMB work with executive branch agencies to identify the barriers to  obtaining timely data to show progress against performance goals and the  best ways to report information where there are unavoidable lags in data  availability.  Interagency councils, such as the President’s Management  Council and the Chief Financial Officers’ Council, may be effective vehicles  for working on these issues.
To facilitate the transformation of agencies’ management cultures to be  more results-oriented, we recommend that the Director of OMB work with  agencies to ensure they are making adequate investments in training on  performance planning and measurement, with a particular emphasis on  how to use performance information to improve program performance.
section_title: Matters for Congressional Consideration
paragraphs: To ensure that agency strategic plans more closely align with changes in  the federal government leadership, Congress should consider amending  GPRA to require that updates to agency strategic plans be submitted at  least once every 4 years, 12-18 months after a new administration begins its  term.  Additionally, consultations with congressional stakeholders should  be held at least once every new Congress and interim updates made to  strategic and performance plans as warranted.  Congress should consider  using these consultations along with its traditional oversight role and  legislation as opportunities to clarify its performance expectations for  agencies.  This process may provide an opportunity for Congress to  develop a more structured oversight agenda.
To provide a framework to identify long-term goals and strategies to  address issues that cut across federal agencies, Congress also should  consider amending GPRA to require the President to develop a  governmentwide strategic plan.
section_title: Agency Comments
paragraphs: We provided a copy of the draft report to OMB for comment.  OMB’s  written comments are reprinted in appendix VIII.  In general, OMB agreed  with our findings and conclusions.  OMB agreed to implement most of our  recommendations, noting that these recommendations will enhance its  efforts to make the government more results oriented.  OMB agreed to   (1) work with agencies to ensure they are provided adequate training in  performance management, (2) revise its guidance to clarify the integrated  and complementary relationship between GPRA and PART, and   (3) continue to use PART to improve agency performance measurement  practices and share those practices across government.
In response to our recommendation that OMB fully implement GPRA’s  requirement to develop a governmentwide performance plan, OMB stated  that the President’s Budget represents the executive branch’s  governmentwide performance plan.  However, the agency-by-agency focus  of the budget over the past few years does not provide an integrated  perspective of government performance, and thus does not meet GPRA’s  requirement to provide a “single cohesive picture of the annual  performance goals for the fiscal year.”  In response to our matter for  congressional consideration that Congress should consider amending  GPRA to require the President to develop a governmentwide strategic plan,  OMB noted that the budget serves as the governmentwide strategic plan.   However, in our opinion, the President’s Budget focuses on establishing  agency budgets for the upcoming fiscal year.  Unlike a strategic plan, it  provides neither a long-term nor an integrated perspective on the federal  government’s activities.  A governmentwide strategic plan should provide a  cohesive perspective on the long-term goals of the federal government and  provide a basis for fully integrating, rather than primarily coordinating, a  wide array of federal activities.
We provided relevant sections of the draft report to Education, DOE, HUD,  SBA, SSA, and DOT.  Education and SBA did not provide any comments,  while DOT provided minor technical comments.  Written comments from  DOE, HUD, and SSA are reprinted in appendixes IX, X, and XI, respectively,  along with our responses.
DOE disagreed with portions of our analyses of its 2004 Annual  Performance Plan and its 2002 Performance and Accountability Report.   Our analysis of DOE’s documents was based on specific criteria (see  appendixes IV and V for details) and was characterized in relation to our  reviews of the other five agencies’ documents.  We modified or clarified  certain characterizations in response to DOE comments, but for the most  part found that our characterizations were appropriate.
SSA generally agreed with our observations and agreed to incorporate  them in its future planning efforts.  SSA made several points of clarification  and disagreed with our observation that its performance and accountability  report does not clearly state how program evaluations were used to answer  questions about program performance and results and how they can be  improved.  SSA noted that its evaluations rely on surveys, and these  surveys form the basis for its efforts to deliver high-quality service.  SSA  also noted that it lists other evaluations that are of great importance to its  ongoing operations.  We do not discount the usefulness of SSA’s surveys in  assessing its day-to-day management of programs.  Rather, we believe that  it would be helpful for SSA to clearly identify the range of evaluations  conducted and how each of them contributed to improved program  performance.
HUD noted that all of the areas we suggested for further improvement are  under consideration for improvement.  However, they disagreed with us on  two observations related to the strategic plan:  (1) that the link between its  long-term and intermediate goals is difficult to discern and (2) that it did  not explain how it used the results of program evaluations to update the  current plan and did not include a schedule for future evaluations.  On the  basis of OMB guidance for preparing strategic plans and the criteria we  used to evaluate all six agencies’ strategic plans (see app. III for more  detail), we maintain that these two observations are valid and require  further attention.  HUD also disagreed with how we presented the  performance information in its summary report cards (see fig. 22).  HUD  noted that many of the results were explained in the individual indicator  write-ups that followed the summary information.  Our analysis of HUD’s  information included qualitative aspects of how the information was  presented, such as its usefulness to inform the average reader with little or  no exposure to the subject matter, and the extent to which HUD presented  a complete summary of performance information in a user-friendly format.
Technical comments from DOE, HUD, and SSA were incorporated, as  appropriate.