Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies
Published: 1996-07-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO discussed the challenges in implementing the Government Performance and Results Act (GPRA) in federal civilian science agencies, focusing on ways to improve the management of federal civilian science agencies and aid congressional decisionmaking.
section_title: What GAO Found
paragraphs: GAO noted that: (1) GPRA enables Congress to make difficult funding, policy, and programming decisions relative to science agencies' demands; (2) the science agencies are required to set strategic and annual goals, measure performance rates, and report their goals attainment; (3) GPRA has shifted the focus of federal management and accountability from staffing and activity levels to federal program outcomes; (4) GPRA is being phased in through 70 pilot projects, to provide agencies with guidance in meeting GPRA requirements; (5) the successful implementation of this results-oriented management approach depends on defining missions and desired outcomes, measuring program performance, and using performance data to improve program operations; (6) measuring program performance can be extremely difficult because of the wide range of factors to consider in determining a project's commercial viability; and (7) most agencies are struggling to integrate GPRA goal-setting and performance measurement requirements into their daily program operations.

section_title: Letter
paragraphs: The commitment to reduce the deficit is forcing Congress and the executive branch to undertake a basic reexamination of the value of programs across the federal government and is placing pressure as never before on all federal agencies, including the civilian science agencies, to clearly demonstrate that they are making effective use of taxpayers’ dollars.
The landmark Government Performance and Results Act (GPRA) provides a legislative vehicle for Congress and agencies to use to improve program effectiveness and make the difficult trade-offs that the current budget environment demands. GAO’s recently released Executive Guide: Effectively Implementing the Government Performance and Results Act discusses three key steps for federal agencies to successfully implement GPRA. These three steps are (1) define mission and desired outcomes, (2) measure performance, and (3) use performance information. The experiences of civilian science agencies suggest that each one of these steps offers important opportunities for improving agency management and congressional decisionmaking.
GAO work has shown that the effectiveness of the Department of Energy, the National Aeronautics and Space Administration, and a host of other agencies has been hampered by unfocused missions and unclear goals. GPRA is intended to provide this focus, in part by requiring that agencies develop strategic plans based on consultation with Congress and other stakeholders. These consultations are an important opportunity for Congress and the executive branch to jointly reassess and clarify the agencies’ missions and desired outcomes.
Measuring the performance of science-related projects can be extremely difficult because a wide range of factors determine if and how a particular research and development (R&D) project will result in a commercial application or have other benefits. It can also take many years between when a research project is undertaken and when the outcome occurs. Due to the difficulties in measuring performance, R&D agencies typically have chosen to measure a variety of proxies for outcomes, such as the number of patents resulting from the federally funded research and expert review and judgments of the quality and importance of research findings. operations. This integration is important because GPRA performance information is to be used to guide an array of congressional and executive branch decisions. Consistent congressional interest—in hearings such as today’s, for example—on the status of an agency’s GPRA efforts, the performance measures it is using, and how performance information is being used to make decisions will send an unmistakable message to agencies that Congress expects GPRA to be conscientiously implemented.
subsections: section_title: Managing for Results: Key Steps and Challenges in Implementing GPRA in Science Agencies
paragraphs: Mr. Chairman and Members of the Committee: We are pleased to have this opportunity to discuss the key steps and challenges in using the Government Performance and Results Act (GPRA) to improve federal civilian science agencies’ management and congressional decisionmaking.
Each year, American taxpayers invest about $70 billion of federal funds in military and civilian research and development (R&D) efforts. Over the years, this investment has yielded substantial benefits to the health, welfare, and security of the American people. Yet the powerful pressures to reduce the deficit are forcing Congress and the executive branch to undertake a basic reexamination of the value of programs across the federal government and to update the nation’s spending priorities. The effort to reduce the deficit is therefore placing pressure as never before on all federal agencies, including the civilian science agencies, to clearly demonstrate that they are making sound and effective use of taxpayers’ dollars.
Fortunately, the landmark GPRA provides a legislative vehicle for agencies to use as they seek to demonstrate and improve their effectiveness. Equally important, if successfully implemented, GPRA should help Congress make the difficult funding, policy, and program decisions that the current budget environment demands. Under GPRA, agencies are to set strategic and annual goals, measure performance, and report on the degree to which goals are met. Congress intended for GPRA to fundamentally shift the focus of federal management and accountability from a preoccupation with staffing and activity levels to a focus on “outcomes” of federal programs. Outcomes are results expressed in terms of the difference federal programs make in people’s lives.
In crafting GPRA, Congress recognized that the types of management changes that successful implementation will require will not come quickly or easily for many agencies. As a result, GPRA is being phased in initially through almost 70 pilot projects during fiscal years 1994 through 1996 to provide agencies with experience in meeting its requirements before governmentwide implementation in the fall of 1997. Several agencies with a major civilian science focus, such as the Department of Energy (DOE), the Environment Protection Agency, the National Oceanic and Atmospheric Administration, and the National Science Foundation (NSF), have programs that are included in the pilot phase of GPRA.
Our recently released Executive Guide: Effectively Implementing the Government Performance and Results Act, which, at the request of the Committee, we are providing for the record, is intended to help federal managers implement GPRA and make the difficult transition to a form of management and accountability that stresses outcomes. In the guide, we discuss three key steps for federal agencies to successfully implement results-oriented management. These three steps are (1) define mission and desired outcomes, (2) measure performance, and (3) use performance information. The guide shows the relationship of these steps to GPRA and highlights important practices associated with each step. The guide also discusses the role of top leadership in implementing GPRA and the practices leaders can follow to make GPRA a driving force in federal decisionmaking. Accompanying the discussion of each practice is a case illustration describing a federal agency that has made progress incorporating the practice into its operations. The practices discussed in the guide emerged from the experiences of leading public organizations here and abroad and have been shown to be effective in the federal management environment. These practices provide a useful framework for agencies working to implement GPRA and for assessing their progress.
Our comments today use that framework to underscore the opportunities and challenges to using GPRA as a vehicle that agencies and Congress can employ as they seek to improve the effectiveness and efficiency of civilian science programs. Our comments are based on completed and ongoing reviews of efforts to implement GPRA in pilot and nonpilot agencies across the federal government and of the management of civilian science agencies.
subsections: section_title: Step 1: Define Mission and Desired Outcomes
paragraphs: Our work has shown that all too frequently individual agencies have lacked clear missions and goals, and related agencies’ efforts have not been complementary. Moreover, legislative mandates may be unclear and Congress, the executive branch, and other stakeholders may not agree on the goals an agency and its programs should be trying to achieve, the strategies for achieving those goals, and the ways to measure their success. Thus, many agencies cannot confidently answer the basic questions in defining a mission—what is our purpose, whom do we serve, and how do we meet our mission? GPRA seeks to address these problems by requiring executive branch agencies to develop strategic plans that are to define missions and articulate strategic goals.
Although statutory requirements are to be the starting point for agency mission statements, agencies are to consult with Congress and other stakeholders in defining their missions. In the case of Congress, this may entail identifying legislative changes that are needed either to clarify Congress’ intent and expectations or to address differing conditions and citizens’ needs that have arisen since initial statutory requirements were established. Congressional consultation may also involve obtaining guidance on Congress’ priorities in those frequent cases where agencies have more than one statutory mission.
subsections: section_title: Clarifying Agencies’ Missions
paragraphs: These consultations are an important opportunity for Congress and the executive branch to work together in reassessing and clarifying the missions of federal agencies and the desired outcomes of agencies’ programs. For example, our work has shown that DOE sorely needs a reevaluation of its basic mission, including its significant science responsibilities. DOE’s mission and priorities have changed dramatically over time so that DOE is now very different from what it was in 1977 when it was created in response to the nation’s energy crisis. While energy research, conservation, and policymaking dominated early DOE priorities, national defense and environmental clean-up now overshadow those efforts. Meanwhile, new mission areas in science and industrial competitiveness, such as applied R&D programs supporting technology to secure future energy supplies, have emerged and are pressing for priority attention. Each new phase in DOE’s evolution has been accompanied by new leadership with vastly different agendas concerning DOE’s basic mission and how it should be managed. the collective strengths of the laboratories to meet pressing national needs. As a result, DOE was unable to address issues that required cooperation and coordination across its many mission areas. For example, we reported that although solutions to the proliferation of nuclear weapons require expertise in identifying the effects of weapons, the research on nonproliferation and effects of weapons was carried out in different laboratories and was managed by different assistant secretaries. Overall, laboratory managers said that they feared that the lack of proper departmental direction was compromising both their effectiveness and their ability to respond to new national priorities.
Planning-related problems were not unique to DOE. As another example, our past work at the National Aeronautics and Space Administration (NASA) showed the need for NASA to develop a strategic plan that realistically matched its program plans to its likely budgets. NASA strategic planning efforts in the early 1990s were incomplete and unrealistic because they did not indicate the relative priority of NASA’s key missions and large programs and provided no balance between planning and budgeting. For example, NASA’s failure to ground its goals in realistic budget expectations forced it to make significant program adjustments to make up for the lower-than-planned funding levels. Over the past several years we have reported that space shuttle operations, the space station, the Advanced X-Ray Astrophysics Facility, the Earth Observing System, and other NASA programs and projects had been or were being restructured primarily because they were not deemed affordable.
Both DOE and NASA are undertaking strategic planning efforts intended in part to address these longstanding problems. Sustained congressional involvement in these efforts and similar planning efforts undertaken by other agencies are vital to ensuring that missions are focused, goals are clearly established, and strategies and funding expectations are appropriate and reasonable. The experiences of leading organizations suggest that planning efforts that have such characteristics can become driving forces in improving the effectiveness and efficiency of program efforts. The GPRA strategic planning process thus provides Congress with a potentially powerful vehicle for clarifying its expectations for agencies and the program results expected from funding decisions.
section_title: Identifying and Addressing Crosscutting Issues
paragraphs: programs and aid congressional decisionmaking as it relates to the individual agencies. Equally important, we believe, is congressional involvement in strategic planning for program efforts that cut across several agencies. This is particularly true for science programs where, in many cases, the greatest return for the federal dollar can come through the coordinated efforts of a number of agencies. Not surprisingly, given the planning problems that individual agencies have confronted, program efforts that involve several agencies also have suffered from a lack of focus and unclear goals.
For example, we also have reported that the multiagency High Performance Computing and Communication (HPCC) initiative could benefit from a strengthened program direction. Our work showed that a more focused management approach and an identification of priority areas were needed to help ensure that the program’s goals were met. The HPCC program was originally established, by design, as a loosely coordinated, scientifically oriented research effort rather than a rigorously managed development program. Once the administration expanded the role of HPCC to support the national information infrastructure, a more rigorous and coordinated management approach was required that better targeted the specific technology areas that most needed to be developed to support the information superhighway. The HPCC effort at NSF, an agency with a major role in the program, is one of NSF’s four GPRA pilot projects. NSF is using GPRA as a vehicle for clarifying the long-term goals for its HPCC program and developing performance measures to gauge progress. grants, a wide range of R&D performed by businesses, universities, and other organizations.
The Office of Management and Budget (OMB) recognizes the key role that the development of strategic plans under GPRA can have in helping to ensure federal efforts across agencies are properly coordinated. OMB is now undertaking a “Summer Review” during which it is examining the progress that agencies are making in meeting the GPRA strategic planning requirements. As part of that review, OMB is seeking to identify any steps that should be taken on a multiagency basis to coordinate and harmonize goals and objectives for cross-agency programs and functions. OMB’s efforts under GPRA should assist Congress as it looks for opportunities to streamline and improve the effectiveness of the federal government.
section_title: Step 2: Measure Performance
paragraphs: GPRA requires executive agencies to develop annual plans with suitable performance measures to reinforce the connection between the long-term strategic goals outlined in their strategic plans and the day-to-day activities of their managers and staff. Measuring performance allows an organization to track its progress toward its goals and gives managers important information on which to base their organizational and management decisions. At a broader level, measuring R&D agencies’ performance helps Congress to know the results of R&D investments and to effectively allocate budgets among competing programs.
Science agencies, like other agencies, must guard against the understandable tendency to overly rely on goals and measures that are easily quantifiable, such as numbers of research grants provided and completed, at the expense of what is truly important but more difficult to measure, such as the difference a research grant made. Organizations that measure and manage on the basis of easily quantifiable goals rather than results run the risk of striving to achieve goals that may be only marginally related to the reasons the program was created. For example, we recently reviewed the short-term performance results and long-term evaluation strategy of the Advanced Technology Program (ATP). ATP is administered by the National Institute of Standards and Technology (NIST) within the Department of Commerce and has had its funding grow from $68 million in fiscal year 1993 to $431 million in fiscal year 1995, more than doubling each year.
NIST identified several evaluation measures that it expects will indicate the long-term economic success of ATP projects. One NIST measure is “straightforward tracking of technical milestones.” However, we found that achieving technical milestones may not be a valid indicator of the economic success of ATP projects because achieving technical milestones does not always lead to economic success. For example, earlier versions of the ATP evaluation plan pointed to one ATP project that was achieving all of its technical milestones as evidence of the project’s likely success in stimulating economic growth. However, the lead company involved in this joint venture went bankrupt before the project was completed. Although the other company in the joint venture has stated its intention to continue the joint venture commercialization plan, the lead company’s bankruptcy reduces the likelihood of future economic benefits being realized from this ATP project.
subsections: section_title: The Challenge to Measuring Science Programs’ Outcomes
paragraphs: The tendency to focus on what is relatively easy to measure may be particularly strong for science agencies because the selection and use of performance measures have presented a longstanding challenge for such agencies, especially for those that support fundamental scientific research. In particular, assessing the outcomes of science-related projects can be extremely difficult because a wide range of factors determine if and how a particular R&D project will result in commercial or other benefits. It can also take many years between when a research project is undertaken and when the outcome occurs. For example, the National Institutes of Health and other federal institutions support research at universities by investing in (1) the development of principal investigators, who work at the forefront of scientific and engineering research and (2) the training of new PhDs and other professionals. Hence, these people are “products” of the research projects the agencies support. However, the outcomes—in terms of scientific contributions—produced by these individuals as a result of the federal agencies’ investments in their development and training frequently extend well beyond a specific federally funded research project and can be exceedingly difficult to measure. publications or citations, and contributions to expanding the number of research scientists.
To help address the challenges of measuring the results of R&D programs, the Research Roundtable, a consortium of federal agency representatives, has been meeting periodically to share ideas and approaches for implementing GPRA. The Roundtable has been considering the extent to which R&D agencies can and should adopt a common approach to measuring performance. It is still too early to tell whether this group will define and recommend a common model to meet the performance requirements under GPRA. Nonetheless, the Roundtable’s efforts are promising in that they show that officials in science agencies recognize the performance measurement challenges they confront and are working collectively to address those challenges. Congress can support the Roundtable’s efforts by working with it to ensure that congressional data needs are met by any common performance measurement model that the Roundtable may recommend.
GPRA recognizes how difficult it is to state the goals and measure the results of some programs. While the law encourages the use of objective measures of performance, it authorizes agencies—with the approval of OMB—to use alternative, subjective measures of performance. Congress expects that one form of alternative measurement will be to define the characteristics of a marginally effective program and a fully successful program and to assess progress against those definitions. NSF is seeking OMB approval to use an alternative format for articulating its performance goals under GPRA. This request, and any additional ones, to employ an alternative form of measurement may become an important indicator of the difficulty of fully achieving GPRA’s design in science programs.
section_title: Step 3: Use Performance Information
paragraphs: As to be expected during the initial efforts of such a challenging management reform effort as GPRA, most agencies, including science agencies, are still struggling to integrate the mission-based goal-setting and performance measurement requirements of GPRA into their daily program operations. This integration is important because GPRA performance information is to be used to guide an array of congressional and executive branch decisions. being used to make decisions will send an unmistakable message to agencies that Congress expects GPRA to be implemented as conscientiously as possible. Congressional hearings, such as the one the Committee is holding today, are one key to showing agencies that Congress is looking to use GPRA performance goals and information to help inform its decisions and that it expects agencies to do the same. Moreover, congressional engagement in the strategic planning efforts now under way in executive agencies provides an excellent opportunity to clarify agencies’ missions and goals and ensure that the resulting performance information will meet congressional and other decisionmakers’ needs.
In summary, if successfully implemented in the science agencies, GPRA should help the Committee make the difficult science policy and program decisions confronting the nation. It also will help science agencies manage their programs and provide Congress and the American people with better assurance that tax dollars are being wisely spent. But the changes in management and accountability envisioned by Congress in passing GPRA are not coming quickly or easily, particularly in science agencies. The continued support and interest of this Committee may well determine the degree to which GPRA is successfully implemented in those agencies.
section_title: Related GAO Products
paragraphs: Executive Guide: Effectively Implementing the Government Performance and Results Act (GAO/GGD-96-118, June 1996).
Managing for Results: Achieving GPRA’s Objectives Requires Strong Congressional Role (GAO/T-GGD-96-79, Mar. 6, 1996).
Federal R&D Laboratories (GAO/RCED/NSIAD-96-78R, Feb. 29, 1996).
GPRA Performance Reports (GAO/GGD-96-66R, Feb. 14, 1996).
Office of Management and Budget: Changes Resulting From the OMB 2000 Reorganization (GAO/GGD/AIMD-96-50, Dec. 29, 1995).
Department of Energy: A Framework For Restructuring DOE and Its Missions (GAO/RCED-95-197, Aug. 21, 1995).
Managing for Results: Status of the Government Performance and Results Act (GAO/T-GGD-95-193, June 27, 1995).
Department of Energy: Framework Is Needed to Reevaluate Its Role and Missions (GAO/T-RCED-95-232, June 21, 1995).
Managing for Results: Critical Actions for Measuring Performance (GAO/T-GGD/AIMD-95-187, June 20, 1995).
Government Reorganization: Issues and Principles (GAO/T-GGD/AIMD-95-166, May 17, 1995).
Performance Measurement: Efforts to Evaluate the Advanced Technology Program (GAO/RCED-95-68, May 5, 1996).
Managing for Results: Experiences Abroad Suggest Insights for Federal Management Reforms (GAO/GGD-95-120, May 2, 1995).
Department of Energy: Alternatives for Clearer Missions and Better Management at the National Laboratories (GAO/T-RCED-95-128, Mar. 9, 1995).
Department of Energy: Research and Agency Missions Need Reevaluation (GAO/T-RCED-95-105, Feb. 13, 1995).
Department of Energy: National Laboratories Need Mission Focus and More Effective Departmental Management (GAO/RCED-95-10, Jan. 27, 1995).
Department of Energy: Need to Reevaluate Its Role and Missions (GAO/T-RCED-95-85, Jan. 18, 1995).
DOE’s National Laboratories: Adopting New Missions and Managing Effectively Pose Significant Challenges (GAO/T-RCED-94-113, Feb. 3, 1994).
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.