GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

United States General Accounting Office

Washington, DC 20548

Resources, Community, and

Economic Development Division

B-286003

August 25, 2000

The Honorable F. James Sensenbrenner

Chairman, Committee on Science

House of Representatives

Subject: Government Performance and Results Act: Information on Science Issues

in the Department of Energy’s Accountability Report for Fiscal Year 1999

and Performance Plans for Fiscal Years 2000 and 2001

As you requested, we reviewed the Department of Energy’s (DOE) accountability

report for fiscal year 1999 and performance plans for fiscal years 2000 and 2001,

required by the Government Performance and Results Act (GPRA) of 1993. In

essence, under GPRA, annual performance plans are to establish performance goals

and measures covering a given fiscal year and provide direct linkage between an

agency’s long-term goals and day-to-day activities. Annual performance reports are

to subsequently report on the degree to which those performance goals were met.

Specifically, we reviewed the portions of DOE’s accountability (performance) report

and performance plans that pertain to its science programs. As agreed with your

office, we examined whether DOE’s accountability report for fiscal year 1999 and

performance plans for fiscal years 2000 and 2001 (1) convey a coherent picture of the

Department’s science programs and (2) clearly link to its budget. In addition, you

asked us to review how DOE has addressed management weaknesses that we had

identified in previous reports. This report provides our observations on these issues.

Enclosure I provides specific examples from the accountability report and

performance plans that illustrate our observations.

Results in Brief

DOE’s accountability report for fiscal year 1999 and performance plans for fiscal

years 2000 and 2001 do not convey a clear picture of the Department’s science

programs because they are not consistent over time. First, when compared with the

accountability report and 2000 performance plan, DOE’s fiscal year 2001 performance

plan made significant changes to its business line concerning science, such as

rewriting the strategic goal for science, without explaining the reasons for the

changes or their impact on DOE’s focus. Second, DOE significantly changed all of
B-286003

2 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

the strategic objectives supporting this new business line, making it difficult to

clearly track the focus of DOE’s science efforts. Without an explanation for the

change, it is difficult to determine whether DOE intends to continue its previous

work or refocus its efforts. For example, one objective DOE included in its fiscal

year 2000 performance plan, “improve the management of DOE’s research enterprise

to enhance the delivery of leading-edge science and technology at reduced costs,”

was dropped from its fiscal year 2001 plan and was not replaced with a related

objective. DOE did not provide a reason for this change. Last, many of the annual

performance goals and measures DOE used in its accountability report and

performance plans did not sufficiently describe the Department’s specific science

activities; explain how these activities will help achieve the overall goal for the

science business line; or, when the activities are completed, indicate how much

progress will have been made toward reaching the goal. As a result, it was difficult to

determine what DOE was trying to accomplish in science or how it planned to get

there.

DOE’s accountability report for fiscal year 1999 and performance plans for fiscal

years 2000 and 2001 provided clear links to the budget. DOE’s 2001 performance plan

tried to strengthen this linkage by presenting performance measures by budgetary

decision unit rather than by objective and performance goal. However, in changing

the format of the plan, DOE did not provide a clear crosswalk with its plans in

previous years, making it hard to find specific measures in the latest plan or to track

measures across years. Furthermore, the fiscal year 2001 performance plan does not

specifically cite the performance goals presented in the accountability report and the

2000 performance plan, making it harder to compare the two plans. DOE provides a

code that links performance measures in the 2001 plan with their performance goals

from the 2000 plan, but this coding system is difficult to follow.

Although the accountability report for fiscal year 1999 and the performance plans for

fiscal years 2000 and 2001 address weaknesses we previously identified, the

documents do not indicate enough progress for us to believe that the weaknesses

have been resolved. For example, DOE provided measures to address one science–

related management challenge—completing large projects—but these measures have

focused on procedures rather than on achieving the desired outcome--completing

large projects on time and within cost estimates. The quality of information on

performance is also limited because the accountability report does not explain how

DOE verified and validated the results tied to each performance measure.

Accountability Report and Performance Plans Do Not Convey a Coherent

Story About DOE’s Science Programs

DOE’s accountability report for fiscal year 1999 and performance plans for fiscal

years 2000 and 2001 do not tell a coherent story about the Department’s science

programs because they are not consistent over time. When compared with the

accountability report and 2000 performance plan, the 2001 performance plan made

significant changes to its business line concerning science. In its 2001 performance

plan, DOE substantially rewrote its strategic goal for the new business line without
B-286003

3 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

accounting for the change or indicating where items related to technology reside in

the plan.

Similarly, DOE significantly changed all the objectives that support the science

business line. From its initial strategic plan in fiscal year 1997, through its fiscal year

1999 accountability report and fiscal year 2000 performance plan, DOE cited four

strategic objectives for the business line concerning science. These objectives are

listed below:

• “Develop the science that underlies DOE’s long-term mission.

• Deliver leading-edge technologies that are critical to the DOE mission and the

Nation.

• Improve the management of DOE’s research enterprise to enhance the delivery of

leading-edge science and technology at reduced costs.

• Assist in the government-wide effort to advance the Nation’s science education

and literacy.”

In the fiscal year 2001 performance plan, these objectives were replaced in the

revised business line with a new set of objectives:

• “Fuel the future with science for clean and affordable energy.

• Protect our living planet with scientific understanding of energy impacts on

people and the biosphere.

• Explore matter and energy as elementary building blocks from atoms to life.

• Provide the extraordinary tools, scientific workforce, and infrastructure that

assure our Nation’s leadership in the physical, biological, and computational

sciences and in multidisciplinary research.”

These new objectives suggest a change in DOE’s focus for science activities.

However, since DOE did not explain why it changed the objectives, it is difficult to

determine whether the Department is refocusing its emphasis in science or just

reorganizing the information in the plan. For example, since DOE does not explain

why the strategic objective “improve the management of DOE’s research enterprise

to enhance the delivery of leading-edge science and technology at reduced cost” was

removed, it is unclear what emphasis, if any, the Department is giving this issue.

At a more detailed level, many of DOE’s annual performance goals and measures in

the accountability report and performance plans are not clear and raise questions

about what is being accomplished to meet the objectives and goal in science. For

example, one of DOE’s performance goals in its accountability report is to “Develop

the technologies required to meet DOE’s energy, national security, and environmental

quality goals.” This description provides no details by itself about how DOE will

develop the technologies or the energy, national security, and environmental quality

goals that the technologies will try to meet. In addition, because “developing” lacks a

specific target, almost any DOE activity could be interpreted as meeting the goal.
B-286003

4 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

Furthermore, DOE describes many of its science-related performance measures in

ways that make it difficult to determine how much closer the Department will be to

accomplishing an objective and goal when a measure is met. For example, to help

achieve one of its goals, “Developing science to support DOE’s participation in energy

and other national policy formulations,” DOE states in its fiscal year 2000

performance plan that it will “Continue collaborative efforts with the NASA [National

Aeronautic and Space Administration] on space science and exploration.” The

measure does not specifically indicate how and to what end DOE will collaborate

with NASA. As a result, it is difficult to tell, if the performance measure is achieved,

how far DOE will be towards achieving its goal. In general, because DOE’s

performance goals and measures lack clarity, it was difficult to determine what the

Department was trying to accomplish in science or how it planned to get there.

In addition, we question whether some measures in the plan are measures or merely

evaluation activities that would normally be performed to assess individual projects

or a program as a whole. For example, in examining the 2000 performance plan, we

found several measures that appear to be evaluation activities. Among them are the

following:

“Receive from the Program Advisory Committees [PACS], an

assessment of the quality of research and program relevance at

major Fusion Energy operating facilities.”

“Conduct regular peer review and merit evaluation based on the

principles set down in 10 CFR Part 605 for grants and cooperative

agreements, with all research projects reviewed at least once and no

project extending beyond more than four years without review.”

These activities do not further DOE’s particular goals and objectives for science.

Rather, the results of these activities would contribute to an evaluation of program

areas. DOE could improve its performance plans by presenting these activities as

part of its overall program evaluation efforts rather than as specific actions to be

performed in the plan.

In instances like these, DOE did not follow its own criteria (stated in app. A of its

fiscal year 2001 performance plan) that performance measures and goals should,

among other things, precisely state what will be done in the fiscal year, clearly state

the measurement and target level of performance, and use baselines for context.

Fiscal Year 2001 Plan Provides More Direct Links to Budget Measures but

Loses Links With Previous Plans

DOE’s accountability report for fiscal year 1999 and performance plans for fiscal

years 2000 and 2001 provided clear links to the budget. In the 1999 report, DOE

presented the total budget authority and human resources allocated to each of the

Department’s five business lines, including the business line concerning science. In

the 2000 plan, DOE provides the fiscal year 2000 budget request for each program

office working to achieve specific objectives in the plan. DOE’s 2001 performance
B-286003

5 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

plan tried to strengthen these linkages by presenting performance measures by

budgetary decision unit rather than by objective and performance goal.

However, in the new format, DOE did not provide a clear crosswalk with its plans in

previous years, making it difficult to find specific performance measures in the latest

plan. For example, in the accountability report and fiscal year 2000 performance

plan, the performance measure “Supply quality stable and radioactive isotopes for

industrial, research, and medical applications that continue to meet customer

specifications and maintain 95 percent on-time deliveries” is located in the “Science

and Technology” business line. In the fiscal year 2001 performance plan, DOE places

this measure in the “Energy Resources” business line without providing any

explanation for the move. Without a crosswalk, the reader can only find the measure

by examining the entire plan.

In addition, while trying to strengthen the links between performance measures and

budgetary decision units, the 2001 plan does not specifically cite the performance

goals presented in the accountability report and the 2000 performance plan. Rather,

the plan paraphrases parts of the goals in discussions describing the decision units.

This approach loses the direct link between specific goals and annual performance

measures that was clear in DOE’s earlier documents. The plan also raises the

question of whether the specific performance measures still exist in the 2001

performance plan. For example, one of DOE’s performance measures concerns

delivering on the “U.S./DOE Large Hadron Collider Project.” In the accountability

report and 2000 performance plan, this measure supports the performance goal

“Develop the science that underlies DOE’s long-term mission.” Although the measure

is also included in the 2001 performance plan in science, a performance goal is not

presented. As a result of this inconsistency, it is difficult to track the progress DOE

has made in its performance goals and measures across years.

Although DOE does not include the performance goals it presented in previous plans,

it provides a code that links certain performance measures in the 2001 plan with their

performance goals from the 2000 performance plan. However, this coding system is

difficult to follow and requires the reader to examine the performance plans from

both years in order to make sense of the system.

Accountability Report and Performance Plan Do Not Adequately Address

Management Weaknesses

DOE’s accountability report and fiscal year 2001 performance plan attempt to address

the managerial weaknesses that we previously identified. In particular, the 2001 plan

includes a number of measures that, while not specifically contained in the Science

business line, attempt to address management weaknesses or challenges, such as

“Difficulty Completing Large Projects,” that involve science programs. In our

opinion, neither the accountability report nor the 2001 performance plan has or will

produce enough progress for us to believe the challenges have been resolved. For

example, DOE has taken a number of steps, such as creating the Chief Operating

Officer’s Project Management Watch List, to help ensure that projects are completed

on time. While these steps increase the attention this challenge may receive from
B-286003

6 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

DOE, they focus on procedures rather than achieving the desired outcome—

completing large projects on time and within cost estimates.

In addressing another area of weakness that we identified—the quality of information

on performance—DOE’s accountability report also falls short. Specifically, the

accountability report does not provide a discussion on how DOE verified and

validated the results tied to each performance measure. The fiscal year 2001

performance plan contains an overview of DOE’s validation and verification process,

but provides no details or evidence to indicate that DOE’s data will be of sufficient

quality to assess the Department’s performance. As a result, we have little

confidence that the plan’s performance data are credible.

Scope and Methodology

To meet the Committee’s tight reporting time frames, our observations were

generally based on the requirements of GPRA, guidance to agencies from the Office

of Management and Budget (OMB) for developing performance plans and reports

(OMB Circular A-11, part 2), previous reports and evaluations by us and others, our

knowledge of DOE’s operations and programs, and our observations on DOE’s other

GPRA-related efforts. We did not independently verify the information contained in

the accountability report or plans. We conducted our review from June through

August 2000 in accordance with generally accepted government auditing standards.

Agency Comments and Our Evaluation

We provided a draft of this report to DOE for its review and comment. We discussed

the report with DOE officials, including the Director, Office of Strategic Planning and

Program Evaluation, and the Director, Office of Planning and Analysis, Office of

Science. DOE did not provide comments on the specifics of the report, primarily

because it believes our review revisits issues discussed in our previous reports

concerning the Department’s GPRA efforts. Instead, the officials discussed the

DOE’s efforts to improve the linkage among its strategic planning, performance

planning, performance reporting, and budget process. According to the officials,

these efforts recognize that GPRA’s traditional requirements—for example, that

performance measures and goals be short-term, narrowly drawn, and quantitative--do

not work well for many of DOE’s science efforts, since these efforts may involve

longer-term activities that are open-ended or difficult to quantify. As a result, DOE is

working with other science agencies, such as the National Science Foundation, to

develop methods that will result in better planning and budget processes and clearer

explanations of what the public receives for its federal investments in science.

We agree with these efforts and encourage DOE to consider alternative ways of

meeting the intent of GPRA that reflect the unique nature of science and the inherent

difficulties in quantifying the types of research it performs.
B-286003

7 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

-----

As agreed with your office, unless you publicly announce its contents earlier, we plan

no further distribution of this report until 30 days after the date of this report. At that

time, we will send copies to the Honorable Bill Richardson, Secretary of Energy, and

make copies available to others upon request.

Please call me or Derek B. Stewart on (202) 512-3841 if you or your staff have any

questions about this report. Key contributors to this report were Robert Baney, Dan

Haas, Richard Iager, and Daren Sweeney.

Sincerely yours,

Jim Wells

Director, Energy, Resources,

and Science Issues

Enclosure
Enclosure

8 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

Problems With Performance Goals and Measures for Doe’s Science

Programs

The following are examples of performance goals and measures for the Department of

Energy’s (DOE) science-related programs stated in DOE’s accountability report for fiscal

year 1999 and performance plans for fiscal years 2000 and 2001. These examples

illustrate problems we found generally in the form of (1) unclear goals, (2) unclear

measures, and (3) performance measures that are evaluations rather than activities that

help achieve a goal. The performance goals and measures are taken verbatim from the

accountability report and/or performance plans.

Unclear Performance Goals in 1999 and/or 2000

1999 and 2000 Performance Goal—ST2-1 Developing the Technologies to Meet

DOE’s Energy, National Security, and Environmental Goals. Develop the

technologies required to meet DOE’s energy, national security, and environmental quality

goals.

• GAO’s assessment—This goal is unclear because, by itself, does not provide

information about what DOE has to do to “develop” the technologies. There is no

discussion on the nature or level of effort to undertake or satisfy the goal and no

information about the energy, national security, and environmental goals that the new

technologies are being developed to meet. Because “developing” is a vague term,

almost any activity by DOE could achieve the performance goal.

1999 and 2000 Performance Goal—ST-2 Pursuing Technology Research

Partnerships. Pursue technology research partnerships with industry, academia, and

other government agencies and proactively accelerate the transition of technologies.

• GAO’s assessment: This goal does not clearly state, by itself, what DOE is trying to

achieve or how it plans to “pursue” and “proactively accelerate” the transition of

technologies. Developing research partnerships may advance technologies, but the

nature of DOE’s activity and the level of expected performance are not established in

the description of the goal, thus making it difficult to determine how much progress

is necessary to meet the goal.

1999 and 2000 Performance Goal—ST3 -1 Managing the National Laboratories,

Science-User Facilities, and Other Research Providers and Research Facilities.

Manage the National Laboratories, science-user facilities, and other DOE research

facilities in a more integrated, responsive, and cost-effective way, building on unique
Enclosure

9 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

core strengths and corresponding roles. Design, construct, and operate research

facilities in a timely and cost-effective manner.

• GAO’s assessment: This performance goal is important, but, by itself, DOE provides

no specifics about (1) what it wants to achieve and how it plans to manage in a more

integrated, responsive, and cost-effective ways; (2) what core strengths and

corresponding roles DOE is referring to; and (3) how this better management will

translate into better operation of these facilities. Moreover, the goal is so general

that almost any new DOE management activity at a DOE facility could be viewed as

contributing to the goal. Because “better management” is a vague term and no

baseline has been presented, it will be difficult to determine how much progress is

made toward the strategic objective of delivering leading-edge technologies.

1999 and 2000 Performance Goal--ST4-1 Developing and Promoting

Technologies and Programs that Deliver Information and Contribute to

Learning in Science, Math, Engineering, and Technology. Develop and promote

technologies and programs that deliver information and contribute to learning in science,

math, engineering and technology and, in general, expand access to DOE’s technical

information. Leverage DOE’s human and physical research infrastructure, working with

the National Science Foundation and other Federal agencies, to promote science

awareness, enable advanced educational research opportunities, build capabilities at

educational institutions, and improve educational opportunities for diverse groups.

• GAO’s assessment: This performance goal is unclear, by itself, because it provides no

clear understanding of what DOE is going to do to “develop,” “promote,” or

“leverage.” DOE provides no quantitative targets to meet. The text describes an

open-ended activity, not a goal. DOE could claim that any research it does addresses

and meets this goal. It is also unclear how much progress is necessary to achieve the

goal, since there is no baseline from which to measure progress.

Unclear Performance Measures in 1999 and 2000:

1999 Performance Measure--ST2-1: Initiate construction and commissioning of

the Los Alamos Target Irradiation Station, to improve isotope quality with

greater operating efficiency.
Enclosure

10 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

• GAO’s assessment—This measure is unclear because it does not say what DOE is

doing as part of the construction and commissioning of the Irradiation Station to

improve isotope quality and operating efficiency. Construction and commissioning of

the facility will permit the operation of the facility but may not result in great strides

towards improved isotope quality or operating efficiency. DOE provides no

explanation of how this station will improve isotope quality or how the greater quality

will be achieved.

2000 Performance Measure--SC2-1: Continue the Natural and Accelerated

Bioremediation Research (NABIR) program support fundamental research in

environmental and molecular sciences that will underpin the development of

biomediation for containing hazardous waste and cleaning DOE sites. Site

characterization of the first NABIR Field Research Center will proceed, and

activities necessary to enable research sample distribution to investigators will

commence.

• GAO’s assessment: This measure is vague in terms of the nature and extent of DOE’s

support for this activity and what is expected from this overall effort. Because the

measure applies to the cleanup of certain wastes at DOE sites, further clarification of

what outcome is being sought would improve the measure.

2000 Performance Measure--ST1-3: Continue Partnerships for AcademicIndustrial Research where peer reviewed grants are awarded to university

researchers for fundamental high-risk work jointly defined by the academic and

industrial partners.

• GAO’s assessment: This measure, while understandable, is silent on the nature of

DOE's commitment or expectations. As a result, apart from quantifying the resources

provided, it is difficult to know how the measure will be assessed. DOE’s required

level of performance is low--if it obligates the resources, it meets the goal.

2000 Performance Measure--ST1-1: Maintain the high quality and relevance of

DOE's science as evaluated by annual peer review and advisory committees.

• GAO’s assessment: This measure is vague because it does not indicate how DOE’s

science can be measured or quantified. This is a highly interpretative description and

one that reads more like a strategic objective than an annual performance measure.
Enclosure

11 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

Since DOE has a strategic plan and annual plans for the areas of science it wants to

pursue, it would appear that more specific measures could be developed.

2000 Performance Measure--ST3-3: Meet 75 percent of the requirements of

computer facilities and network users.

• GAO’s assessment: This performance measure is unclear because it does not specify

what computer facilities, network or users involved. In addition, the measure is

unclear about the type of requirements needed.

2000 Performance Measure--SC1-5: Operate a novel magnetic fusion

confinement device, the National Spherical Torus Experiment, with 0.5

megaampere plasma currents approaching 0.5 second pulse lengths and I

megaampere, currents for shorter pulses.

• GAO’s assessment: This measure is unclear in terms of what is expected from the

activity. While it reflects a certain target level of achievement for fusion experiments,

its relative significance to the advancement of our understanding of fusion energy is

not clear.

Annual Performance Measures that are Program Evaluations

2001 Performance Measure—Measure the progress and success of the Nuclear

Physics Program in responding to priorities and recommendations contained

within the long range plan of the DOE/NSF Nuclear Science Advisory Committee

(NASC) as measured by NASC evaluation letter to the Nuclear Physics Program.

• GAO’s assessment: This measure resembles an evaluation exercise assessing the

progress achieved by the program in relation to a suggestion received from an

advisory committee. In our view, this is an evaluation of the Nuclear Physics

Program more than it is an output of the program and as such may not be appropriate

as a performance measure.

2001 Performance Measure—Establish customer feedback mechanism to assess

and improve DOE’s information program and related products and services, and

partner with other government and private sector entities to improve ease of

use systems.

• GAO’s assessment: This measure appears to be an assessment of the information

program rather than a specific program activity. Part of the measure involves an
Enclosure

12 GAO/RCED-00-268R Portions of DOE’s GPRA Reports Concerning Science

activity to improve certain systems, but these tasks are not specifically identified and

expectations are not set.

(141478)
Orders by Internet

For information on how to access GAO reports on the Internet, send an e-mail message

with “info” in the body to

info@www.gao.gov

or visit GAO’s World Wide Web home page at

http://www.gao.gov

To Report Fraud, Waste, and Abuse

in Federal Programs

Web site: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: 1-800-424-5454