GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

United States General Accounting Office

Washington, DC 20548

Resources, Community, and

Economic Development Division

B-286138

August 30, 2000

The Honorable F. James Sensenbrenner

Chairman, Committee on Science

House of Representatives

Subject: Government Performance and Results Act: Information on the National

Science Foundation’s Performance Report for Fiscal Year 1999 and

Performance Plans for Fiscal Years 2000 and 2001

Dear Mr. Chairman:

As agreed with your office, we are providing you with information on our review of

the National Science Foundation’s (NSF) fiscal year 1999 performance report and

fiscal year 2000 and 2001 performance plans. NSF serves a unique role in supporting

basic research across many disciplines. Because basic research results are difficult

to quantify, NSF uses external expert review panels to qualitatively evaluate its

research outcome goals that are found in its performance plans and report. NSF also

has quantitative performance goals to gauge performance related to its internal

investment and management processes.

Specifically, we are providing information on (1) the extent to which the fiscal year

2000 and 2001 performance plans presented a consistent and coherent story

regarding the agency’s support of basic research efforts; (2) the goals and results as

reported in its fiscal year 1999 performance report; (3) the extent to which NSF’s

fiscal year 1999 performance report and fiscal year 2000 and 2001 performance plans

link to its fiscal year 2000 and 2001 budget requests; and (4) limitations with NSF’s

fiscal year 1999 performance report and fiscal year 2001 performance plan. We

focused on the fiscal year 2001 performance plan so that our feedback may be

considered as NSF finalizes the plan. Additional information on NSF’s

implementation of the 1993 Government Performance and Results Act (Results Act)

that we provided to your office on August 11, 2000, is contained in enclosure I.

NSF’s performance plans and report generally present a consistent and coherent

story regarding NSF’s support of basic research. NSF’s 2001 performance plan

incorporates its outcome goals from its fiscal year 2000 performance plan, into three

new outcome goals that broadly capture its investments aimed at encouraging new

scientific discoveries, developing a world-class workforce, and using state-of-the-art

tools for research and education. Among the key changes in the fiscal year 2001
GAO/RCED-00-281R NSF's GPRA Reports Concerning Science 2

performance plan, NSF expands the definition of merit review by including the

integration of research and education as a factor to be considered in making awards.

The plan also sets higher standards for certain aspects of the awards process and for

increasing reliance on electronic transfer of information. The plan varies in the level

of detail provided regarding its strategies for achieving its goals.

The fiscal year 1999 performance report evaluated NSF’s performance with respect to

5 outcome goals, 13 investment process goals and 5 management goals. The

performance report indicated that NSF achieved each of its 5 outcome goals; 9 of its

13 investment process goals; and 3 of its 5 management goals. According to external

expert reviews of its outcome goals, NSF supported research awards that led to

important discoveries; that were readily disseminated to the scientific, engineering,

and educational communities; and that provided opportunities for world-class

professional experiences for scientists and engineers. In addition, NSF’s outcome

goals also supported improvements in math and science skills and made timely data

available to international science and engineering users. In addressing its investment

process goals, NSF effectively implemented merit review, identified emerging

opportunities, encouraged integration of research and education, and maintained

construction and upgrades of facilities within cost and schedule estimates. However,

improvements are needed in NSF’s customer service. Specifically, NSF is not yet

meeting its goal of making 95 percent of its program announcements and solicitations

available at least 3 months prior to proposal deadlines. In addition, NSF must also

improve its proposal processing time so that it is completed within 6 months of a

proposal’s receipt, at least 70 percent of the time. NSF’s goal of increasing the

percentage of competitive research grants going to new investigators was nearly

achieved. NSF’s goal was to have 30 percent of its grants made to new investigators;

it achieved 27 percent. As for its management goals, NSF achieved its electronic

proposal processing, staff diversity, and year 2000 information goals. However, it did

not meet the training goal for FastLane, its electronic information system. Moreover,

NSF did not achieve its goal of having 70 percent of all reports submitted through the

new electronic system. In fiscal year 1999, 59 percent of its projects were reported

through the electronic system.

It is difficult to link NSF’s performance report and performance plans to its budget.

For the fiscal year 1999 performance report and fiscal year 2000 and 2001

performance plans, we found no clear linkage with the agency’s fiscal year 2000 and

2001 budget requests. NSF acknowledges that there is no clear linkage between the

outcome goals found in its performance report and plans and the agency’s budget. As

a result, NSF has attempted to improve the linkage between the fiscal year 2001

performance plan and the agency’s fiscal year 2001 budget request, by including

crosswalks on (1) the distribution of the budget across individual outcome goals, and

(2) the staffing levels and budget figures for each individual NSF directorate.

Moreover, NSF established a team that is reviewing the account structure to find

improved approaches for linking the budget with the goals. According to NSF

officials, a draft report from this team is due to the Director’s office in September

2000.

We identified a number of limitations with NSF’s fiscal year 1999 performance report

and fiscal year 2001 plan. For example, reports from the external expert review

panels, which were the foundation for the qualitative assessments of whether the
GAO/RCED-00-281R NSF's GPRA Reports Concerning Science 3

agency met its outcome goals, varied in their attention to detail. Some depended on

anecdotal evidence rather than systematic information as the basis for assessments;

some cited specific examples or included meaningful narrative to support their

evaluations while others provided none. Likewise, some provided specific criticisms

and recommendations for improvements, while others did not. In some cases, no

ratings were provided, and outcome goals were not even addressed. In addition, the

fiscal year 2001 performance plan includes performance indicators for each goal, but

these indicators do not provide a clear statement of expected performance for each

goal. In addition, the performance plan does not provide links between resources

and areas of emphasis in the plan; or discuss strategies and resources needed to

achieve goals. In all cases where we have concerns, NSF recognized that

improvements were warranted and had an action plan for improving internal

processes. The challenges ahead for NSF lay primarily in implementing these

improvements, including expansion of its electronic proposal and award information

system, streamlining its electronic proposal preparation process, and redoing its

strategy for conducting external expert reviews to increase the credibility of

information provided by these evaluating committees.

Agency Comments

We provided copies of a draft of this report to the National Science Foundation for its

review and comment. NSF officials, including the Director, Office of Integrative

Activities, Chairman of the GPRA Infrastructure Implementation Council, as well as

the Chief Information Officer and Acting Chief Financial Officer generally agreed

with the information presented in the report.

Scope and Methodology

Our observations were generally based on the requirements of the Results Act,

guidance to agencies from the Office of Management and Budget (OMB) for

developing performance plans and reports (OMB Circular A-11, part 2) and previous

reports and evaluations by us. We also analyzed the final draft of NSF’s 1999

performance report, the latest version of the 2000 performance plan dated January

2000, and the latest version of the 2001 performance plan dated February 7, 2000. We

also met with officials from NSF’s Office of Integrative Activities, Office of Inspector

General and Office of Budget, Finance and Award Management. We conducted our

review from July 2000 through August 2000 in accordance with generally accepted

government auditing standards.

-----

As agreed with your office, unless you publicly announce its contents earlier, we plan

no further distribution of this report until 30 days from the date of this letter. At that

time, we will send copies to Dr. Rita R. Colwell, Director, National Science

Foundation and make copies available to others upon request.
GAO/RCED-00-281R NSF's GPRA Reports Concerning Science 4

Please call me or Derek B. Stewart on (202) 512-3841 if you or your staff have any

questions about this report. Key contributors to this report were Shawn Arbogast,

Diane Raynes, and Ed Zadjura.

Sincerely yours,

Jim Wells,

Director, Energy, Resources,

and Science issues

Enclosure
5 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

Enclosure I

Briefing Charts

GPRA: Information on the National Science Foundation’s

Performance Report for Fiscal Year 1999 and

Performance Plans for Fiscal Years 2000 and 2001

For the House Committee on Science

August 2000
6 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

2

The National Science Foundation and the Results Act

• Background

• Outcome Goals in Performance Plans and Reports

• Summary of FY 1999 Goals From the Performance Report

• Linkage of Report and Plans to the Agency’s Budget

• Limitations With the FY 1999 Performance Report and FY 2001

Performance Plan

• Challenges Ahead for NSF’s Implementation of the Results Act
7 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

3

The National Science Foundation and the Results Act

 Background

• NSF supports basic research across many disciplines

• Other federal agencies have mission-oriented research objectives,

such as energy (DOE), biomedicine (NIH), space exploration

(NASA), and weapons systems (DOD)

• Implementing the Results Act has been a challenge for NSF and

other agencies whose missions involve research activities because

• The substance and timing of research outcomes are

unpredictable

• It is difficult to link research outcomes to annual budgets

• Research results can be difficult to report quantitatively
8 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

4

The National Science Foundation and the Results Act

 Background

• With OMB’s approval, NSF uses an alternative format to evaluate

the extent to which its research results attain NSF’s outcome goals

• The alternative format is a qualitative scale for the assessment

of outcomes

• In using the alternative format, NSF relies on external expert

review panels

• These review panels assess the quality of research results and

report progress toward the outcome goals

• NSF uses quantitative goals for its internal investment and

management process goals
9 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

5

The National Science Foundation and the Results Act

 Outcome Goals in Performance Plans and Report 

• The outcome goals of the FY 1999 performance report and the FY

2000 and FY 2001 performance plans address the range of science

and engineering research and education supported by NSF

• The FY 2001 performance plan combined the five outcome goals

from previous years into three outcome goals
10 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

6

The National Science Foundation and the Results Act

 Outcome Goals in Performance Plans and Report

• The FY 2001 performance plan

• Combined the first and second outcome goals from the 1999

performance report and 2000 performance plan

• 1999 and 2000: “Discoveries at and across the frontier of science” and “connections

between discoveries and their use in service to society”

• 2001: “IDEAS--Discovery at and across the frontier of science and engineering, and

connections to its use in society”

• Combined the third and fourth outcome goals from the 1999

performance report and 2000 performance plan

• 1999 and 2000: “A diverse, globally-oriented workforce of scientists and engineers” and

“Improved achievement in mathematics and science skills needed by all Americans”

• 2001: “PEOPLE--A diverse, internationally-competitive, and globally-engaged workforce

of scientists, engineers, and well-prepared citizens”
11 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

7

The National Science Foundation and the Results Act

 Outcome Goals in Performance Plans and Report

• Included the fifth goal of the 1999 performance report and

2000 performance plan under a new and broader outcome

goal

• 1999 and 2000: “Timely and relevant information on the national and international

science and engineering enterprise”

• In 2001: “TOOLS--Broadly accessible state-of-the-art information bases and

shared research and education tools”
12 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science
13 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

9

The National Science Foundation and the Results Act

 Summary of FY 1999 Goals From the Performance Report

• The FY 1999 performance report addressed

• Outcome goals

• Investment process goals

• Management goals
14 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

10

The National Science Foundation and the Results Act

 Summary of FY 1999 Results From the Performance Report

• The FY 1999 performance report addressed five outcome goals:

• Discoveries at and across the frontier of science and engineering

• NSF is successful when awards lead to important discoveries; new knowledge and

techniques, both expected and unexpected, within and across traditional disciplinary

boundaries; and high-potential links across these boundaries.

• In FY 1999, all 43 external reports rated NSF successful in achieving this goal.

• Connections between discoveries and their use in service to

society

• NSF is successful when the results of NSF awards are rapidly and readily available and

feed, as appropriate, into education, policy development, or use by other federal

agencies or the private sector.

• In FY 1999, 42 of 43 external reports rated NSF successful in achieving this goal.
15 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

11

The National Science Foundation and the Results Act

 Summary of FY 1999 Results From the Performance Report

• A diverse, globally oriented workforce of scientists and engineers

• NSF is successful when participants in NSF activities experience world-class professional

practices in research and education, using modern technologies and incorporating

international points of reference; when academia, government, business, and industry

recognize their quality; and when the science and engineering workforce show increased

participation of underrepresented groups.

• In FY 1999, 38 of 44 external reports rated NSF successful in achieving all or most areas of

this goal.

• Improved achievement in the mathematics and science skills

needed by all Americans

• NSF is successful when its awards lead to the development, adoption, adaptation, and

implementation of effective models, products, and practices that address the needs of all

students; well-trained teachers who implement standards-based approaches in their

classroom; and improved student performance in participating schools and districts.

• In FY 1999, 18 of 22 external reports rated NSF successful in achieving all or most areas of

this goal.
16 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

12

The National Science Foundation and the Results Act

 Summary of FY 1999 Results From the Performance Report

• Timely and relevant information on the national and international

science and engineering enterprise

• Decrease the time interval by 10% from the current average of 540 days between the

reference period (the time to which the data refer) and the reporting of data.

• Achieve customer satisfaction ratings with the relevance of products offered of at least 45%

“excellent” and at least 90% “excellent” or “good.” FY 1998 baseline is 38% “excellent” and

88% “excellent” or “good” based on a 1996 customer survey.

• In FY 1999, this goal was achieved; the average time interval decreased to 485 days.

• In FY 1999, this goal was achieved; ratings were 60% “excellent” and 90% “excellent to

good.”
17 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

13

The National Science Foundation and the Results Act

 Summary of FY 1999 Results From the Performance Report

• The FY 1999 performance report addressed 13 investment process

goals, such as

• Use of merit review

• At least 90% of NSF’s funds will be allocated to projects reviewed by appropriate peers

external to NSF and selected through a merit-based competitive process.

• In FY 1999, 95% of projects allocated funds were merit reviewed.

• Implementation of merit review criteria

• NSF’s performance in implementation of the new merit review criteria is successful

when reviewers address the elements of both generic review criteria appropriate to the

proposal at hand and when program officers take the information provided into account

in their decisions on awards.

• NSF performance is minimally effective when reviewers consistently use only a few of

the suggested elements of the generic review criteria, although others might be

applicable.

• In FY 1999, 36 of 44 external reports rated NSF successful in achieving this goal.
18 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

14

The National Science Foundation and the Results Act

 Summary of FY 1999 Results From Performance Report

• The FY 1999 performance report addressed five management goals,

including

• Electronic Proposal Processing

• NSF will achieve at least 25% of full proposal submissions electronically through

FastLane (the electronic reporting system) improving on the FY 1998 baseline of 17.5%.

• In FY 1999, 44% of full proposal submissions were received through FastLane.

• Diversity

• In FY 1999, as all appointments for scientists and engineers are considered, the

recruiting organization will demonstrate efforts to attract applications from groups that

are underrepresented in the science and engineering staff as compared to their

representation among Ph.D. holders in their fields.

• In FY 1999, this goal was achieved.

• Project Reporting System

• During FY 1999, at least 70% of all project reports will be submitted through the new

electronic Project Reporting System.

• In FY 1999, this goal was not achieved; only 59% of all project reports were submitted

through the electronic Project Reporting System module included in FastLane.
19 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

15

The National Science Foundation and the Results Act

 Linkage of Report and Plans to the Agency’s Budget

• There is no clear linkage between the FY 1999 performance report

and the agency’s budget

• NSF notes that there is no OMB requirement to link the performance

report to the budget.

• NSF acknowledges there is no clear linkage between the outcome

goals and the agency’s budget in their reports and plans

• NSF has improved this linkage in its FY 2001 budget request.

• NSF has established a team to review the account structure to find

improved approaches for linking the budget with the goals; a draft

report from this team is expected in September.
20 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

16

The National Science Foundation and the Results Act

 Linkage of NSF’s Performance Report and Plans to the Agency’s

 Budget

• There is no clear linkage between the FY 2000 performance plan

and the agency’s budget, rather, the plan provides

• A crosswalk that includes key functions rather than outcome goals

stated in the plan

• A table that provides the interactions (or overlap) of the outcome goal

by key functions

• There is an improved linkage between the FY 2001 performance

plan and the agency’s budget because the plan provides

• A crosswalk on the distribution of NSF’s budget across individual

outcome goals

• A crosswalk depicting staffing level and programmatic budget figures

by individual NSF directorates.
21 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

17

The National Science Foundation and the Results Act

 Limitations With NSF’s FY 1999 Performance Report and FY 2001

 Performance Plan

• The FY 1999 performance report

• Overall, NSF was generally successful in meeting its outcome

goals, based on program evaluations by external expert review

panels, and examples of high quality scientific outputs and

outcomes chosen to show NSF’s achievement

• However, that conclusion depends on the quality of the expert

review panels’ reports, which vary widely

• Many of the evaluation reports are inconsistent

• Other reviews depended on anecdotal knowledge, rather

than on systematic information, as the basis for the

assessment.
22 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

18

The National Science Foundation and the Results Act

 Limitations With the FY 1999 Performance Report and FY 2001

 Performance Plan

• The quality of external expert review panel reports varied, for

example

• Some panels cited specific examples to support their

evaluations while others provided fewer or no examples

• Some panels provided specific criticism or recommendations

while others provided little or no such information

• Some panels did not follow the evaluation template
23 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

19

The National Science Foundation and the Results Act

 Limitations With the FY 1999 Performance Report and FY 2001

 Performance Plan

 Examples of Expert Review Panel Assessments from the Division of

Bioengineering and Environmental Systems Report

• Provided examples of NSF-funded activities to justify their ratings such as research at

Stanford University that resulted in the:

• Creation of novel “gene cassettes” expanding chemical diversity

• Filing of patents as a results of the studies undertaken

• Development of at least one new commercial venture

• Generation of interest in “bio-combinational chemistry” by pharmaceutical

companies

• Provided suggestions of metrics to use for evaluation effort

• Number of patents, patent disclosures, or licensures 

• Number of start-ups

• Number of publications

• Provided specific criticism and recommendations for improvement

• Improve reporting format to better support evaluation committee needs

• Anecdotal information was collected and required validating by Principal

Investigators
24 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

20

The National Science Foundation and the Results Act

 Limitations With the FY 1999 Performance Report and FY 2001

 Performance Plan

• Examples of Expert Review Panel Assessments from the Division

of Materials Research: Materials Theory Program Report

• Provided specific examples of NSF-funded activities to justify their ratings

• Researchers Stormer and Tsui won the Nobel Prize for research in Quantum Hall

Effect

• Researcher Kohn shared in the Nobel Prize for Chemistry for his invention of

Density Functional Theory

• Division of Biological Infrastructure: Training Cluster Report

• Provided no examples or meaningful narrative to justify ratings

• Division of Information and Intelligent Systems Report

• Did not follow format specified in the NSF external expert review panel guidelines

• Provided no ratings

• Did not address outcome goals or GPRA requirements
25 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

21

The National Science Foundation and the Results Act

 Limitations With the FY 1999 Performance Report and FY 2001

 Performance Plan

• The FY 2001 performance plan

• Does not provide a clear statement of expected performance

for subsequent comparison

• Provides only general criteria for evaluating success in

achieving the revised outcomes

• Does not clearly discuss strategies and resources needed to

achieve goals

• Does not specifically provide links between the resources and

areas of emphasis in the plan

• Does not address the problem of the inconsistent quality of

external expert review panel reports and thus offers only limited

confidence that performance information will be credible
26 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

22

The National Science Foundation and the Results Act

 Challenges Ahead for NSF’s Implementation of the Results Act

• Improving FastLane, an electronic proposal and award information

system

• Holds promise for providing systematic, evaluative information over time

• Plans include incorporating the external expert review panel process into this system

• Developing a strategy to ensure or increase the credibility of

information compiled by external expert review panels

• Reissue guidance and template providing more assistance

• Reduce subjectivity in reports

• Improve the extent to which the reports address the requirements of the Results Act

• Developing ways to address the problems created by the timing of

Results Act reports and plan

• Information needed for FY 1999 reports is also essential for the FY 1999 accountability

report and the performance plans for fiscal year 2000 and 2002
27 GAO/RCED-00-281R NSF's GPRA Reports Concerning Science

23

The National Science Foundation and the Results Act

 Challenges Ahead for NSF’s Implementation of the Results Act 

• Diversity of awardees

• Currently implementing approaches to increase and retain the number of women and

underrepresented minorities in the proposal application pool

• Proposal preparation process

• Currently identifying bottlenecks in process and putting in place an electronic system to

improve timeliness in proposal preparations

• In FY 2001, NSF plans to conduct 10 pilot paperless projects that will manage the

competitive review process in a totally electronic environment

(141471)
Orders by Internet

For information on how to access GAO reports on the Internet, send an e-mail message

with “info” in the body to

info@www.gao.gov

or visit GAO’s World Wide Web home page at

http://www.gao.gov

To Report Fraud, Waste, and Abuse

in Federal Programs

Web site: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: 1-800-424-5454