National Science Foundation: Status of Achieving Key Outcomes and Addressing Major Management Challenges
Released: 2001-07-16T00:00:00

Letter

Background

GPRA is intended to shift the focus of government decisionmaking, management, and accountability from activities and processes to the results and outcomes achieved by federal programs. New and valuable information on the plans, goals, and strategies of federal agencies has been provided since federal agencies began implementing GPRA. Under GPRA, annual performance plans are to clearly inform the Congress and the public of (1) the annual performance goals for agencies’ major programs and activities, (2) the measures that will be used to gauge performance, (3) the strategies and resources required to achieve the performance goals, and (4) the procedures that will be used to verify and validate performance information. These annual plans, issued soon after transmittal of the President’s budget, provide a direct linkage between an agency’s longer-term goals and mission and day-to-day activities. Annual performance reports are to subsequently report on the degree to which performance goals were met. The issuance of the agencies’ performance reports, due this year by March 31, represents a new and potentially more substantive phase in the implementation of GPRA—the opportunity to assess federal agencies’ actual performance for the prior fiscal year and to consider what steps are needed to improve performance and reduce costs in the future.

NSF’s mission is to promote the progress of science; to advance the national health, prosperity, and welfare; and to secure the national defense. NSF carries out its mission primarily by making merit-based grants and cooperative agreements to individual researchers and groups in partnership with colleges, universities, and other public and private institutions. For fiscal year 2001, NSF has a budget of $4.4 billion and a staff of about 1,200 government employees to accomplish its mission.

Implementing GPRA has been a challenge for NSF, whose mission involves funding research activities, because the substance and timing of research outcomes are unpredictable and research results can be difficult to report quantitatively. With OMB’s approval, NSF uses an alternative format—a qualitative scale for the assessment of outcomes—for which it relies on independent committees of scientific experts. These committees determine the level of NSF’s success in achieving its goals. NSF uses quantitative goals for its management and investment process goals.

Assessment of NSF’s Progress and Strategies in Achieving Selected Key Outcomes

This section discusses our analysis of NSF’s performance in achieving the selected key outcomes, as well as the strategies it has in place— particularly strategic human capital management and information technology strategies—for achieving these outcomes. In discussing these outcomes, we have also provided information drawn from our prior work on the extent to which NSF has provided assurance that the performance information it is reporting is accurate and credible.

Discoveries at and Across the Frontier of Science and Engineering

NSF, in its fiscal year 2000 performance report, states that it met its discoveries outcome and cites numerous examples of its achievements in such scientific fields as mapping the Arctic Ocean floor and extra-solar planetary discovery. NSF judged its performance as successful on the basis of assessments by independent committees of scientific experts. In compiling committee members’ scores and aggregating their comments, NSF took into account only those reports with substantive comments and ratings that were clearly justified. NSF officials told us that, for the scientific discoveries outcome goal, all of the committees judged NSF as successful in achieving it and justified their assessments. However, the performance report did not provide information on the specific numbers of reports it included and excluded in reaching its judgments for this outcome or any of the other outcomes. Furthermore, NSF discussed the independent scientific committees’ results for only one of the scientific discoveries five areas of emphasis—namely, the balance of innovative, risky, and interdisciplinary research area. Instead of providing a more complete analysis of the scientific committees’ assessments, NSF contracted with an external third party—PricewaterhouseCoopers—to make an independent assessment of the performance results. PricewaterhouseCoopers concluded that NSF’s fiscal year 2000 results were valid and verifiable.

NSF’s fiscal year 2002 performance plan included a new section on the means and strategies for success related to this outcome that includes strategies that generally are clear and reasonable. To implement its outcome goal, NSF has both (1) process strategies, such as supporting the most promising ideas through merit-based grants and cooperative agreements, and (2) program strategies, such as supporting programmatic themes identified as areas of emphasis. However, NSF’s plan generally does not address key components of strategic human capital management, although its “people” and “management” outcome goals include such human capital initiatives as workforce diversity, an NSF Academy for workforce training, and a survey on the work environment. NSF is in the process of developing a 5-year strategic plan on its workforce needs that must be submitted to OMB by July 20, 2001. This strategic plan will guide NSF’s future effort in this area.

Efficiency and Effectiveness of Administration of Research Grants

NSF reported that it made substantial progress, achieving most of its performance goals related to the award and administration of research grants. While not listed as an outcome goal, the administration of grants includes many of NSF’s management and investment process goals. For example, NSF exceeded by 21 percent one of its management performance goals—to receive at least 60 percent of full grant proposal submissions electronically through a new computer system called FastLane. NSF also exceeded by 5 percent another management goal that at least 90 percent of its funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process. NSF continued to miss one of its investment process goals—to process 70 percent of proposals within 6 months of receipt—dropping from 58 percent to 54 percent in fiscal year 2000. As part of its review of NSF, PricewaterhouseCoopers concluded that NSF’s fiscal year 2000 processes were valid and verifiable and relied on sound business processes, system and application controls, and manual checks of system queries to confirm the accuracy of reported data.

NSF’s fiscal year 2002 performance plan generally includes strategies for achieving NSF’s performance goals that appear to be clear and reasonable. However, in some cases, the strategies are vague, and how NSF will use them to achieve its performance goals is unclear. For example, one of NSF’s three strategies for identifying best management practices for its large infrastructure projects is to ensure input from members of the external community who build, operate, and utilize research facilities. Furthermore, while NSF has strategies for the process of funding awards, it does not generally address the oversight needs to ensure that funding recipients meet the awards’ requirements. NSF’s 5-year workforce strategic plan is addressing concerns regarding the management of a growing portfolio of program activities with relatively flat personnel levels—a key issue for developing strategic human capital management strategies.

Comparison of NSF’s Fiscal Year 2000 Performance Report and Fiscal Year 2002 Performance Plan With the Prior Year’s Report and Plan for Selected Key Outcomes

For the selected key outcomes, this section describes major improvements or remaining weaknesses in NSF’s (1) fiscal year 2000 performance report in comparison with its fiscal year 1999 report and (2) fiscal year 2002 performance plan in comparison with its fiscal year 2001 plan. It also discusses the degree to which the agency’s fiscal year 2000 report and fiscal year 2002 plan address concerns and recommendations by NSF’s Inspector General.

Comparison of Performance Reports for Fiscal Years 1999 and 2000

NSF improved its fiscal year 2000 performance report, making major changes to address the weaknesses we reported in the prior year’s performance report. Our prior year’s review noted that NSF did not discuss either its reasons for falling short of a performance goal or its strategies for attaining the goal in the future. NSF’s 2000 report corrected this weakness. For example, regarding the technology-related goal to submit, review, and process proposals electronically, the report states that the reason for not achieving the goal was due to the technological, financial, and legal issues related to electronic signatures. The strategy for addressing the technological issue was to demonstrate the paperless review capability by conducting 10 pilot paperless projects in 2001 that manage the review process in an electronic environment. We also questioned the quality of the information in the 1999 performance report, noting that it provided virtually no assurance that the information was credible. As mentioned earlier, NSF contracted with PricewaterhouseCoopers to review aspects of its GPRA data collection efforts and its performance assessment results. PricewaterhouseCoopers found no basis for questioning the integrity of the results.

NSF can improve its future reports in several ways. The results of the independent committees’ reviews would benefit from more detailed information, such as including all of the areas of emphasis and the results. In addition, last year, we noted that the 1999 performance report did not describe NSF’s financial role in the examples of scientific successes presented. Such information, we said, would help to judge the extent of NSF’s role in achieving these successes. NSF officials maintain that determining NSF’s financial role in these successes would be extremely difficult and would take a considerable effort. NSF officials told us that the successes they identified for this outcome were primarily due to NSF awards. That statement would have been useful in assessing the 2000 performance report.

Comparison of Performance Plans for Fiscal Years 2001 and 2002

NSF made improvements to its fiscal year 2002 performance plan. For example, last year, we reported that the performance plan contained little useful information about NSF’s intended strategy to achieve its goals, including a discussion of the problems. The 2002 plan includes a new section on the means and strategies for success. For example, for its new goal of award oversight and management, NSF will ensure that the internal committee reviewing the oversight activities for large infrastructure projects has broad disciplinary expertise and experience in managing facilities. As previously mentioned, NSF is also addressing data quality concerns, providing confidence that future performance information will be credible. Furthermore, NSF revised its outcome goal such that it does not have to succeed in demonstrating significant achievement in discoveries that advance the frontiers of science, engineering, or technology. Rather, discoveries is now one of six performance indicators for which NSF will consider itself successful when a majority is achieved. Last year, we also reported that the strategies for achieving the goals were not clearly discussed. NSF includes a new section on the means and strategies for success under each goal.

NSF can improve its future performance plans by addressing its resource needs. Last year, we noted that the plan did not clearly discuss the resources for achieving the goals or the specific links between the resources and the areas of emphasis. The 2002 performance plan still does not do so. As discussed earlier, NSF’s 5-year workforce strategic plan is expected to address human capital issues, providing a basis for addressing this issue in next year’s performance plan.

NSF’s Efforts to Address Its Major Management Challenges Identified by GAO

GAO has identified two governmentwide high-risk areas: strategic human capital management and information security. Regarding strategic human capital management, we found that NSF’s performance plan generally did not have goals and measures related to strategic human capital management, and NSF’s performance report did not explain its progress in resolving strategic human capital management challenges. However, as mentioned earlier, NSF is developing a 5-year workforce strategic plan. With respect to information security, we found that NSF’s performance plan had a goal and measures related to information security. While NSF’s performance report did not explain its progress in resolving information security challenges, it did indicate that NSF has internal management controls that continually monitor data security.

Agency Comments

We provided NSF and the Office of the Inspector General with a draft of this report for their review and comment. We met with NSF officials, including the Chief Information Officer and the Inspector General. The NSF officials generally agreed with the report. However, they noted that the fiscal year 2000 performance report did not respond to some of the Inspector General’s management challenges primarily because these challenges were identified in a November 30, 2000, letter. The Inspector General agreed that some of these management challenges were new. The NSF officials recognize that certain challenges not in the current plan and report are important, and they noted that these challenges are being addressed through internal management controls and processes. They added that NSF will continue to consider these challenges for incorporation in future performance plans. The NSF officials also provided technical clarifications, which we incorporated as appropriate.

Scope and Methodology

Our evaluation was generally based on the requirements of GPRA, the Reports Consolidation Act of 2000, guidance to agencies from OMB for developing performance plans and reports (OMB Circular A-11, Part 2), previous reports and evaluations by us and others, our knowledge of NSF’s operations and programs, GAO’s identification of best practices concerning performance planning and reporting, and our observations on NSF’s other GPRA-related efforts. We also discussed our review with NSF officials in the Office of Information and Resource Management; the Office of Budget, Finance, and Award Management; the Office of Integrative Activities; and the Office of Inspector General.

The agency outcomes that were used as the basis for our review were identified by the Ranking Minority Member of the Senate Committee on Governmental Affairs as important mission areas for NSF and do not reflect the outcomes for all of NSF’s programs or activities. The major management challenges confronting NSF, including the governmentwide high-risk areas of strategic human capital management and information security, were identified by (1) our January 2001 high-risk update and (2) NSF’s Office of Inspector General in November 2000. We did not independently verify the information contained in the performance report and plan, although we did draw from other GAO work in assessing the validity, reliability, and timeliness of NSF’s performance data. We conducted our review from April through June 2001 in accordance with generally accepted government auditing standards.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies to appropriate congressional committees; the Director, NSF; and the Director of OMB. Copies will also be made available to others on request.

If you or your staff have any questions, please call me at (202) 512-3841. Key contributors to this report were Richard Cheston, Alan Stapleton, Elizabeth Johnston, and Sandy Joseph.

Appendix I: Observations on NSF’s Efforts to Address Its Management Challenges

The following table discusses the major management challenges confronting the National Science Foundation (NSF), including the governmentwide high-risk areas of strategic human capital management and information security, identified by our January 2001 high-risk update and NSF’s Office of Inspector General (IG) in November 2000. The first column of the table lists the management challenges identified by our office and NSF’s IG. The second column discusses NSF’s progress, as discussed in its fiscal year 2000 performance report, in resolving these challenges. The third column discusses the extent to which NSF’s fiscal year 2002 performance plan includes performance goals and measures to address each of these challenges. We found that while the fiscal year 2000 performance report discussed NSF’s progress in resolving most of its major challenges, it did not discuss NSF’s progress in resolving the following challenges: (1) addressing strategic human capital management issues regarding strategic human capital planning and organizational alignment, leadership continuity and succession planning, and creating results-oriented organizational cultures; (2) developing appropriate data security controls to reduce the ever increasing risk of unauthorized access; (3) developing a more coherent award administration program that ensures that grantees comply with NSF’s award requirements; (4) ensuring that NSF grantees meet their cost-sharing obligations; and (5) providing the science, operations, and logistics support needed to manage the U.S. Antarctic Program. Of NSF’s 10 major management challenges, its fiscal year 2002 performance plan (1) had goals and measures that were directly related to 5 of the challenges; (2) had goals and measures that were indirectly applicable to 1 challenge; (3) had no goals and measures related to 1 challenge but discussed strategies to address it; and (4) did not have goals, measures, or strategies to address 3 challenges.