NASA: Status of Achieving Key Outcomes and Addressing Major Management Challenges
Released: 2001-07-31T00:00:00

Letter

Background

GPRA is intended to shift the focus of government decisionmaking, management, and accountability from activities and processes to the results and outcomes achieved by federal programs. New and valuable information on the plans, goals, and strategies of federal agencies has been provided since federal agencies began implementing GPRA. Under GPRA, annual performance plans are to clearly inform the Congress and the public of (1) the annual performance goals for agencies’ major programs and activities, (2) the measures that will be used to gauge performance, (3) the strategies and resources required to achieve the performance goals, and (4) the procedures that will be used to verify and validate performance information. These annual plans, issued soon after transmittal of the President’s budget, provide a direct linkage between an agency’s longer-term goals and mission and day-to-day activities. Annual performance reports are to subsequently report on the degree to which performance goals were met. The issuance of the agencies’ performance reports, due by March 31, represents a new and potentially more substantive phase in the implementation of GPRA—the opportunity to assess federal agencies’ actual performance for the prior fiscal year and to consider what steps are needed to improve performance, and reduce costs in the future.

NASA’s mission encompasses human exploration and development of space, the advancement and communication of scientific knowledge, and research and development of aeronautics and space technologies. Its activities span a broad range of complex and technical endeavors—from investigating the composition, evaluation, and resources of Mars; to working with its international partners to complete and operate the International Space Station; to providing satellite and aircraft observations of earth for scientific and weather forecasting purposes; to developing new technologies designed to improve air flight safety.

Assessment of NASA’s Progress and Strategies in Accomplishing Selected Key Outcomes

This section discusses our analysis of NASA’s performance in achieving its selected key outcomes and the strategies the agency has in place to achieve unmet performance goals and measures in the future. In discussing these outcomes, we have also provided information drawn from our prior work on the extent to which the agency provided assurance that the performance information it is reporting is credible.

Scientific Knowledge of the Earth System

The performance report indicated that NASA made progress toward achieving its key outcome of expanding the scientific knowledge of the Earth system. NASA reported that all performance targets for this outcome were met, except two. However, NASA provided reasonable explanations for not meeting them in fiscal year 2000. For example, NASA reported that a lack of science quality spectroradiometer ocean color data prevented one of the targets from being fully achieved, and difficulties with an international partner prevented the other target, “Launch the NASA-National Center for Space Studies Jason-1 mission,” from being achieved.

The NASA Advisory Council provided an independent evaluation of NASA’s fiscal year 2000 performance and the evaluation was included in the performance report. Its evaluation of performance related to this outcome was positive, indicating the failure to launch the Jason-1 spacecraft as the most significant shortfall. However, the Council concluded that many of the performance targets across all of NASA’s Enterprises were too vague and did not sufficiently relate to the actual programs being implemented. The Council did not identify which targets it viewed as vague in its report, but it recommended that the targets be better written and that NASA communicate to the public the reason the metric or program is important. In response to the Council’s recommendation, NASA added statements to performance targets that explain why the performance results are meaningful.

Regarding data credibility, NASA disclosed the methods used for verifying and validating the data and the data sources for each performance target associated with this outcome in the fiscal year 2000 report. This is an important step toward providing confidence that performance results are credible. However, for many performance targets, NASA did not discuss limitations in the data and steps it would take to correct them. A recent NASA Office of Inspector General (OIG) report states that beginning with the fiscal year 2002 final performance plan, NASA will discuss anticipated data limitations. Also, in some cases, NASA did not clearly address how the data was validated. For example, the verification and validation narrative for the performance target, “Continue the ocean color time with 60 percent global coverage every four days—a 35 percent improvement over fiscal year 1999,” reads as follows. “The two-day coverage is required to account for the losses due to the tilt maneuver of the sensor and interorbit gaps. When clouds are taken into consideration, the coverage is reduced to 50-60 percent. The Sea-viewing Wide Field-of-view Sensor Project has increased the Global Area Coverage data beyond that expected by eight percent by collecting data to higher latitudes than planned on all orbits. However, pole-to-pole coverage each day is not possible since data at low Sun angles are not scientifically useful for ocean color research.

The Moderate Resolution Imaging Spectroradiometer instrument aboard Terra is beginning to supply additional data to meet to meet the ocean color requirement.” “Salinities in the ocean are typically 33-32 PSU. In 1998, they got +/- 1 PSU, so the target for a 10x improvement would be 0.1 PSU. Due to pointing error on the plane L-Band, radiometer accuracy is adversely affected. From a satellite sensor this error is much reduced. The airborne results coupled with theoretical studies now show that a monthly average of sea surface salinity, at a resolution of 1degree latitude x 1 degree longitude can be produced with an accuracy of <0.1-0.2 PSU, which will meet the target. Important to this analysis is getting the sea surface temperature right, and also getting sea surface roughness from scatterometers. The monthly data product is suitable for ocean circulation studies. It will also be possible to produce a weekly data product, more for meteorological use.”

Regarding its plans for achieving the two unmet targets for this outcome, NASA established reasonable strategies and time frames. It reported that progress was significant in one of the performance targets (footnote 5) and that full achievement of that target, anticipated in fiscal year 2001, is dependent on the “availability of valid moderate resolution imaging spectroradiometer data.” The report noted that the planned delivery of new processing software in November 2000 was expected to improve data quality to a level sufficient for an initial merging of SeaWiFs and MODIS oceans products. Achievement of the other target, “Launch the NASA- National Center for Space Studies Jason-1 mission,” is also anticipated in fiscal year 2001.

Lastly, NASA credits the contribution of the other agencies for the successful achievement of performance targets related to this outcome. For example, for the achieved target, “Demonstrate the utility of spaceborne data for flood plain mapping with the Federal Emergency Management Agency (FEMA),” the report credits FEMA, the Army Corps of Engineers, and NASA for conducting cooperative demonstration projects to evaluate NASA and commercially provided digital topographic and image-based information products to re-map flood plains.

Commercial Development of Space

The performance report indicated that the agency made some progress toward achieving its key outcome of expanding the commercial development of space. Over half of the performance targets that we assessed for this outcome were reported as having been met. For example, NASA reported that it achieved its targets to (1) promote privatization and commercialization of space shuttle payload operations through the transition of payload management functions by fiscal year 2000 and (2) establish up to two new commercial space centers. NASA provided clear and reasonable explanations for targets that were not met. For example, NASA reported that its performance target to promote privatization of space shuttle operations and reduce civil service requirements for operations by 20 percent in fiscal 2000 was not met following the agency’s decision to hire additional staff to ensure that safety would not be compromised for space shuttle missions. In August 2000, we reported that several internal NASA studies had shown that the agency’s space shuttle program’s workforce had been affected negatively by NASA’s downsizing, much of which occurred after 1995.

NASA reported that its performance target to complete small payload focused technologies and select concepts for flight demonstration of a reusable first stage was not met because the agency decided to terminate this activity once it was clear that the cost objectives could not be met. Furthermore, the performance targets for the X-33 and X-34 programs— which sought to develop and demonstrate technologies needed for future reusable spacecraft in partnership with private industry—were not met since they were not competitively selected for additional funding.

The NASA Advisory Council had concerns about this outcome. In particular, the Council noted that efforts planned under the new Space Launch Initiative appear to be elusive, at best. The Council also stated that this highlights the time-lapse between definition and evaluation of specific (and critical) performance targets, but the Council did not provide further elaboration.

NASA did not provide strategies and time frames for achieving the unmet target of pursuing the commercial marketing of space shuttle payloads by working to allow the space flight operations contractor to target two reimbursable flights, one in fiscal year 2001 and one in fiscal year 2002. NASA stated the target remains feasible, but no reimbursable flights in fiscal 2001 and fiscal year 2002 are planned due to policy limitations impeding the marketing process. Regarding strategic human capital management, NASA set one related target to promote privatization of space shuttle operations and reduce civil service resource requirements for operations by 20 percent (from the fiscal year 1996 full-time equivalent levels) in fiscal year 2000. However, this target was not met since NASA had decided to end its downsizing efforts.

International Space Station Deployment, Operation, and Cost

The performance report indicated that NASA’s progress toward achieving the key outcome of deploying and operating the space station safely and cost-effectively was limited with respect to achievement of the agency’s planned performance targets. Since the key outcome is not included in the report as a specific goal or objective, we based our assessment of it on a related objective in the report. The related objective is to deploy and operate the space station to advance scientific, exploration, engineering, and commercial objectives. All of the performance targets for this objective are associated with a NASA launch, except one, and none was achieved as planned for fiscal year 2000. The explanations for not achieving these targets were reasonable. For the launch-dependent targets, NASA reported that a schedule slip caused by Russian Proton failures and Service Module launch delays slowed down the entire space station assembly sequence. This, in turn, prevented NASA from achieving the launch-dependent targets in fiscal year 2000. NASA also reported that nonachievement for the one remaining target, “Complete the production of the X-38 first space flight test article in preparation for a Shuttle test flight in 2001,” was due to budget reductions. The NASA Advisory Council’s report indicated that the ISS program had a “productive year” after the schedule slip caused by the Russian Proton rocket failures, but it did not provide further elaboration.

In the fiscal year 2000 performance report, NASA identified reports that we and NASA’s OIG issued in fiscal year 2000 that addressed space station cost overruns and other issues. From the reports, NASA briefly summarized some of the concerns and corrective actions it agreed to take in relation to these issues, including space station cost growth. However, the agency did not set performance measures that more directly address cost growth, despite drastic increases in space station costs over the past several years and recent agency projections of potential cost overruns in excess of $4 billion. Furthermore, the issue of space station growth has been a long-standing problem. Although the NASA Advisory Council did not comment on the issue of cost-control measures for the space station in its evaluation of NASA’s performance, we continue to believe—as we have reported in the past—that NASA should develop performance measures that directly address space station cost-control issues, including risk mitigation and contingency planning activities.

NASA’s strategies and time frames for achieving the unmet targets were reasonable. For example, as of April 2001, NASA had launched four of the missions that were delayed in fiscal year 2000. The other two unachieved missions are also anticipated for launch in 200l. For the one target that was affected by budget reductions, “Complete the production of the X-38 first space flight test article in preparation for a Shuttle test flight in 2001,” the report stated that it would be achieved in fiscal year 2001. The shuttle test flight that was planned for September 2001 was extended to mid-2002.

Comparison of NASA’s Fiscal Year 2000 Performance Report With the Prior Year Report

For the selected key outcomes, this section describes major improvements or remaining weaknesses in NASA’s fiscal year 2000 performance report in comparison with its fiscal year 1999 report. It also discusses the degree to which the agency’s fiscal year 2000 report addresses concerns and recommendations by the Congress, GAO, NASA’s OIG, and others.

Comparison of Performance Reports for Fiscal Years 1999 and 2000

NASA’s portrayal of its verification and validation efforts applicable to all outcomes is an improvement over the fiscal year 1999 report, and it provides greater confidence that the performance results are credible. In our review of NASA’s fiscal year 1999 report, we criticized NASA for not describing procedures used to verify and validate performance information and addressing data limitation issues in the data. Unlike the prior year report, the fiscal year 2000 report provides a description of the methods used for verifying and validating the data as well as the data sources for each performance target. However, NASA can further improve the usefulness of the performance information by highlighting the limitations in the data and including steps it will take to correct them. This can be done by adding a separate data limitations narrative for each performance target and by using terminology such as “none” where there are no limitations in the performance data. Additionally, for a few performance targets, the reliability and validity of the performance information would be strengthened if the related verification and validation narratives were conveyed in a manner that would more clearly (1) demonstrate actual validation of the performance and (2) identify the validation methods. This is particularly true for the Earth Science Enterprise. This can be done by writing the narratives in a more convincing tone and in less technical language to enhance understanding of the validation approach.

NASA’s OIG report commended NASA for a significant improvement in the reporting of actual performance for fiscal year 2000. The OIG report found weaknesses in the accuracy and reliability of reported performance for 4 of 23 selected performance targets it reviewed in the fiscal year 2000 performance report. (NASA’s performance report includes a total of 211 performance targets). The OIG report indicated that based on OIG recommendations, NASA management made the necessary corrections or clarifications before issuing the fiscal year 2000 report.

For the most part, the performance targets continue to be output measures. Explanations are added to the performance targets as to why the performance results are meaningful. Generally, these explanations help to better understand the linkage between the targets and results. In our review of NASA’s fiscal year 1999 performance report, we noted that the continued use of output measures burdens the agency by requiring it to continuously demonstrate the linkages between program efforts and results and to make improvements to strengthen such linkages. Moreover, in its evaluation of NASA’s fiscal year 2000 performance, the NASA Advisory Council asked NASA to portray its performance in a way that is more understandable to the public. As previously mentioned, NASA’s fiscal year 2000 report adds explanations of why the performance results are meaningful. NASA acknowledges that developing annual outcome- related performance metrics for multiyear research and development programs is particularly challenging since these programs may not be mature enough to deliver outcome results for several years. The report notes that the stated objectives of programs within the agency are long- term in character. However, NASA also states in the report that it would continue to strive to meet the challenge of developing science and technology metrics that are outcome-oriented and useful in demonstrating how these outcomes benefit the public.

Lastly, in our review of the fiscal year 1999 report, we suggested that NASA document in its performance plans and reports the rationale for newly established performance targets to clarify the reasons for such targets. We had noted that while many of NASA’s performance targets were new each year, there was no stated basis for the changes. In its fiscal year 2000 report, NASA provides the rationale that in many cases, new targets are developed in response to program changes or as a result of experience gained in the performance planning process. NASA also includes charts at the end of each Enterprise and Crosscutting Process section that provide a trend assessment when a fiscal year 2000 target has a corresponding fiscal year 1999 target. Newly developed targets that have no corresponding fiscal year 1999 target to facilitate an assessment are characterized as “new target.”

NASA’s Efforts to Address its Major Management Challenges Identified by GAO

GAO has identified two governmentwide high-risk areas: strategic human capital management and information security. NASA’s performance report does not fully explain NASA’s progress in resolving human capital challenges. The report states that NASA has begun to focus on workforce renewal and revitalization, but it does not elaborate on strategies for undertaking this effort or address human capital challenges in other key areas. In addition, the report does not address the challenge of information security. Doing so is important for NASA. In 1999, we reported that the agency lacked an effective agencywide security program and that tests we conducted at one of NASA’s 10 field centers found that mission-critical information systems were vulnerable to unauthorized access.

In addition, we have identified three major management challenges facing NASA: (1) correcting contract management weaknesses, (2) controlling International Space Station costs, and (3) effectively implementing the faster-better-cheaper approach to space exploration projects. We found that NASA’s report addresses the problems of contract management and implementing the faster-better-cheaper approach. With respect to contract management, it is important to note that until NASA’s Integrated Financial Management System—-which is central to providing effective management and oversight over its procurement dollars—is operational, performance assessments relying on cost data may be incomplete, and full costing will be only partially implemented.

As we discussed under outcomes, NASA did not address the challenge of controlling space station costs. As we reported in January 2001, the International Space Station program continues to face cost-control challenges. As with contract management, until NASA’s Integrated Financial Management System is operational, NASA may lack the cost information needed to control space station costs.

Scope and Methodology

As agreed, our evaluation was generally based on the requirements of GPRA, the Reports Consolidation Act of 2000, guidance to agencies from the Office of Management and Budget (OMB) for developing performance plans and reports (OMB Circular A-11, Part 2), previous reports and evaluations by us and others, our knowledge of NASA’s operations and programs, our identification of best practices concerning performance planning and reporting, and our observations on NASA’s other GPRA- related efforts. We also discussed our review with NASA officials and with NASA’s OIG. The agency outcomes that were used as the basis for our review were identified by the Ranking Minority Member of the Senate Committee on Governmental Affairs as important mission areas for NASA and do not reflect the outcomes for all of NASA’s programs or activities. The major management challenges confronting NASA, including the governmentwide high-risk areas of strategic human capital management and information security, were identified in our January 2001 performance and accountability series and high risk update, and by NASA’s OIG in December 2000. We did not independently verify the information contained in the performance report, although we did draw from other GAO work in assessing the validity, reliability, and timeliness of NASA’s performance data. We conducted our review from April 2001 through June 2001 in accordance with generally accepted government auditing standards.

Agency Comments

In written comments on a draft of our report, NASA said that it had no issues with the report. NASA stated that as it develops its next performance plan, it is looking into the decreased use of output metrics, so as to focus more on outcomes. NASA also stated that it is reviewing its coverage of such areas as the International Space Station and information security in the performance plan. NASA’s comments are reproduced in appendix II.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies to appropriate congressional committees; the NASA Administrator; and the Director, Office of Management and Budget. Copies will also be made available to others on request.

If you or your staff have any questions, please call me at (202) 512-4841. Key contributors to this report were Richard J. Herley, Shirley B. Johnson, Charles W. Malphurs, Cristina T. Chaplain, John de Ferrari, Diane G. Handley, and Fannie M. Bivins.

Appendix I: Observations on NASA’s Efforts to Address Its Major Management Challenges

The following table identifies the major management challenges confronting NASA, including the governmentwide high-risk areas of strategic human capital management and information security. The first column of the table lists the management challenges that we and/or NASA’s Office of Inspector General (OIG) have identified. The second column discusses what progress, as discussed in its fiscal year 2000 performance report, NASA made in resolving its challenges. We found that NASA’s performance report discusses the agency’s progress in resolving many of its challenges but does not discuss progress in resolving the governmentwide challenge of information security. In addition, the report does not address the major management challenge: The Need to Control International Space Station Development and Support Costs.

Appendix II: Comments From the National Aeronautics and Space Administration