NASA: Status of Plans for Achieving Key Outcomes and Addressing Major Management Challenges
Released: 2001-12-27T00:00:00

Letter

Background

GPRA is intended to shift the focus of government decisionmaking, management, and accountability from activities and processes to the results and outcomes achieved by federal programs. New and valuable information on the plans, goals, and strategies of federal agencies has been provided since federal agencies began implementing GPRA. Under GPRA, annual performance plans are to clearly inform the Congress and the public of (1) the annual performance goals for agencies’ major programs and activities, (2) the measures that will be used to gauge performance, (3) the strategies and resources required to achieve the performance goals, and (4) the procedures that will be used to verify and validate performance information. These annual plans, issued soon after transmittal of the President’s budget, provide a direct linkage between an agency’s longer-term goals and mission and day-to-day activities. Annual performance reports are to report subsequently on the degree to which performance goals were met. The issuance of the agencies’ performance reports, due by March 31 of each year, represents a new and potentially more substantive phase in the implementation of GPRA—the opportunity to assess federal agencies’ actual performance for the prior fiscal year and to consider what steps are needed to improve performance, and reduce costs in the future. NASA’s final performance plan was provided to the Congress on July 17, 2001.

NASA’s mission encompasses human exploration and development of space, the advancement and communication of scientific knowledge, and research and development of aeronautical and space technologies. Its activities span a broad range of complex and technical endeavors—from investigating and evaluating the composition and resources of Mars; to working with international partners to complete and operate the International Space Station; to providing satellite and aircraft observations of earth for scientific and weather forecasting purposes; to developing new technologies designed to improve air flight safety.

Assessment of NASA’s Performance Goals and Strategies for Accomplishing Selected Key Outcomes

This section discusses our analysis of NASA’s performance goals and measures and strategies the agency has in place, particularly strategic human capital management and information technology, for accomplishing the outcomes. In discussing these outcomes, we have also provided information drawn from our prior work on the extent to which the agency provides assurance that its reported performance information will be credible.

Scientific Knowledge of the Earth System

NASA revised its strategic goal and most of its objectives for this key outcome in its fiscal year 2002 performance plan. We based our assessment on the strategic goal to observe, understand, and model the Earth system to learn how it is changing and the consequences of change for life on this planet. The previous goal was to expand scientific knowledge by characterizing the Earth system.

NASA’s performance plan does not explain why it adjusted its previous strategic goal and objectives, nor does it explain why it developed newly formulated annual performance goals and supporting indicators for fiscal year 2002. In discussing the reasons for these changes, NASA officials told us that changes in goals and measures were necessitated by the formulation of new strategic science questions for the Earth Science Enterprise and a refocused strategic plan. In our view, providing this explanation in the plan would have been useful. The performance plan includes a chart that displays annual performance goals and associated performance assessments for fiscal years 1999 to 2002, to help demonstrate cumulative progress towards achievement of strategic goals and objectives and to facilitate performance trend analysis. However, explaining changes in goals and measures over time would improve the performance trend analyses, and clarify the reasons for the new measures. On the other hand, by changing its performance goals annually, NASA could hinder its ability to make comparisons between fiscal years and effectively analyze trends in performance.

Generally, NASA’s fiscal year 2002 annual performance goals for this outcome appear to be objective and help to measure progress toward achieving it. Specific ways to measure the performance goals are established through two or more indicators that in many cases provide specific, quantifiable values that increase the measurability of the performance goal. An example of such an indicator is: “Increase the coverage of space-based maps of coral reef distribution by 25 percent beyond current estimates using remotely sensed imagery.” NASA added discussions on how each performance goal would benefit the public, as recommended by the NASA Advisory Council in its evaluation of NASA’s fiscal year 2000 performance report. In some cases these discussions clearly articulate the benefit to the public; in other cases they only provide descriptive or background information. For example, one performance goal calls for increasing the understanding of stratospheric ozone changes, as the abundance of ozone-destroying chemicals decreases and new substitutes increases. The public benefit statement, “Reduction in atmospheric ozone amounts leads to an increased flux of ultraviolet radiation at the Earth’s surface, with harmful effects on plant and animal life including human health,” explains the effect of reduced ozone amounts rather than how the goal will benefit the public.

NASA indicates that it will consider many of the Earth Science goals as fully met if a specified number of the supporting indicators (such as 3 out of 4) are achieved in fiscal year 2002. NASA officials told us that this approach allows for some flexibility in rating success. Specifically, since research and development by its very nature is unpredictable, these officials believe that, for example, not meeting all indicators still implies significant progress in achieving scientific goals. NASA could fully explain in the plan why it does not believe it has to meet all supporting indicators. This would put its actual performance in the proper perspective. The following is one example of such a goal: “Annual performance goal – Increase understanding of global precipitation, evaporation and how the cycling of water through the Earth system is changing by meeting at least 3 of 4 performance indicators.

Combine analysis of global water vapor, precipitation and wind data sets to decipher variations (and possible trends) in the cycling of water through the atmosphere and their relation to sea surface temperature changes.

Analyze data from polar and geostationary satellites in a consistent fashion over at least two decades to evaluate whether the detectable moisture fluxes are increasing beyond the expected ranges of natural variability.

Determine the time and spatial variability of the occurrence of strong convection regions, precipitation events, and areas of drought to assess whether or not there are discernable global changes in the distribution of moisture availability useful to food and fiber production and management of fresh water resources. Establish passive and active rainfall retrievals of zonal means to establish a calibration point for long-term data records of the World Climate Research Program, Global Precipitation Climatology Project (GPCP).”

Concerning interagency and crosscutting activities, we note that within this outcome, NASA does not include annual performance goals and indicators that reflect programs or activities being undertaken with other agencies in fiscal year 2002 for the strategic goal covered by our review, even though NASA’s latest Strategic Plan identifies 16 federal agencies that contribute to this goal. These agencies include the departments of Defense and Commerce, the National Oceanic and Atmospheric Administration, and the Federal Emergency Management Agency. If NASA has planned collaborative efforts related to the performance goals and indicators for this outcome, these are not identified in the performance plan. The performance plan provides an opportunity to evidence coordination among crosscutting programs and reflect the expected contribution of other agencies toward related goals.

NASA states that its implementation strategy for Earth Science research programs is focused on a set of strategic science questions directed at understanding how the Earth system is changing and the consequences for life on Earth. The plan indicates that these questions can be addressed effectively with NASA’s capabilities, which include observational programs, research and analysis, modeling, and advanced technology. In general, the plan provides clear and reasonable information technology. One annual performance goal within this outcome is to successfully disseminate Earth Science data to enable NASA’s Earth Science research and applications goals and objectives. To achieve this goal, NASA set several specific performance indicators, such as increasing the number of distinct NASA Earth Observing System Data and Information System (EOSDIS) customers by 20 percent compared to fiscal year 2001; increasing scientific and applications-data products delivered from the Earth Observing System (EOS) Distributed Active Archive Centers (DAAC) by 10 percent compared to fiscal year 2001; and increasing the number of favorable comments from DAAC and other users over fiscal year 2001 and decreasing the total percentage of order errors by 5 percent over fiscal year 2001. These indicators provide specific, quantifiable ways to measure increases in output from NASA’s EOSDIS and DAACs. Based on NASA’s reported success in meeting similar indicators for fiscal year 2000, these indicators appear to be reasonable for fiscal year 2002.

Commercial Development of Space

In some instances, NASA revised or added strategic goals and objectives, and annual performance goals within this outcome for fiscal year 2002. For example, the agency added (1) a new strategic goal to provide commercial industry with the opportunity to meet NASA’s future launch needs, including human access to space, with new launch vehicles that promise to dramatically reduce cost and improve safety and reliability and (2) a new strategic objective to develop new capabilities for human space flight and commercial applications through partnerships with the private sector. Furthermore, most of the annual performance goals are either new or revised from targets in NASA’s prior year performance plan. Further, NASA does not provide any rationale or reasons for the changes in the plan. NASA officials told us that, generally, the changes were made to (1) improve NASA’s ability to assess progress toward achieving goals and objectives; (2) reflect commitment to safety and privatization efforts; or (3) reflect the broader scope of programs and activities and shifts in Enterprise responsibilities. Again, providing such explanations in the performance plan, in our view, would have been useful.

NASA displays its annual performance goals and associated performance assessments for fiscal years 1999 to 2002 to help demonstrate cumulative progress towards achievement of strategic goals and objectives and to facilitate performance trend analysis. As emphasized earlier, changing performance goals annually could hinder NASA’s ability to make comparisons between years and effectively analyze trends in performance. NASA generally presents performance goals that appear to be objective and help to measure progress toward this outcome. The agency also explains how each performance goal will benefit the public.

The strategies for achieving the performance goals are generally clear and reasonable. However, there are some exceptions. For example, one performance goal involves developing and testing – on the ground and in space—competing technologies for human missions beyond low earth orbit in cooperation with international partners. One indicator related to this performance goal involves organizing and conducting an “international forum” at which preliminary concepts, plans, and technology options for future human/robotic exploration and development of space would be reviewed. However, the indicator does not address the testing of competing technologies. Also, NASA has a performance goal to engage the commercial community and encourage non-NASA investment in commercial space research by meeting at least three of four performance indicators, but the plan does not state why all of the supporting indicators will not be achieved.

International Space Station’s Operation, Cost, and Safety

Since the selected key outcome of deploying and operating the space station safely and cost effectively is not included in NASA’s fiscal year 2002 performance plan as a specific strategic goal or objective, we based our assessment of it on a related strategic objective in the plan—to operate the space station to advance science, exploration, engineering, and commerce.

NASA set four annual performance goals for this outcome for fiscal year 2002. The performance goals are new, but the plan does not provide any rationale for the changes. This is an important omission, because as pointed out earlier, explaining changes in goals and measures over several years would improve performance trend analyses and clarify why such changes were made. In discussing the reasons for these changes, NASA officials told us that for the International Space Station, the fiscal year 2001 goals and objectives relied on milestones that were reported as either complete or incomplete, with no provision for reporting progress toward completion. The improved goals and objectives for fiscal year 2002 are tied to milestones that allow reporting of progress in terms of the percent of the milestones completed. Thus, the new measures will provide greater visibility and improve NASA’s ability to assess progress toward achieving goals and objectives. In our view, providing this explanation in the plan would have enhanced NASA’s discussion on this outcome. Also, similar to the previous outcomes, NASA displays performance goals and associated assessments for fiscal years 1999 to 2002 to help demonstrate cumulative progress towards achievement of strategic goals and objectives and to facilitate performance trend analysis. However, changing performance goals annually could hinder NASA’s ability to make comparisons between fiscal years and effectively analyze trends in performance.

Generally, NASA’s four annual performance goals and supporting indicators for the space station outcome appear to be objective and help to measure progress toward this outcome. In addition, the agency explains how the goals will benefit the public, stating how completing them successfully will provide many benefits of space research through new discoveries and improved technological applications in areas such as medicine, industrial processes, and fundamental knowledge. One performance goal addresses space station safety. Specifically, the goal is to demonstrate space station on-orbit vehicle’s operational safety, reliability, and performance. The goal has an indicator that provides for zero safety incidents (such as no on-orbit injuries), which appears reasonable. The other indicator is not articulated understandably, making it difficult to ascertain its relationship to the performance goal or to assess its measurability. (The language is phrased as: “Actual resources available to the payloads measured against the planned payload allocation for power, crew time, and telemetry.”) Also, the plan does not clearly indicate the means or strategies NASA will use to ensure that the safety performance goal is achieved in fiscal year 2002. Similarly, the plan does not provide clear strategies for achieving the remaining three performance goals of (1) demonstrating space station progress and readiness at a level sufficient to show adequate readiness in the assembly schedule, (2) successfully completing 90% of the space station’s planned mission objectives, and (3) demonstrating progress toward space station research hardware development.

NASA does not address space station cost control as part of this outcome. However, within its commercialization of space outcome, NASA set a performance goal in fiscal year 2002 to develop and execute a management plan and open future ISS hardware and service procurements to cost-effective innovation through competition, including launch services and a non-governmental organization for space station research. NASA’s indicator for the management plan includes reforms that (1) strengthen its headquarters involvement, (2) increase communications, (3) provide more accurate assessment, and (4) maintain budget accountability. NASA reports that the benefit to the public of the management plan and reforms is to ensure that future space station costs will remain within the President’s fiscal year 2002 budget plan. In our view, NASA’s discussion of the proposed management plan is minimal and lacks specificity. While the management plan will reportedly include ISS budget accountability reforms, NASA does not elaborate on the nature of such reforms or indicate to what extent this plan will address space station cost growth, a long-standing management problem. Furthermore, NASA does not acknowledge anywhere in the performance plan that space station cost control is a major management challenge, although it has done so for some of the other challenges. In past years and as recently as January 2001, we have identified the need to control space station costs as a major management challenge for NASA. We believe that the agency has the opportunity to use the completed management plan to facilitate the development of space station cost control measures in future annual performance plans. The lack of performance measures that address space station cost control is a shortcoming that we have identified in our previous reviews of the agency’s annual performance plans and reports.

Comparison of NASA’s Fiscal Year 2002 Performance Plan With the Prior Year Plan

For the selected key outcomes, this section describes major improvements or remaining weaknesses in NASA’s fiscal year 2002 performance plan in comparison with its fiscal year 2001 performance plan. It also discusses the degree to which the agency’s fiscal year 2002 plan addresses concerns and recommendations by the Congress, GAO, NASA’s OIG, and others.

Comparison of Performance Plans for Fiscal Years 2001 and 2002

NASA’s fiscal year 2002 performance plan differs in several significant ways from the prior plan. First, NASA portrays its planned efforts to verify and validate performance information more comprehensively than in 2001, providing greater confidence that the performance results will be credible. In our review of NASA’s 2001 plan, we criticized the agency for not explicitly describing those efforts and for not addressing data limitation issues and problems. The 2002 plan includes specific agency data bases and describes methods NASA will rely on to support the credibility of reported performance information. For example, the plan references the NASA Personnel Payroll System, Incident Reporting System, Financial and Contractual Status of Programs System, and NASA Environmental Tracking System as specific data bases that will be used to verify and validate performance data. The plan describes specific processes in place to support performance claims associated with NASA’s Integrated Financial Management System, performance-based contracts, contracts awarded to small and small disadvantaged businesses, and information technology. And it describes a broad array of methods to verify and validate reported performance data such as monthly reports from NASA field centers, Web statistics, count of publications, and NASA’s Education Computer Aided Tracking System. Despite improvements in addressing data verification and validation methods, NASA still does not acknowledge data limitations that could hinder performance measurement. We continue to believe that NASA can further enhance the credibility of its verification and validation procedures and the usefulness of its performance data by disclosing the expected limitations of its performance data in its annual performance plans. A March 2001 NASA Office of Inspector General report identified limitations in NASA’s fiscal year 2000 performance data and indicated that NASA would discuss anticipated data limitations in its performance planning beginning with its fiscal year 2002 final performance plan. However, we reviewed the final version of the plan, and such a discussion is not included.

Second, several added features help to enhance the format and/or content of the fiscal year 2002 plan. NASA’s use of “annual performance goals” in the plan characterizes its annual performance measures more clearly than the “annual performance targets,” used in previous plans. The addition of discussions on how annual performance goals benefit the public helps to better understand the linkage between the goal and the expected results, although in some cases additional clarification could even better convey the actual benefit to the public. Value is added to the plan by NASA’s display of annual performance goals and associated performance assessments for fiscal years 1999 to 2002, to help demonstrate cumulative progress towards achievement of strategic goals and objectives and facilitate performance trend analysis. However, changes in performance goals over many years could hinder NASA’s ability to make comparisons between years and effectively analyze trends in performance.

Also, this year’s plan includes an agencywide strategic objective to invest in the use of human capital. NASA set two annual performance goals for fiscal year 2002 as progress towards this objective: (1) align management of human resources to best achieve agency strategic goals and (2) attract and retain a workforce that is representative at all levels of America’s diversity. However, there are no human capital initiatives specifically linked to the outcomes or annual performance goals and indicators that link to specific programs, such as the space shuttle program. (See details under management challenges.)

Third, NASA could explain in the plan what procedures it has used to characterize its performance goals as fully achieved when it has not met all of the supporting indicators for those goals. This is particularly true for the Earth Science outcome. Providing such an explanation would put the actual performance in the proper perspective.

Fourth, similar to the prior plan, the fiscal year 2002 plan still does not provide a clear rationale for how information technology-related strategies and programs will contribute specifically to achievement of NASA’s goals or show any allocation of information technology-related dollars and personnel to performance goals. Goals for managing information technology are generally stated in terms of broad categories for improvement, such as increased capability and efficiency and enhanced security, and include few quantitative measures. One exception is the goal of increasing dissemination of Earth Science data, which is accomplished through EOSDIS. The plan sets several specific goals for increasing the volume and distribution of Earth Science data and products.

Lastly, in our review of NASA’s fiscal year 2001 plan, we suggested that NASA document in its annual performance plans and reports, the rationale for establishing new performance targets to clarify the reasons for adding such targets. We had noted that while many of NASA’s annual performance targets were new each year, there was no stated basis for the changes. In its fiscal year 2002 performance plan, NASA has formulated new annual performance goals and has changed many of its strategic goals and objectives without including the reasons for doing so. We continue to believe that providing the rationale for these changes will clarify the reasons for the new goals and measures and augment the value of performance trend analyses. Also, the plan does not indicate whether or not achieving any specific goals would be negatively affected by external factors. However, like the prior plan, the fiscal year 2002 plan states that successful execution of NASA’s strategic goals and objectives depends on receipt of its requested appropriations, as well as provision of funds, materials, or services, that have been committed to the cooperative agreements or partnerships which are referenced in the performance plan.

NASA’s Efforts to Address its Major Management Challenges Identified by GAO

We have identified two governmentwide high-risk areas: strategic human capital management and information security. Regarding strategic human capital management, NASA’s fiscal year 2002 performance plan contains a strategic objective and annual performance goals and indicators directly related to human capital. Concerning information security, NASA’s performance plan contains a strategic objective, an annual performance goal, and indicators directly related to this management challenge. The plan states that safety and security is one of four areas on which NASA’s information technology planning is focused. The fiscal year 2002 plan is an improvement over the 2001 plan, which did not include quantifiable measures for improving information security. However, the plan’s performance goals do not fully respond to the recommendations we made in 1999 when we reported that the agency lacked an effective agencywide security program. For example, the plan sets a performance indicator of completing 90 percent of information technology security plans for critical systems. However, we recommended that all systems be formally authorized before they became operational and at least every 3 years thereafter.

In addition, we have identified three major management challenges facing NASA: (1) correcting contract management weaknesses, (2) controlling International Space Station costs, and (3) effectively implementing the faster-better-cheaper approach to space exploration projects. We found that NASA’s performance plan contains an annual performance goal and indicators directly related to the problem of contract management. It is important to note that until NASA’s Integrated Financial Management System—-which is central to providing effective management and oversight over its procurement dollars—is operational, performance assessments relying on cost data may be incomplete and full costing will be only partially implemented.

While NASA’s performance plan contains an annual performance goal and an indicator that indirectly addresses the challenge of controlling space station costs, it does not indicate the extent that NASA will address space station cost growth. As we discussed in our January 2001 report, the International Space Station Program continues to face cost control challenges. As with contract management, until the Integrated Financial Management System is operational, NASA may lack the cost information needed to control space station costs.

Further, NASA’s performance plan did not directly address the challenge of effectively implementing the faster-better-cheaper approach to space exploration projects. In January 2001, we also reported that NASA faces significant challenges as it attempts to create highly reliable missions and foster open communications under the budget constraints of the agency’s faster-better-cheaper space exploration strategy. In addition, the real success of this strategy will require a comprehensive integration of lessons learned from failures on an agencywide basis. Until NASA resolves these problems, its financial resources are vulnerable to inefficient use. Appendix I provides detailed information on how NASA addressed these challenges and high-risk areas as identified by GAO and NASA’s Office of Inspector General (OIG).

Scope and Methodology

As agreed, our evaluation was generally based on the requirements of GPRA, guidance to agencies from the Office of Management and Budget (OMB) for developing performance plans (OMB Circular A-11, Part 2), previous reports and evaluations by us and others, our knowledge of NASA’s operations and programs, our identification of best practices concerning performance planning, and our observations on NASA’s other GPRA-related efforts. We also discussed our review with NASA officials and with officials of NASA’s OIG. The agency outcomes that were used as the basis for our review were identified by the Ranking Minority Member of the Senate Committee on Governmental Affairs as important mission areas for NASA and do not reflect the outcomes for all of NASA’s programs or activities. The major management challenges confronting NASA, including the governmentwide high-risk areas of strategic human capital management and information security, were identified in our January 2001 performance and accountability series and high risk update, and by NASA’s OIG in December 2000. We conducted our review from August 2001 through October 2001 in accordance with generally accepted government auditing standards.

Agency Comments

We provided copies of a draft of this report to NASA for its review and comment. In written comments on the report, NASA generally agreed with the information presented in the report and noted several improvements it would make. Concerning our suggestion that NASA could fully explain in its performance plan why it believes it is not necessary to achieve all performance indicators to demonstrate annual performance goal achievement, NASA stated that it would provide a statement containing the supporting rationale for this approach in its fiscal year 2003 performance plan.

In responding to our observation that the fiscal year 2002 plan lacked sufficient detail on the nature of the ISS’s budget accountability reforms or how the reforms will address longstanding and ongoing management problems, including cost growth, NASA commented that the reforms are contained in its Program Management Action Plan that will be referred to in the fiscal year 2003 performance plan. We note that the ISS Program is being restructured in response to a potential cost growth of $4.8 billion. The restructuring has raised widespread concerns about the potential science benefits to be realized by the United States and international partners. For this reason, we believe it is increasingly important for NASA’s performance plan to provide a clear path showing how NASA intends to implement the needed reforms and how the reforms will add credibility to future ISS budgets and resolve the uncertainties concerning the utility of the ISS.

NASA also commented on a statement in our draft report that the agency does not provide a clear rationale for how IT-related strategies and programs will contribute specifically to achievement of its goals or show the allocation of IT-related dollars and personnel to performance goals. According to NASA, the IT service delivery metric in the plan aggregates each major IT service, such as NASA’s Integrated Services Network. Remaining IT investments are embedded in each NASA project and managed as part of the project. While this statement may be true on the individual program level, it does not address GPRA objectives to demonstrate how IT-related strategies and programs contribute specifically to the achievement of agency goals or show the allocation of related resources.

Finally, in response to our observation regarding NASA’s lack of explanations in the 2002 plan for annual performance changes, NASA agreed that including such explanations in the plan would be useful and that it would characterize reasons for annual performance changes in its 2003 performance plan.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies to appropriate congressional committees; the NASA Administrator; and the Director, Office of Management and Budget. Copies will also be made available to others on request.

If you or your staff have any questions, please call me at (202) 512-4841. Key contributors to this report were Richard J. Herley, Shirley B. Johnson, Charles W. Malphurs, Christina Chaplain, John de Ferrari, Diane G. Handley, and Fannie M. Bivins.

Appendix I: Observations on NASA’s Efforts to Address Its Major Management Challenges

The following table identifies the major management challenges confronting NASA, including the governmentwide high-risk areas of strategic human capital management and information security. The first column of the table lists the management challenges that we and/or NASA’s Office of Inspector General (OIG) have identified. The second column discusses the extent to which NASA’s fiscal year 2002 performance plan includes performance goals and measures to address the challenges that we and the OIG identified. Of the agency’s fifteen major management challenges, its performance plan has (1) goals and measures that are directly related to thirteen of the challenges; (2) a goal and measure indirectly applicable to one challenge; and (2) no goals and measures directly related to one of the challenges.

Some of the NASA performance plan’s goals and measures we discuss may not track specifically with the key considerations of NASA OIG’s management challenges since the challenges themselves were presented in a broad context. GAO has performed reviews affecting a number of the areas mentioned. This appendix highlights the results of our assessments, where applicable.

Appendix II: Comments From the National Aeronautics and Space Administration