Report to the Ranking Minority

Member, Committee on Governmental

Affairs, U.S. Senate

United States General Accounting Office

GAO

November 2001 NASA

Status of Plans for

Achieving Key

Outcomes and

Addressing Major

Management

Challenges

GAO-02-184
Page i GAO-02-184 Status of Plans for Achieving Key Outcomes

Letter 1

Results in Brief 2

Background 5

Assessment of NASA’s Performance Goals and Strategies for

Accomplishing Selected Key Outcomes 6

Comparison of NASA’s Fiscal Year 2002 Performance Plan With the

Prior Year Plan 12

NASA’s Efforts to Address its Major Management Challenges

Identified by GAO 15

Scope and Methodology 16

Agency Comments 16

Appendix I Observations on NASA’s Efforts to Address Its Major

Management Challenges 19

Appendix II Comments From the National Aeronautics and Space

Administration 33

Tables

Table 1: Major Management Challenges 20

Contents
Page 1 GAO-02-184 Status of Plans for Achieving Key Outcomes

November 27, 2001

The Honorable Fred Thompson

Ranking Minority Member

Committee on Governmental Affairs

United States Senate

Dear Senator Thompson:

As you requested, we reviewed the National Aeronautics and Space

Administration’s (NASA) fiscal year 2002 performance plan required by the

Government Performance and Results Act of 1993 (GPRA)1

 to assess the

agency’s planned performance for the selected key outcomes that you

identified as important mission areas for the agency.2

 These are the same

outcomes we addressed in our June 2000 review of NASA’s fiscal year 1999

performance report and fiscal year 2001 performance plan to provide a

baseline by which to measure the agency’s performance from year to year.3

We also addressed these outcomes in our July 2001 review of NASA’s

fiscal year 2000 performance report.4

 As agreed with your office, we are

issuing this separate report on our assessment of the selected outcomes in

NASA’s fiscal year 2002 performance plan. The selected key outcomes are

to

• expand scientific knowledge of the Earth system,

• expand the commercial development of space, and

• deploy and operate the International Space Station (ISS) safely and cost

effectively.5

 1

P.L. 103-62, 107 Stat. 285.

2

This report is one of a series of reports on the 24 Chief Financial Officers (CFO) Act of

1990 (P.L. 101-576, 104 Stat. 2838) agencies’ fiscal year 2000 performance reports and fiscal

year 2002 performance plans.

3

Observations on the National Aeronautics and Space Administration’s Fiscal Year 1999

Performance Report and Fiscal Year 2001 Performance Plan (GAO/NSIAD-00-192R, June

30, 2000).

4

NASA: Status of Achieving Key Outcomes and Addressing Major Management

Challenges (GAO-01-868, July 31, 2001).

5

NASA’s fiscal year 2002 performance plan identifies an objective closely related to this key

outcome. That objective is to operate the space station to advance science, exploration,

engineering, and commerce. We based our assessment on that objective.

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-02-184 Status of Plans for Achieving Key Outcomes

As agreed, using the selected key outcomes for NASA as a framework, we

(1) assessed NASA’s planned progress toward achieving these outcomes,

and the strategies the agency has in place to achieve them; and (2)

compared NASA’s fiscal year 2002 performance plan with the agency’s

prior year performance plan for these outcomes. Additionally, we agreed

to analyze how NASA addressed its major management challenges,

including the governmentwide high-risk areas of strategic human capital

management and information security, that we and NASA’s Office of

Inspector General (OIG) identified. Appendix I provides detailed

information on how NASA addressed these challenges. Appendix II

contains NASA’s comments on a draft of our report.

NASA has improved its fiscal year 2002 performance plan and responded

to recommendations or suggestions by us and others to make its plan

more useful—particularly by providing more comprehensive explanations

on how it plans to verify and validate performance data and by better

portraying how its performance goals will benefit the public. Generally,

NASA’s annual performance goals for its outcomes appear to be objective

and help to measure progress toward the outcomes. However, the plan

still does not explain the reasons for changes in performance goals. Not

having these explanations could hinder the ability to assess NASA’s

performance over time.

• Planned outcome: Expanding scientific knowledge of the Earth system.

NASA’s annual performance goals for this outcome appear objective; and

many of the supporting performance indicators increase the measurability

of the performance goals. Also, in response to the NASA Advisory

Council’s specific recommendation, NASA explains how each

performance goal will benefit the public. However, these performance

goals and indicators do not reflect programs and activities being

undertaken with other agencies for the strategic goal, “observe,

understand, and model the Earth system to learn how it is changing and

the consequences for life on Earth,” even though NASA’s recently updated

Strategic Plan identifies 16 federal partners contributing to this goal.6

 6

We based our assessment of the Earth Science outcome on this strategic goal. The

performance plan does reflect collaboration with other partners for one strategic goal

within this outcome not covered by our review.

Results in Brief
Page 3 GAO-02-184 Status of Plans for Achieving Key Outcomes

NASA’s strategies for achieving this outcome appear to be clear and

reasonable.

• Planned outcome: Expanding the commercial development of space.

NASA generally presents performance goals that appear objective and

measurable for this outcome and explains how each will benefit the

public. Although the agency’s strategies for meeting these performance

goals are generally clear and reasonable, there are some exceptions. For

example, the goal to develop and test competing technologies for human

missions beyond low earth orbit in cooperation with international partners

does not address how NASA will actually test those technologies.

• Planned outcome: Deploying and operating the International Space Station

safely and cost-effectively.

NASA’s performance goals for this outcome appear objective and

measurable, and NASA explains how they will benefit the public. However,

one of the indicators for the safety performance goal is not articulated

understandably, which makes it difficult to ascertain its relationship to the

performance goal or assess its measurability. Also, the plan does not

clearly indicate how NASA will ensure that the safety goal is achieved, nor

does it show how the agency will achieve the remaining three performance

goals of (1) demonstrating space station progress and readiness at a level

sufficient to show adequate readiness in the assembly schedule,

(2) successfully completing 90% of the space station’s planned mission

objectives, and (3) demonstrating progress toward developing space

station research hardware. NASA does not sufficiently elaborate on the

nature of its space station budget accountability reforms mentioned in the

plan, nor how the reforms will address space station cost growth, a longstanding and ongoing management problem. Furthermore, it does not

acknowledge anywhere in the plan that the International Space Station

cost control issue is a management challenge—although it does so for

some of the other challenges.

NASA has made improvements in its fiscal year 2002 performance plan

when compared to its 2001 plan. In addition to its verification and

validation efforts, NASA adds several features that enhance the format

and/or content of its performance plan, including a better characterization

of its annual performance measures as “goals” instead of “targets.” It also

adds discussions on how its annual performance goals will benefit the

public and displays annual performance goals and associated performance

assessments for fiscal years 1999 to 2002, to help demonstrate cumulative
Page 4 GAO-02-184 Status of Plans for Achieving Key Outcomes

progress towards achievement of strategic goals and objectives. Moreover,

NASA adds an agencywide human capital objective and associated annual

performance goals and indicators to the plan. However, NASA can further

enhance the credibility of its verification and validation efforts by being

more forthcoming about possible limitations in its performance data.

Furthermore, it can relate its human capital performance goals and

indicators to specific programs where issues of critical staffing shortages

have been identified. It can also enhance understanding of its rationale for

characterizing successful achievement despite not meeting all supporting

performance indicators for its goals by including in the plan an

explanation for its decision to use this approach. And it can provide a

clear rationale for how information technology-related strategies and

programs will contribute specifically to the achievement of its goals and

show the allocation of information technology-related dollars and

personnel to performance goals.

In assessing how NASA addressed major management challenges, we

found the agency’s performance plan has strategic objectives and

performance goals and measures related to two of our governmentwide

high-risk areas—strategic human capital management and information

security. While NASA’s inclusion of human capital as a strategic objective

is an improvement, it did not tie its performance goals and indicators to

specific programs facing human capital shortages, such as the space

shuttle program. Regarding information security, the plan’s performance

goals are not fully responsive to recommendations we made in our 1999

report.

In addition, we identified three other challenges facing NASA: (1)

correcting contract management weaknesses, (2) controlling International

Space Station costs, and (3) effectively implementing the faster-bettercheaper approach to space exploration projects. We found that NASA’s

performance plan has an annual performance goal and measures directly

related to the challenge of correcting contract management weaknesses.

However, until NASA’s Integrated Financial Management System—which

is central to providing effective management and oversight over its

procurement dollars—is operational, performance and cost assessments

may be incomplete. The plan contains an annual performance goal and

indicator that are indirectly related to the challenge of controlling

International Space Station costs. But the plan does not indicate the extent

NASA will address space station cost growth. Furthermore, the plan does

not include goals and measures directly related to effective

implementation of the faster-better-cheaper approach to space exploration

projects.
Page 5 GAO-02-184 Status of Plans for Achieving Key Outcomes

We provided copies of a draft of this report to NASA for its review and

comment. In written comments on the report, NASA generally agreed with

the information presented in the report and noted several improvements it

would make to its fiscal year 2001 performance report and/or fiscal year

2003 performance plan. NASA’s written comments are included as

appendix II.

GPRA is intended to shift the focus of government decisionmaking,

management, and accountability from activities and processes to the

results and outcomes achieved by federal programs. New and valuable

information on the plans, goals, and strategies of federal agencies has been

provided since federal agencies began implementing GPRA. Under GPRA,

annual performance plans are to clearly inform the Congress and the

public of (1) the annual performance goals for agencies’ major programs

and activities, (2) the measures that will be used to gauge performance,

(3) the strategies and resources required to achieve the performance goals,

and (4) the procedures that will be used to verify and validate

performance information. These annual plans, issued soon after

transmittal of the President’s budget, provide a direct linkage between an

agency’s longer-term goals and mission and day-to-day activities.7

 Annual

performance reports are to report subsequently on the degree to which

performance goals were met. The issuance of the agencies’ performance

reports, due by March 31 of each year, represents a new and potentially

more substantive phase in the implementation of GPRA—the opportunity

to assess federal agencies’ actual performance for the prior fiscal year and

to consider what steps are needed to improve performance, and reduce

costs in the future.8

 NASA’s final performance plan was provided to the

Congress on July 17, 2001.

NASA’s mission encompasses human exploration and development of

space, the advancement and communication of scientific knowledge, and

research and development of aeronautical and space technologies. Its

activities span a broad range of complex and technical endeavors—from

investigating and evaluating the composition and resources of Mars; to

working with international partners to complete and operate the

International Space Station; to providing satellite and aircraft observations

 7

The fiscal year 2002 performance plan is the fourth of these annual plans under GPRA.

8

The fiscal year 2000 performance report is the second of these annual reports under

GPRA.

Background
Page 6 GAO-02-184 Status of Plans for Achieving Key Outcomes

of earth for scientific and weather forecasting purposes; to developing

new technologies designed to improve air flight safety.

This section discusses our analysis of NASA’s performance goals and

measures and strategies the agency has in place, particularly strategic

human capital management9

 and information technology, for

accomplishing the outcomes. In discussing these outcomes, we have also

provided information drawn from our prior work on the extent to which

the agency provides assurance that its reported performance information

will be credible.

NASA revised its strategic goal and most of its objectives for this key

outcome in its fiscal year 2002 performance plan. We based our

assessment on the strategic goal to observe, understand, and model the

Earth system to learn how it is changing and the consequences of change

for life on this planet. The previous goal was to expand scientific

knowledge by characterizing the Earth system.

NASA’s performance plan does not explain why it adjusted its previous

strategic goal and objectives, nor does it explain why it developed newly

formulated annual performance goals and supporting indicators for fiscal

year 2002. In discussing the reasons for these changes, NASA officials told

us that changes in goals and measures were necessitated by the

formulation of new strategic science questions for the Earth Science

Enterprise and a refocused strategic plan. In our view, providing this

explanation in the plan would have been useful. The performance plan

includes a chart that displays annual performance goals and associated

performance assessments for fiscal years 1999 to 2002, to help

demonstrate cumulative progress towards achievement of strategic goals

and objectives and to facilitate performance trend analysis. However,

explaining changes in goals and measures over time would improve the

performance trend analyses, and clarify the reasons for the new measures.

On the other hand, by changing its performance goals annually, NASA

 9

Key elements of modern strategic human capital management include strategic human

capital planning and organizational alignment; leadership continuity and succession

planning; acquiring and developing staffs whose size, skills, and deployment meet agency

needs; and creating results-oriented organizational cultures.

Assessment of NASA’s

Performance Goals

and Strategies for

Accomplishing

Selected Key

Outcomes

Scientific Knowledge of

the Earth System
Page 7 GAO-02-184 Status of Plans for Achieving Key Outcomes

could hinder its ability to make comparisons between fiscal years and

effectively analyze trends in performance.

Generally, NASA’s fiscal year 2002 annual performance goals for this

outcome appear to be objective and help to measure progress toward

achieving it. Specific ways to measure the performance goals are

established through two or more indicators that in many cases provide

specific, quantifiable values that increase the measurability of the

performance goal. An example of such an indicator is: “Increase the

coverage of space-based maps of coral reef distribution by 25 percent

beyond current estimates using remotely sensed imagery.” NASA added

discussions on how each performance goal would benefit the public, as

recommended by the NASA Advisory Council in its evaluation of NASA’s

fiscal year 2000 performance report. In some cases these discussions

clearly articulate the benefit to the public; in other cases they only provide

descriptive or background information. For example, one performance

goal calls for increasing the understanding of stratospheric ozone changes,

as the abundance of ozone-destroying chemicals decreases and new

substitutes increases. The public benefit statement, “Reduction in

atmospheric ozone amounts leads to an increased flux of ultraviolet

radiation at the Earth’s surface, with harmful effects on plant and animal

life including human health,” explains the effect of reduced ozone amounts

rather than how the goal will benefit the public.

NASA indicates that it will consider many of the Earth Science goals as

fully met if a specified number of the supporting indicators (such as 3 out

of 4) are achieved in fiscal year 2002. NASA officials told us that this

approach allows for some flexibility in rating success. Specifically, since

research and development by its very nature is unpredictable, these

officials believe that, for example, not meeting all indicators still implies

significant progress in achieving scientific goals. NASA could fully explain

in the plan why it does not believe it has to meet all supporting indicators.

This would put its actual performance in the proper perspective. The

following is one example of such a goal:

“Annual performance goal – Increase understanding of global

precipitation, evaporation and how the cycling of water through the

Earth system is changing by meeting at least 3 of 4 performance

indicators.
Page 8 GAO-02-184 Status of Plans for Achieving Key Outcomes

Indicators

• Combine analysis of global water vapor, precipitation and wind data

sets to decipher variations (and possible trends) in the cycling of

water through the atmosphere and their relation to sea surface

temperature changes.

• Analyze data from polar and geostationary satellites in a consistent

fashion over at least two decades to evaluate whether the

detectable moisture fluxes are increasing beyond the expected

ranges of natural variability.

• Determine the time and spatial variability of the occurrence of

strong convection regions, precipitation events, and areas of

drought to assess whether or not there are discernable global

changes in the distribution of moisture availability useful to food

and fiber production and management of fresh water resources.

• Establish passive and active rainfall retrievals of zonal means to

establish a calibration point for long-term data records of the World

Climate Research Program, Global Precipitation Climatology

Project (GPCP).”

Concerning interagency and crosscutting activities, we note that within

this outcome, NASA does not include annual performance goals and

indicators that reflect programs or activities being undertaken with other

agencies in fiscal year 2002 for the strategic goal covered by our review,

even though NASA’s latest Strategic Plan identifies 16 federal agencies that

contribute to this goal.10 These agencies include the departments of

Defense and Commerce, the National Oceanic and Atmospheric

Administration, and the Federal Emergency Management Agency. If NASA

has planned collaborative efforts related to the performance goals and

indicators for this outcome, these are not identified in the performance

plan. The performance plan provides an opportunity to evidence

coordination among crosscutting programs and reflect the expected

contribution of other agencies toward related goals.

NASA states that its implementation strategy for Earth Science research

programs is focused on a set of strategic science questions directed at

understanding how the Earth system is changing and the consequences for

 10The performance plan does include an objective and annual performance goals that

reflect collaboration with other agencies for one strategic goal within this outcome not

covered by our review. That strategic goal is to develop and adopt advanced technologies

to enable mission success and serve national priorities.
Page 9 GAO-02-184 Status of Plans for Achieving Key Outcomes

life on Earth. The plan indicates that these questions can be addressed

effectively with NASA’s capabilities, which include observational

programs, research and analysis, modeling, and advanced technology. In

general, the plan provides clear and reasonable information technology.

One annual performance goal within this outcome is to successfully

disseminate Earth Science data to enable NASA’s Earth Science research

and applications goals and objectives. To achieve this goal, NASA set

several specific performance indicators, such as increasing the number of

distinct NASA Earth Observing System Data and Information System

(EOSDIS) customers by 20 percent compared to fiscal year 2001;

increasing scientific and applications-data products delivered from the

Earth Observing System (EOS) Distributed Active Archive Centers

(DAAC) by 10 percent compared to fiscal year 2001; and increasing the

number of favorable comments from DAAC and other users over fiscal

year 2001 and decreasing the total percentage of order errors by 5 percent

over fiscal year 2001. These indicators provide specific, quantifiable ways

to measure increases in output from NASA’s EOSDIS and DAACs. Based

on NASA’s reported success in meeting similar indicators for fiscal year

2000, these indicators appear to be reasonable for fiscal year 2002.

In some instances, NASA revised or added strategic goals and objectives,

and annual performance goals within this outcome for fiscal year 2002.

For example, the agency added (1) a new strategic goal to provide

commercial industry with the opportunity to meet NASA’s future launch

needs, including human access to space, with new launch vehicles that

promise to dramatically reduce cost and improve safety and reliability and

(2) a new strategic objective to develop new capabilities for human space

flight and commercial applications through partnerships with the private

sector. Furthermore, most of the annual performance goals are either new

or revised from targets in NASA’s prior year performance plan. Further,

NASA does not provide any rationale or reasons for the changes in the

plan. NASA officials told us that, generally, the changes were made to

(1) improve NASA’s ability to assess progress toward achieving goals and

objectives; (2) reflect commitment to safety and privatization efforts; or

(3) reflect the broader scope of programs and activities and shifts in

Enterprise responsibilities. Again, providing such explanations in the

performance plan, in our view, would have been useful.

NASA displays its annual performance goals and associated performance

assessments for fiscal years 1999 to 2002 to help demonstrate cumulative

progress towards achievement of strategic goals and objectives and to

facilitate performance trend analysis. As emphasized earlier, changing

Commercial Development

of Space
Page 10 GAO-02-184 Status of Plans for Achieving Key Outcomes

performance goals annually could hinder NASA’s ability to make

comparisons between years and effectively analyze trends in performance.

NASA generally presents performance goals that appear to be objective

and help to measure progress toward this outcome. The agency also

explains how each performance goal will benefit the public.

The strategies for achieving the performance goals are generally clear and

reasonable. However, there are some exceptions. For example, one

performance goal involves developing and testing – on the ground and in

space—competing technologies for human missions beyond low earth

orbit in cooperation with international partners. One indicator related to

this performance goal involves organizing and conducting an

“international forum” at which preliminary concepts, plans, and

technology options for future human/robotic exploration and development

of space would be reviewed. However, the indicator does not address the

testing of competing technologies. Also, NASA has a performance goal to

engage the commercial community and encourage non-NASA investment

in commercial space research by meeting at least three of four

performance indicators, but the plan does not state why all of the

supporting indicators will not be achieved.

Since the selected key outcome of deploying and operating the space

station safely and cost effectively is not included in NASA’s fiscal year

2002 performance plan as a specific strategic goal or objective, we based

our assessment of it on a related strategic objective in the plan—to

operate the space station to advance science, exploration, engineering,

and commerce.

NASA set four annual performance goals for this outcome for fiscal year

2002. The performance goals are new, but the plan does not provide any

rationale for the changes. This is an important omission, because as

pointed out earlier, explaining changes in goals and measures over several

years would improve performance trend analyses and clarify why such

changes were made. In discussing the reasons for these changes, NASA

officials told us that for the International Space Station, the fiscal year

2001 goals and objectives relied on milestones that were reported as either

complete or incomplete, with no provision for reporting progress toward

completion. The improved goals and objectives for fiscal year 2002 are tied

to milestones that allow reporting of progress in terms of the percent of

the milestones completed. Thus, the new measures will provide greater

visibility and improve NASA’s ability to assess progress toward achieving

goals and objectives. In our view, providing this explanation in the plan

International Space

Station’s Operation, Cost,

and Safety
Page 11 GAO-02-184 Status of Plans for Achieving Key Outcomes

would have enhanced NASA’s discussion on this outcome. Also, similar to

the previous outcomes, NASA displays performance goals and associated

assessments for fiscal years 1999 to 2002 to help demonstrate cumulative

progress towards achievement of strategic goals and objectives and to

facilitate performance trend analysis. However, changing performance

goals annually could hinder NASA’s ability to make comparisons between

fiscal years and effectively analyze trends in performance.

Generally, NASA’s four annual performance goals and supporting

indicators for the space station outcome appear to be objective and help

to measure progress toward this outcome. In addition, the agency explains

how the goals will benefit the public, stating how completing them

successfully will provide many benefits of space research through new

discoveries and improved technological applications in areas such as

medicine, industrial processes, and fundamental knowledge. One

performance goal addresses space station safety. Specifically, the goal is

to demonstrate space station on-orbit vehicle’s operational safety,

reliability, and performance. The goal has an indicator that provides for

zero safety incidents (such as no on-orbit injuries), which appears

reasonable. The other indicator is not articulated understandably, making

it difficult to ascertain its relationship to the performance goal or to assess

its measurability. (The language is phrased as: “Actual resources available

to the payloads measured against the planned payload allocation for

power, crew time, and telemetry.”) Also, the plan does not clearly indicate

the means or strategies NASA will use to ensure that the safety

performance goal is achieved in fiscal year 2002. Similarly, the plan does

not provide clear strategies for achieving the remaining three performance

goals of (1) demonstrating space station progress and readiness at a level

sufficient to show adequate readiness in the assembly schedule, (2)

successfully completing 90% of the space station’s planned mission

objectives, and (3) demonstrating progress toward space station research

hardware development.

NASA does not address space station cost control as part of this outcome.

However, within its commercialization of space outcome, NASA set a

performance goal in fiscal year 2002 to develop and execute a

management plan and open future ISS hardware and service procurements

to cost-effective innovation through competition, including launch

services and a non-governmental organization for space station research.

NASA’s indicator for the management plan includes reforms that

(1) strengthen its headquarters involvement, (2) increase communications,

(3) provide more accurate assessment, and (4) maintain budget

accountability. NASA reports that the benefit to the public of the
Page 12 GAO-02-184 Status of Plans for Achieving Key Outcomes

management plan and reforms is to ensure that future space station costs

will remain within the President’s fiscal year 2002 budget plan. In our

view, NASA’s discussion of the proposed management plan is minimal and

lacks specificity. While the management plan will reportedly include ISS

budget accountability reforms, NASA does not elaborate on the nature of

such reforms or indicate to what extent this plan will address space

station cost growth, a long-standing management problem. Furthermore,

NASA does not acknowledge anywhere in the performance plan that space

station cost control is a major management challenge, although it has done

so for some of the other challenges. In past years and as recently as

January 2001, we have identified the need to control space station costs as

a major management challenge for NASA.11 We believe that the agency has

the opportunity to use the completed management plan to facilitate the

development of space station cost control measures in future annual

performance plans. The lack of performance measures that address space

station cost control is a shortcoming that we have identified in our

previous reviews of the agency’s annual performance plans and reports.

For the selected key outcomes, this section describes major improvements

or remaining weaknesses in NASA’s fiscal year 2002 performance plan in

comparison with its fiscal year 2001 performance plan. It also discusses

the degree to which the agency’s fiscal year 2002 plan addresses concerns

and recommendations by the Congress, GAO, NASA’s OIG, and others.

NASA’s fiscal year 2002 performance plan differs in several significant

ways from the prior plan. First, NASA portrays its planned efforts to verify

and validate performance information more comprehensively than in 2001,

providing greater confidence that the performance results will be credible.

In our review of NASA’s 2001 plan, we criticized the agency for not

explicitly describing those efforts and for not addressing data limitation

issues and problems. The 2002 plan includes specific agency data bases

and describes methods NASA will rely on to support the credibility of

reported performance information. For example, the plan references the

NASA Personnel Payroll System, Incident Reporting System, Financial and

 11 Major Management Challenges and Program Risks: National Aeronautics and Space

Administration (GAO-01-258, Jan. 2001).

Comparison of

NASA’s Fiscal Year

2002 Performance

Plan With the Prior

Year Plan

Comparison of

Performance Plans for

Fiscal Years 2001 and 2002
Page 13 GAO-02-184 Status of Plans for Achieving Key Outcomes

Contractual Status of Programs System, and NASA Environmental

Tracking System as specific data bases that will be used to verify and

validate performance data. The plan describes specific processes in place

to support performance claims associated with NASA’s Integrated

Financial Management System, performance-based contracts, contracts

awarded to small and small disadvantaged businesses, and information

technology. And it describes a broad array of methods to verify and

validate reported performance data such as monthly reports from NASA

field centers, Web statistics, count of publications, and NASA’s Education

Computer Aided Tracking System. Despite improvements in addressing

data verification and validation methods, NASA still does not acknowledge

data limitations that could hinder performance measurement. We continue

to believe that NASA can further enhance the credibility of its verification

and validation procedures and the usefulness of its performance data by

disclosing the expected limitations of its performance data in its annual

performance plans. A March 2001 NASA Office of Inspector General report

identified limitations in NASA’s fiscal year 2000 performance data and

indicated that NASA would discuss anticipated data limitations in its

performance planning beginning with its fiscal year 2002 final performance

plan.12 However, we reviewed the final version of the plan, and such a

discussion is not included.

Second, several added features help to enhance the format and/or content

of the fiscal year 2002 plan. NASA’s use of “annual performance goals” in

the plan characterizes its annual performance measures more clearly than

the “annual performance targets,” used in previous plans. The addition of

discussions on how annual performance goals benefit the public helps to

better understand the linkage between the goal and the expected results,

although in some cases additional clarification could even better convey

the actual benefit to the public. Value is added to the plan by NASA’s

display of annual performance goals and associated performance

assessments for fiscal years 1999 to 2002, to help demonstrate cumulative

progress towards achievement of strategic goals and objectives and

facilitate performance trend analysis. However, changes in performance

goals over many years could hinder NASA’s ability to make comparisons

between years and effectively analyze trends in performance.

 12Validation And Verification Of Selected NASA FY 2000 Performance Data Related To

The Government Performance And Results Act (GPRA), (IG-01-020, Mar. 30, 2001).
Page 14 GAO-02-184 Status of Plans for Achieving Key Outcomes

Also, this year’s plan includes an agencywide strategic objective to invest

in the use of human capital. NASA set two annual performance goals for

fiscal year 2002 as progress towards this objective: (1) align management

of human resources to best achieve agency strategic goals and (2) attract

and retain a workforce that is representative at all levels of America’s

diversity. However, there are no human capital initiatives specifically

linked to the outcomes or annual performance goals and indicators that

link to specific programs, such as the space shuttle program. (See details

under management challenges.)

Third, NASA could explain in the plan what procedures it has used to

characterize its performance goals as fully achieved when it has not met

all of the supporting indicators for those goals. This is particularly true for

the Earth Science outcome. Providing such an explanation would put the

actual performance in the proper perspective.

Fourth, similar to the prior plan, the fiscal year 2002 plan still does not

provide a clear rationale for how information technology-related strategies

and programs will contribute specifically to achievement of NASA’s goals

or show any allocation of information technology-related dollars and

personnel to performance goals. Goals for managing information

technology are generally stated in terms of broad categories for

improvement, such as increased capability and efficiency and enhanced

security, and include few quantitative measures. One exception is the goal

of increasing dissemination of Earth Science data, which is accomplished

through EOSDIS. The plan sets several specific goals for increasing the

volume and distribution of Earth Science data and products.

Lastly, in our review of NASA’s fiscal year 2001 plan, we suggested that

NASA document in its annual performance plans and reports, the rationale

for establishing new performance targets to clarify the reasons for adding

such targets. We had noted that while many of NASA’s annual

performance targets were new each year, there was no stated basis for the

changes. In its fiscal year 2002 performance plan, NASA has formulated

new annual performance goals and has changed many of its strategic goals

and objectives without including the reasons for doing so. We continue to

believe that providing the rationale for these changes will clarify the

reasons for the new goals and measures and augment the value of

performance trend analyses. Also, the plan does not indicate whether or

not achieving any specific goals would be negatively affected by external

factors. However, like the prior plan, the fiscal year 2002 plan states that

successful execution of NASA’s strategic goals and objectives depends on

receipt of its requested appropriations, as well as provision of funds,
Page 15 GAO-02-184 Status of Plans for Achieving Key Outcomes

materials, or services, that have been committed to the cooperative

agreements or partnerships which are referenced in the performance plan.

We have identified two governmentwide high-risk areas: strategic human

capital management and information security. Regarding strategic human

capital management, NASA’s fiscal year 2002 performance plan contains a

strategic objective and annual performance goals and indicators directly

related to human capital. Concerning information security, NASA’s

performance plan contains a strategic objective, an annual performance

goal, and indicators directly related to this management challenge. The

plan states that safety and security is one of four areas on which NASA’s

information technology planning is focused. The fiscal year 2002 plan is an

improvement over the 2001 plan, which did not include quantifiable

measures for improving information security. However, the plan’s

performance goals do not fully respond to the recommendations we made

in 1999 when we reported that the agency lacked an effective agencywide

security program.13 For example, the plan sets a performance indicator of

completing 90 percent of information technology security plans for critical

systems. However, we recommended that all systems be formally

authorized before they became operational and at least every

3 years thereafter.

In addition, we have identified three major management challenges facing

NASA: (1) correcting contract management weaknesses, (2) controlling

International Space Station costs, and (3) effectively implementing the

faster-better-cheaper approach to space exploration projects. We found

that NASA’s performance plan contains an annual performance goal and

indicators directly related to the problem of contract management. It is

important to note that until NASA’s Integrated Financial Management

System—-which is central to providing effective management and

oversight over its procurement dollars—is operational, performance

assessments relying on cost data may be incomplete and full costing will

be only partially implemented.

While NASA’s performance plan contains an annual performance goal and

an indicator that indirectly addresses the challenge of controlling space

station costs, it does not indicate the extent that NASA will address space

 13Information Security: Many NASA Mission-Critical Systems Face Serious Risks

(GAO/AIMD-99-47, May 20, 1999).

NASA’s Efforts to

Address its Major

Management

Challenges Identified

by GAO
Page 16 GAO-02-184 Status of Plans for Achieving Key Outcomes

station cost growth. As we discussed in our January 2001 report, the

International Space Station Program continues to face cost control

challenges. As with contract management, until the Integrated Financial

Management System is operational, NASA may lack the cost information

needed to control space station costs.

Further, NASA’s performance plan did not directly address the challenge

of effectively implementing the faster-better-cheaper approach to space

exploration projects. In January 2001, we also reported that NASA faces

significant challenges as it attempts to create highly reliable missions and

foster open communications under the budget constraints of the agency’s

faster-better-cheaper space exploration strategy. In addition, the real

success of this strategy will require a comprehensive integration of lessons

learned from failures on an agencywide basis. Until NASA resolves these

problems, its financial resources are vulnerable to inefficient use.

Appendix I provides detailed information on how NASA addressed these

challenges and high-risk areas as identified by GAO and NASA’s Office of

Inspector General (OIG).

As agreed, our evaluation was generally based on the requirements of

GPRA, guidance to agencies from the Office of Management and Budget

(OMB) for developing performance plans (OMB Circular A-11, Part 2),

previous reports and evaluations by us and others, our knowledge of

NASA’s operations and programs, our identification of best practices

concerning performance planning, and our observations on NASA’s other

GPRA-related efforts. We also discussed our review with NASA officials

and with officials of NASA’s OIG. The agency outcomes that were used as

the basis for our review were identified by the Ranking Minority Member

of the Senate Committee on Governmental Affairs as important mission

areas for NASA and do not reflect the outcomes for all of NASA’s

programs or activities. The major management challenges confronting

NASA, including the governmentwide high-risk areas of strategic human

capital management and information security, were identified in our

January 2001 performance and accountability series and high risk update,

and by NASA’s OIG in December 2000. We conducted our review from

August 2001 through October 2001 in accordance with generally accepted

government auditing standards.

We provided copies of a draft of this report to NASA for its review and

comment. In written comments on the report, NASA generally agreed with

the information presented in the report and noted several improvements it

Scope and

Methodology

Agency Comments
Page 17 GAO-02-184 Status of Plans for Achieving Key Outcomes

would make. Concerning our suggestion that NASA could fully explain in

its performance plan why it believes it is not necessary to achieve all

performance indicators to demonstrate annual performance goal

achievement, NASA stated that it would provide a statement containing

the supporting rationale for this approach in its fiscal year 2003

performance plan.

In responding to our observation that the fiscal year 2002 plan lacked

sufficient detail on the nature of the ISS’s budget accountability reforms or

how the reforms will address longstanding and ongoing management

problems, including cost growth, NASA commented that the reforms are

contained in its Program Management Action Plan that will be referred to

in the fiscal year 2003 performance plan. We note that the ISS Program is

being restructured in response to a potential cost growth of $4.8 billion.

The restructuring has raised widespread concerns about the potential

science benefits to be realized by the United States and international

partners. For this reason, we believe it is increasingly important for

NASA’s performance plan to provide a clear path showing how NASA

intends to implement the needed reforms and how the reforms will add

credibility to future ISS budgets and resolve the uncertainties concerning

the utility of the ISS.

NASA also commented on a statement in our draft report that the agency

does not provide a clear rationale for how IT-related strategies and

programs will contribute specifically to achievement of its goals or show

the allocation of IT-related dollars and personnel to performance goals.

According to NASA, the IT service delivery metric in the plan aggregates

each major IT service, such as NASA’s Integrated Services Network.

Remaining IT investments are embedded in each NASA project and

managed as part of the project. While this statement may be true on the

individual program level, it does not address GPRA objectives to

demonstrate how IT-related strategies and programs contribute

specifically to the achievement of agency goals or show the allocation of

related resources.

Finally, in response to our observation regarding NASA’s lack of

explanations in the 2002 plan for annual performance changes, NASA

agreed that including such explanations in the plan would be useful and

that it would characterize reasons for annual performance changes in its

2003 performance plan.

As arranged with your office, unless you publicly announce its contents

earlier, we plan no further distribution of this report until 30 days after the
Page 18 GAO-02-184 Status of Plans for Achieving Key Outcomes

date of this letter. At that time, we will send copies to appropriate

congressional committees; the NASA Administrator; and the Director,

Office of Management and Budget. Copies will also be made available to

others on request.

If you or your staff have any questions, please call me at (202) 512-4841.

Key contributors to this report were Richard J. Herley, Shirley B. Johnson,

Charles W. Malphurs, Christina Chaplain, John de Ferrari, Diane G.

Handley, and Fannie M. Bivins.

Sincerely yours,

Allen Li

Director, Acquisition and

 Sourcing Management
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 19 GAO-02-184 Status of Plans for Achieving Key Outcomes

The following table identifies the major management challenges

confronting NASA, including the governmentwide high-risk areas of

strategic human capital management and information security. The first

column of the table lists the management challenges that we and/or

NASA’s Office of Inspector General (OIG) have identified. The second

column discusses the extent to which NASA’s fiscal year 2002

performance plan includes performance goals and measures to address

the challenges that we and the OIG identified. Of the agency’s fifteen

major management challenges, its performance plan has (1) goals and

measures that are directly related to thirteen of the challenges; (2) a goal

and measure indirectly applicable to one challenge; and (2) no goals and

measures directly related to one of the challenges.

Some of the NASA performance plan’s goals and measures we discuss may

not track specifically with the key considerations of NASA OIG’s

management challenges since the challenges themselves were presented in

a broad context. GAO has performed reviews affecting a number of the

areas mentioned. This appendix highlights the results of our assessments,

where applicable.

Appendix I: Observations on NASA’s Efforts

to Address Its Major Management Challenges
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 20 GAO-02-184 Status of Plans for Achieving Key Outcomes

Table 1: Major Management Challenges

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

GAO-designated governmentwide high risk

Strategic Human Capital Management: GAO has identified

shortcomings at multiple agencies involving key elements of

modern human capital management, including strategic human

capital planning and organizational alignment; leadership

continuity and succession planning; acquiring and developing

staffs whose size, skills, and deployment meet agency needs; and

creating results-oriented organizational cultures.

In August 2000, we reported that several internal NASA studies

had shown that the agency’s space shuttle program’s workforce

had been affected negatively by NASA’s downsizing, much of

which occurred after 1995. We also reported that NASA had

begun taking actions to address its shuttle workforce problems.a In

September 2001, we reported in testimony that while NASA

continues to make progress in revitalizing the shuttle program’s

workforce, considerable challenges remain.b

In January 2001, we also reported the need to implement a

human capital approach in NASA’s workforce management

strategies as a major management challenge.c

The plan contains a strategic objective, annual performance goals

and indicators directly related to this management challenge. The

plan’s strategic objective is to invest wisely in NASA’s use of

human capital, developing and drawing upon the talents of all

NASA’s people. This objective applies to all of NASA’s workforce

and has two related performance goals: (1) align management of

human resources to best achieve agency strategic goals and

objectives and (2) attract and retain a workforce that is

representative at all levels of America’s diversity. NASA further

indicated in discussion under the second performance goal: (1) in

fiscal year 2002, the agency will develop a process by which the

centers will implement consistent workforce planning resulting in a

plan for each center that links staffing, funding resources, mission

and activities, and core competencies; (2) NASA intends in fiscal

year 2002 to develop an initiative to enhance the centers’

recruitment capabilities, focusing on hiring new college graduates

to counterbalance the aging of the workforce due to the halt in the

influx of new college graduates during the years of downsizing;

and (3) NASA requested additional fiscal year 2002 resources to

expand training delivery methods and emphasize the

development of computer-based training alternatives that can be

accessed at all locations and levels. While NASA’s inclusion of

human capital in its plan as a strategic objective is an

improvement, it needs to relate its human capital annual

performance goals and indicators to specific programs where

issues of critical staffing needs have been identified, such as the

space shuttle program.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 21 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Information Security: Our January 2001 high-risk update noted

that the agencies’ and governmentwide efforts to strengthen

information security have gained momentum and expanded.

Nevertheless, recent audits continue to show federal computer

systems are riddled with weaknesses that make them highly

vulnerable to computer-based attacks and place a broad range of

critical operations and assets at risk of fraud, misuse, and

disruption.

In 1999, we reported that NASA lacked an effective agencywide

security program that includes improvements in five areas:

assessing risks and evaluating needs, implementing policies and

controls, monitoring compliance with policy and effectiveness of

controls, providing computer security training, and coordinating

responses to security incidents.d

 The need for such a framework

was serious; tests we conducted at 1 of NASA’s 10 field centers

found that mission-critical information systems were vulnerable to

unauthorized access. We successfully penetrated several of these

systems, including one responsible for calculating detailed

positioning data for earth-orbiting spacecraft and another that

processes and distributes scientific data received from these

spacecraft.

The plan contains a strategic objective and an annual

performance goal and indicators directly related to this

management challenge. The plan states that safety and security

is one of four areas on which NASA’s information technology

planning is focused. A performance goal is established to

enhance information technology security by meeting established

performance indicators in three critical areas:

(1) vulnerabilities detected, (2) training, and

(3) information technology security plans. The plan includes

specific target percentages for reducing known system

vulnerabilities, training NASA personnel, and preparing

information technology security plans. These are all areas where

problems have been documented at NASA in the past, including

our review of information security for NASA’s mission critical

systems, completed in 1999. The fiscal year 2002 plan is an

improvement over the 2001 plan, which did not include

quantifiable measures for improving information technology

security. However, the plan’s performance goals are not fully

responsive to the recommendations we made in 1999. For

example, the plan sets a performance indicator of completing

90 percent of information technology security plans for critical

systems. We, on the other hand, recommended that all systems

be formally authorized before they became operational and at

least every 3 years thereafter. Likewise, NASA’s performance

indicator for information technology security training sets

employee training targets between 80 and 95 percent for

providing awareness training to different types of NASA

employees. We recommended that NASA establish a more

comprehensive program that would include certifying that NASA

civil servants and contract employees are competent to discharge

their information technology security-related responsibilities.

GAO-designated major management challenges

The Need to Correct Weaknesses in NASA’s Contract

Management: We have reported that NASA’s contract

management is a continuing area of high risk. Implementation of

the financial management system and its integration with full cost

accounting have been delayed. Until the Integrated Financial

Management System is operational, performance assessments

relying on cost data may be incomplete. We have also reported

that NASA is continuing to rely on undefinitized change orders—

that is, contract changes initiating new work before NASA and the

contractor agree on a final estimated cost and fee—to complete

work on its largest space station contract. We stated that this is a

risky way of doing business because it increases the potential for

unforeseen cost increases and scheduling delays.

The plan contains an annual performance goal and indicators

directly related to this management challenge. The performance

goal is to improve the agency’s financial management and

accountability. This goal has two indicators (1) cost at least

75 percent of the budget available to cost during fiscal year 2002;

and (2) initiate the pilot phase of the core financial project and at

least one other project module of NASA’s third attempt at

implementing an Integrated Financial Management System

(IFMS). The plan also states (1) an agency-level project team is in

place at the Marshall Space Flight Center, the lead center for the

core financial project, and the design phase will begin in February

2002 and (2) three “pathfinder” projects have begun to test out the

processes and technical requirements for agency-wide

implementation of new administrative systems. The IFMS is key to

producing accurate and reliable information for full-cost

accounting. Furthermore, the plan does not address the issue of

undefinitized change orders.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 22 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

The Need to Control International Space Station (ISS)

Development and Support Costs: We have reported that the ISS

program continues to face cost control challenges.e

 NASA’s OIG

also reported that the ISS program continued to experience cost

overruns and scheduling delays.

The plan contains an annual performance goal and indicator that

indirectly addresses ISS cost control issues. The performance

goal is to develop a management plan and initiate ISS reforms,

but the plan does not indicate to what extent NASA will address

space station cost growth. This challenge is discussed in detail in

the outcomes section of this report.

The Need to Effectively Implement the Faster-Better-Cheaper

Approach to Space Exploration Projects: We have reported that

NASA faces significant challenges as it attempts to create highly

reliable missions and foster open communications under the

budget constraints of the agency’s faster-better-cheaper space

exploration strategy. In addition, real success will require a

comprehensive integration of lessons learned from failures on an

agencywide basis. Until NASA resolves these problems, its

financial resources are vulnerable to inefficient use.

This was designated as a new major management challenge in

January 2001.

The plan does not have goals and measures directly related to

this management challenge.

OIG-designated major management challenges

Safety and Mission Assurance: NASA’s OIG has reported that

safety and mission assurance has become a serious challenge for

NASA. Key considerations to ensure safety in future NASA

operations include (1) ensuring an appropriate level of training for

staff who conduct safety reviews and evaluations; (2) maintaining

adequate safety reporting systems; (3) ensuring variances to

standard safety procedures are appropriately justified, reviewed,

and approved; (4) maintaining an effective emergency

preparedness program; (5) ensuring NASA and contractor

compliance with safety standards and regulations; (6) ensuring

product safety and reliability; and (7) ensuring the space shuttle

and the ISS maintain crew safety.

In September 2001, we reported while NASA is making strides in

revitalizing its workforce, its ability to implement safety upgrades

in a timely manner is uncertain.f

The plan contains strategic objectives, annual performance goals

and indicators directly related to this management challenge. In

fact, NASA’s plan contains a high emphasis on safety. The plan

has a strategic objective to protect the safety of people and

facilities and the health of the workforce. This objective’s

performance goal directs NASA to increase the safety of its

infrastructure and the health of its workforce through facilities’

safety improvements, reduced environmental hazards, increased

physical security, enhanced safety and health awareness, and

appropriate tools and procedures for health enhancement. There

are eight indicators for this performance goal, which include:

(1) no fatalities will result from NASA mishaps and (2) per the

Federal Worker 2000 Initiative, reduce the overall occurrence of

injuries (due to occupational injury or illness) by 3% per year from

the fiscal year 1997 baseline to 1.15 occurrences per 100

workers. The plan also has strategic objectives to (1) provide and

make use of safe, affordable, and improved access to space and

(2) ensure the health, safety, and performance of humans living

and working in space. Furthermore, the Aerospace Technology

Enterprise has two related strategic goals (1) to revolutionize

aviation – enable the safe, environmentally-friendly expansion of

aviation and (2) to advance space transportation – create a safe,

affordable highway through the air and into space. The plan also

states that NASA management, in the interest of safety,

encourages space shuttle program managers to set aside metrics

when dealing with launches planned versus launches achieved

during a given fiscal year. Our September 2001 testimony noted

that NASA is still assessing the full package of its planned space

shuttle program workforce and safety improvements, and some

projects have already encountered funding and scheduling

problems. Overcoming challenges related to the safety upgrades

is critical since NASA will be relying on the space shuttle longer

than originally anticipated.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 23 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

International Space Station: NASA’s OIG has reported that the

ISS is a significant management challenge due to significant

problems related to ISS cost, contingency planning, and the

X-38/Crew Return Vehicle. Key considerations for continued ISS

assembly and operation are (1) managing the political, financial,

technical, and safety challenges presented by an international

partnership; (2) overcoming technical challenges inherent in

manufacturing, assembling, and testing complex hardware and

software components provided by different nations and integrated

in space; (3) safely maintaining, upgrading, and operating a

structure as complicated as the space station; and (4) maximizing

the beneficial use of the space station for scientific research and

technology development.

In June 2001, we reported that inadequate planning and design

led to ISS propulsion module failure.g

NASA OIG’s description of the ISS major management challenge

is more broadly focused than our related management challenge.

(See discussion under GAO-designated major management

challenge, the need to control ISS development and support

costs.) The plan contains strategic objectives and annual

performance goals directly related to this management challenge.

The plan contains a strategic objective to conduct engineering

research on the International Space Station to enable exploration

beyond Earth orbit. This objective has a performance goal to test,

at the International Space Station, competing technologies for

human missions beyond low earth orbit, in cooperation with other

agencies and international partners and with U.S. industry. The

plan also has a strategic objective to operate the International

Space Station to advance science, exploration, engineering, and

commerce. This objective has four performance goals:

(1) demonstrate ISS on-orbit vehicle operational safety, reliability,

and performance; (2) demonstrate ISS program progress and

readiness at a level sufficient to show adequate readiness in the

assembly schedule; (3) successfully complete 90 percent of the

ISS planned mission objectives; and (4) demonstrate progress

toward ISS research hardware development. The Biological and

Physical Research Enterprise has (1) strategic objectives to

develop strategies to maximize scientific research output on the

International Space Station and other space research platforms;

(2) a strategic objective to foster commercial research endeavors

with the International Space Station and other assets; and (3) a

performance goal to highlight ISS-based commercial space

research at business meetings and conferences. The plan also

has a strategic objective to foster commercial endeavors with the

International Space Station and other assets. This objective has a

performance goal to develop and execute a management plan

and open future station hardware and service procurements to

innovation and cost-saving ideas through competition, including

launch services and a non-government organization for space

station research. However, the plan does not indicate the extent

that NASA will address space station cost growth. This challenge

is discussed in detail in the outcomes section of this report.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 24 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Information Technology: NASA’s OIG has reported that

information technology has become a serious challenge for

NASA. Key considerations for an effective information technology

program include (1) ensuring data security, integrity, and

application controls; (2) protecting operations and

communications with spacecraft; and (3) monitoring and

evaluating the streamlining of operations through outsourcing

information technology operations for cost efficiencies,

dependency on the vendor for technological direction, vulnerability

of strategic information to outsiders, and dependency on the

viability of the vendor.

NASA’s OIG reported that during fiscal year 2000 NASA

continued to have a fragmented information technology (IT)

security program without clear lines of authority, policies,

guidelines, or enforcement. The OIG reported that audits of

several mission critical information systems disclosed that NASA

had not implemented adequate basic controls in areas such as

system access, protection of critical files, system backup and

restore procedures, privileged operations controls, and system

audit and monitoring capabilities.

(See discussion under governmentwide high-risk challenge:

information security for additional details.)

The plan contains a strategic objective, annual performance

goals, and indicators directly related to this management

challenge. The plan states that safety and security is one of four

areas on which NASA’s information technology planning is

focused. The plan’s strategic objective is to enhance the security,

efficiency, and support provided by NASA’s information

technologies resources. The objective has three performance

goals to (1) improve information technology infrastructure service

delivery by providing increased capability and efficiency while

maintaining a customer rating of satisfactory; (2) enhance mission

success through seamless, community-focused electronic service

delivery; and (3) enhance information technology security by

meeting established performance indicators in three critical areas:

(a) vulnerabilities detected, (b) training, and (c) information

technology security plans. The plan includes specific target

percentages for reducing known system vulnerabilities, training

NASA personnel, and preparing information technology security

plans. These are all areas where problems have been

documented at NASA in the past, including our review of

information security for NASA’s mission critical systems,

completed in 1999. The fiscal year 2002 plan is an improvement

over the 2001 plan, which did not include quantifiable measures

for improving information technology security. However, the plan’s

goals are not as comprehensive as the recommendations we

made in 1999. For example, the plan sets a target of completing

90 percent of information technology security plans, including

authorizations to process, for its critical systems. We, on the other

hand, recommended that all systems be formally authorized

before they become operational and at least every 3 years

thereafter. Likewise, NASA’s performance goal indicator for

information technology security training sets targets between

80 and 95 percent for providing awareness training to different

types of NASA employees. We recommended that NASA

establish a more comprehensive program that would include

certifying that NASA civil servants and contract employees are

competent to discharge their information technology securityrelated responsibilities.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 25 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Procurement: NASA’s OIG has reported that procurement is an

ongoing challenge for NASA. Key considerations for effective

procurement at NASA include (1) ensuring proper levels of

staffing to perform contracting requirements; (2) providing

sufficient controls over and monitoring both prime and

subcontractors; (3) implementing or increasing the use of

innovative procurement procedures such as earned value

management and performance incentive fees; and (4) ensuring

costs billed to NASA cost-type contracts, due to the changing

industry environment, are reasonable and allowable.

In August 2001, we reported that NASA’s inability to provide

timely data on obligations or support for actual costs related to the

space station and shuttle raises concerns about NASA being able

to achieve the discipline and accountability called for by the

National Aeronautics and Space Administration Authorization Act

for Fiscal Year 2000 (P.L. 106-391).h

The plan contains a strategic objective, annual performance

goals, and indicators directly related to this management

challenge. The plan’s strategic objective is to achieve the most

productive application of federal acquisition policies. This

objective’s first performance goal is to continue to take advantage

of opportunities for improved contract management by

maintaining a high proportion of performance based contracts

(PBCs). This performance goal’s indicator is to maintain PBC

obligations at greater than 80% of funds available for PBCs. The

objective’s second performance goal is to continue integrating

small, small disadvantaged, and women-owned businesses

together with minority universities into the competitive base from

which NASA can purchase goods and services. NASA further

indicated in discussion under the strategic objective that (1)

NASA’s Office of Procurement has undertaken proactive

management approaches in three key areas: human capital,

outsourcing and oversight, and electronic commerce; and (2) on

November 2000, the Associate Administrator for Procurement and

the Associate Administrator for Safety and Mission Assurance

jointly announced the establishment of the Surveillance Planning

Team to provide policy direction and procedural guidance on

appropriate surveillance planning for NASA-contracted work

based on the risk associated with the work and contractor

involvement. The plan also contains a performance goal to

continue implementation of planned and new privatization efforts

through the space shuttle prime contract and further efforts to

safely and effectively transfer civil service positions and

responsibilities to the space shuttle contractor with associated

performance goal indicators. The plan also contains a

performance indicator to initiate the pilot phase of the core

financial module and at least one other module of the integrated

financial management system. If effectively implemented, this

system should provide systems and processes to oversee

procurement activities. However, this effort will require continued

management attention to correct problems and keep projects on

schedule. Furthermore, the plan does not address the issue of

limiting NASA’s frequent use of undefinitized contract change

orders – that is, unnegotiated contract changes. Relying on

unnegotiated changes as a way of doing business is risky

because it increases the potential for additional unanticipated cost

growth.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 26 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Fiscal Management: NASA’s OIG has reported that fiscal

management continues to be a significant challenge for NASA.

Key considerations to improved fiscal management include:

(1) monitoring contractor performance of financial statement

audits to ensure that the statements are properly prepared and

thoroughly reviewed; (2) ensuring adequate integration and

testing of newly developed automated accounting modules or

capability; and (3) ensuring that NASA continues to properly

account for and record financial transactions as new capability is

implemented.

In March 2001, we reported that NASA’s fiscal year 1999

Statement of Budgetary Resources was misstated by a reported

$644 million due, in part, to a misinterpretation of guidance and

errors in NASA’s ad hoc process for generating budgetary

information.i

 In August 2001, we reported on NASA’s inability to

provide timely data on obligations or support for actual costs

related to the space station and shuttle.j

The plan contains a strategic objective, annual performance

goals, and indicators directly related to this management

challenge. The plan’s strategic objective is to manage NASA’s

fiscal and physical resources optimally. This objective has an

annual performance goal to improve NASA’s financial

management and accountability. The plan states two target

indicators will measure the agency’s progress in meeting this

performance goal: (1) cost at least 75 percent of the budget

available to cost during the fiscal year and (2) initiate the pilot

phase of the core financial module and at least one other module

of the integrated financial management system (IFMS). While the

goal of improving financial management and accountability is

commendable, additional target indicators should be incorporated

to measure NASA’s progress in achieving its goal. Additional

performance indicators are needed to ensure that NASA’s

systems and processes provide a direct linkage between financial

and program operations. For example, both the efficiency and

effectiveness measures should be designed to assess the finance

organization’s ability to support NASA’s mission. Without

performance indicators that provide a clear linkage between

financial management improvement and improved program

results, NASA may not be able to accurately track its progress in

achieving its performance goals. Furthermore, the implementation

of the IFMS is key to producing accurate and reliable information

for full-cost accounting. Until the system is operational,

performance assessments relying on cost data may be

incomplete and full costing will be only partially implemented.

Measuring costs is key to measuring performance in terms of

efficiency and cost-effectiveness.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 27 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Program and Project Management: NASA’s OIG has reported that

NASA faces significant challenges in program and project

management. Key considerations to effectively managing NASA

programs include (1) improving planning to enable NASA to

accomplish its missions in the face of budget and human capital

issues; (2) eliminating duplication in programs and improving

coordination with other research and development organizations;

(3) ensuring that programs and projects accurately assess their

progress and successfully achieve their goals; and (4) effectively

using technology developments to increase NASA productivity.

Although we did not identify this issue as a management

challenge, we have reported in September 2001 that there are

fundamental weaknesses in the collection and sharing of lessons

learned in NASA by program and project managers.k

 In June

2001, we also reported that the initial ISS propulsion module

project did not meet performance, cost, and schedule goals

largely because NASA proceeded with the contractor’s proposal

without following fundamental processes involving project

planning and execution.l

The plan includes a strategic goal and objectives, annual

performance goals, and indicators directly related to this

management challenge. The plan has a strategic goal to enable

NASA’s strategic enterprises and their centers to deliver products

and services to customers more effectively and efficiently. This

goal has four related objectives: (1) enhance program safety and

mission success in the delivery of products and operational

services; (2) improve NASA’s engineering capability to remain as

a premier engineering research and development organization;

(3) capture engineering and technological best practices and

process knowledge to continuously improve NASA’s

program/project management; and (4) facilitate technology

insertion and transfer and utilize commercial partnerships in

research and development to the maximum extent practicable.

The plan contains a performance goal to capture a set of best

practices/ lessons learned from each program, to include at least

one from each of the four “provide aerospace products and

capabilities crosscutting process” subprocesses, commensurate

with current program status. The plan contains a performance

goal to earn external review rating of “achieved performance

target” on making progress in the following area: design, develop,

and launch projects to support future research in pursuit of

Strategic Plan science objectives. This goal has an indicator to

meet no fewer than 75 percent of the development performance

objectives for “major programs/projects,” supported by completion

of performance objectives in majority of “other projects.” The plan

has a performance goal to identify and evaluate candidate

approaches for 100- to 1000- day human missions capable of a

5- to 10- fold cost reduction—while increasing safety and

effectiveness (compared to 1990s projections). The plan contains

a strategic objective to meet sustained space operations needs

while reducing costs. This objective has two performance goals:

(1) the Space Communications program will conduct tasks that

enable commercialization and will minimize investment in

government infrastructure for which commercial alternatives are

being developed; and (2) performance metrics for each mission

will be consistent with detailed program and project operations

requirements in project service level agreements.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 28 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Launch Vehicles: NASA’s OIG reported on challenges in

(1) ensuring the availability of small expendable launch vehicles to

ensure schedule milestones and cost-effectiveness of NASA

missions; (2) evaluating whether NASA is providing the majority of

developmental funds and assigning technology rights to its

industry partners in the development of the new reusable launch

vehicles in the best interest of the government; and (3) ensuring

that plans are in place and are effectively implemented to address

shuttle systems obsolescence, logistics support, technical/safety

upgrades, and funding.

Although we did not identify this issue as a management

challenge, we have reported and testified on the factors that

contributed to the difficulties experienced by the X-33 and X-34

programs and the steps needed to avoid repeating those

problems within the Second Generation Reusable Launch Vehicle

Program.m

The plan includes a strategic goal and objective, annual

performance goal and indicators directly related to this

management challenge. The Aerospace Technology Enterprise

has a strategic goal to provide commercial industry with the

opportunity to meet NASA’s future launch needs, including human

access to space, with new launch vehicles that promise to

dramatically reduce cost and improve safety and reliability. The

strategic objective is to utilize NASA’s Space Transportation

Council in combination with an External Independent Review

Team (EIRT) to assure agency-level integration of near- and farterm space transportation investments. The performance goal is

to review results of NASA and commercial-sector performed

launch system architecture studies, related requirements, and

refinements in planned risk-reduction investments. There are two

performance goal indicators (1) complete an assessment of the

space launch initiative architectures and requirements by EIRT;

the EIRT will submit a written report on their evaluation within

45 days following completion of the review; and (2) the Space

Transportation Council will review progress and planning of the

space launch initiative at least twice during the fiscal year,

including the report filed by the EIRT. However, NASA’s plan does

not contain enough specific information about how NASA will

ensure that the government’s best interests will be served in these

joint government- and industry-funded programs.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 29 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Technology Development: NASA’s OIG has reported that

technology development has become a serious challenge for

NASA. Key considerations to effective technology development

include (1) achieving a balance between scientific research and

technology development and demonstration projects;

(2) continuing to refine the technology transfer process to ensure

that U.S. industry achieves the maximum benefit from the new

technologies identified; (3) determining if NASA’s organizational

structure effectively supports technology development and

transfer; (4) forming innovative partnership arrangements with

U.S. industry to share both the risk and costs of technology

demonstration and commercialization; (5) ensuring that NASA

technology demonstrations do not unfairly distort the marketplace;

(6) ensuring that adequate controls exist on cooperative

technology development programs; and (7) ensuring adequate

protection of NASA-developed technology.

The plan has strategic goals, objectives and annual performance

goals directly related to this management challenge. The space

science enterprise contains a strategic goal: “Technology/LongTerm Future Investments: develop new technologies to enable

innovative and less expensive research and flight missions.” The

strategic objectives for this goal are to: (1) acquire new technical

approaches and capabilities; (2) validate new technologies in

space; and (3) apply and transfer technology. The Earth Science

Enterprise has a strategic goal to develop and adopt advanced

technologies to enable mission success and serve national

priorities. This goal has related objectives, performance goals,

and indicators. The plan has a strategic objective to invest in the

development of high-leverage technologies to enable safe,

effective and affordable human/robotic exploration. This objective

has a performance goal to begin development of high-leverage

technologies to enable safe, effective and affordable

human/robotic exploration missions beyond low earth orbit (LEO).

The plan also includes performance goals to: (1) test at the

International Space Station competing technologies for human

missions beyond LEO, in cooperation with other agencies and

international partners, and with US industry; (2) select and fund at

least 3-5 proposals through the Human Exploration and

Development of Space Enterprise’s (HEDS) Technology and

Commercialization Initiative-focused research and technology

program that feature: highly innovative new technology

development efforts in selected areas associated with human

safety and performance in space; and (3) conduct a competitive

solicitation and selection process that will fund through the HEDS

research and technology program a HEDS Technology and

Commercialization Initiative: (a) systems studies assessing the

commercial potential associated with various prospective HEDS

infrastructures/capabilities; and (b) new technology development

and demonstration efforts with potential longer-term commercial

space value. The Biological and Physical Research Enterprise

has a performance goal to engage the commercial community

and encourage non-NASA investment in commercial space

research by meeting at least three of four performance indicators.

The plan also has a strategic objective to transfer NASA

technologies and innovations to private industry and the public

sector so that the public can benefit economically as well as

intellectually through clear, effective communications concerning

NASA’s activities.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 30 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

International Agreements: NASA’s OIG reported that international

agreements are needed to ensure effective and efficient

programs. Key considerations include (1) program and project

vulnerability to schedule delays and cost overruns that require

diplomatic rather than contractual solutions; (2) security controls

on technology that impacts national security; (3) controls to

assure the quality and timeliness of the goods and services

provided; (4) mechanisms to assure a balance between program

needs and national considerations; (5) plans with specific critical

paths and planned alternative courses of action to maintain

program/project continuity; and (6) proper controls over access to

NASA facilities by foreign national visitors.

Although we did not identify this issue as a major management

challenge, in November 1999, we recommended measures to

enhance NASA’s ability to oversee and implement its export

controls of ISS-related technologies.n

The plan has a strategic objective and annual performance goals

directly related to this management challenge. The plan has

performance goals to (1) collaborate with other federal and

international agencies in developing and implementing better

methods for using remotely sensed observations; (2) test at the

International Space Station competing technologies for human

missions beyond LEO in cooperation with other agencies and

international partners and with US industry; (3) develop and test –

on the ground and in space—competing technologies for human

missions beyond LEO in cooperation with international partners;

(4) conduct a competitive solicitation and selection process that

will fund through the HEDS research and technology program the

HEDS Technology and Commercialization Initiative. The fourth

performance goal includes a performance indicator to develop, in

conjunction with discussions with key international space

organizations, and seeks management approval for an approach

for undertaking the formulation of international partnerships for the

development and/or demonstration of HEDS capabilities. The plan

also states that in response to recommendations from the NASA

OIG, NASA is clarifying the definition of “foreign national” in its

foreign visitors policy to ensure appropriate and consistent use of

the term in the agency’s foreign visitors review program. However,

NASA’s indicators for this challenge lack sufficient specificity. For

example, the plan does not specify actions taken to ensure

security controls on technology that impacts national security.
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 31 GAO-02-184 Status of Plans for Achieving Key Outcomes

Major management challenge

Applicable goals and measures in the fiscal year 2002

performance plan

Environmental Management: NASA’s OIG reported that

environmental management is a significant management

challenge. Key considerations include: (1) prioritizing and

addressing environmental obligations; (2) developing consistent

procedures under an agencywide policy; and (3) negotiating costsharing agreements for environmental cleanup with previous

government and private sector tenants that are also responsible

parties.

The plan includes annual performance goals directly related to

this management challenge. The plan also states that NASA’s

strategy for achieving its environmental vision includes four focus

areas: prevention, compliance, restoration, and conservation.

However, NASA’s plan only has performance metrics for the

areas of compliance and restoration. Additional performance

indicators for the environmental focus areas of prevention and

conservation are needed so that NASA can accurately and fully

track its progress toward achieving its environmental vision. The

compliance related performance goal is that NASA will increase

the safety of its infrastructure and the health of its workforce

through facilities safety improvements, reduced environmental

hazards, increased physical security, enhanced safety and health

awareness, and appropriate tools and procedures for health

enhancement. This performance goal includes an indicator to

reduce the level of agency environmental noncompliance

incidents and releases in order to achieve a 5 percent reduction

from the fiscal year 2000 level by 2005. The environmental

restoration-related performance goal is to revitalize agency

facilities and reduce environmental liability. This performance goal

includes an indicator to reduce NASA’s unfunded environmental

liability through a long-term strategy, annually investing an

amount of not less than 3-5% of the agency’s environmental

liability in environmental compliance and restoration funding. The

plan also states management controls need to be strengthened to

ensure greater visibility of and more consistent implementation of

the National Environmental Policy Act (NEPA) process. The plan

states review of existing management controls, development and

advocacy of improvements, and training activities have been

planned and are being initiated. Furthermore, the Aerospace

Technology Enterprise also has a strategic objective to reduce

emissions – protect local air quality and our global climate with a

related performance goal and indicators. a Space Shuttle: Human Capital and Safety Upgrade Challenges Require

Continued Attention (GAO/NSIAD/GGD-00-186, Aug. 15, 2000). b

 Space Shuttle Safety: Update on NASA’s Progress in Revitalizing the Shuttle

Workforce and Making Safety Upgrades (GAO-01-1122T, Sept. 6, 2001). c

Major Management Challenges and Program Risks: National Aeronautics and

Space Administration (GAO-01-258, Jan. 2001). d

Information Security: Many NASA Mission-Critical Systems Face Serious

Risks (GAO/AIMD-99-47, May 20, 1999). e

 (GAO-01-258, Jan. 2001). f

 (GAO-01-1122T, Sept. 2001). g

Space Station: Inadequate Planning and Design Led to Propulsion Module

Project Failure (GAO-01-633, Jun. 20, 2001). h NASA: International Space Station and Shuttle Support Cost Limits

(GAO-01-1000R, Aug. 31, 2001). i

Financial Management: Misstatement of NASA’s Statement of Budgetary

Resources (GAO-01-438, Mar. 30, 2001) j

 (GAO-01-1000R, Aug. 31, 2001). k Survey of NASA’s Lessons Learned Process (GAO-01-1015R, Sept. 5, 2001)
Appendix I: Observations on

NASA’s Efforts to Address Its

Major Management Challenges

Page 32 GAO-02-184 Status of Plans for Achieving Key Outcomes

l

 (GAO-01-633, June 2001). m

Space Transportation: Status of the X-33 Reusable Launch Vehicle Program

(GAO/NSIAD-99-176, Aug. 11, 1999); Space Transportation: Progress of the X-33

Reusable Launch Vehicle Program (GAO/T-NSIAD-99-243, Sept. 29, 1999); and

Space Transportation: Critical Areas NASA Needs to Address in Managing Its

Reusable Launch Vehicle Program (GAO-01-826T, June 20, 2001). n Export Controls: International Space Station Technology Transfers

(GAO/NSIAD-00-14, Nov. 3, 1999).
Appendix II: Comments From the National

Aeronautics and Space Administration

Page 33 GAO-02-184 Status of Plans for Achieving Key Outcomes

Appendix II: Comments From the National

Aeronautics and Space Administration
Appendix II: Comments From the National

Aeronautics and Space Administration

Page 34 GAO-02-184 Status of Plans for Achieving Key Outcomes (120091)
The General Accounting Office, the investigative arm of Congress, exists to

support Congress in meeting its constitutional responsibilities and to help

improve the performance and accountability of the federal government for the

American people. GAO examines the use of public funds; evaluates federal

programs and policies; and provides analyses, recommendations, and other

assistance to help Congress make informed oversight, policy, and funding

decisions. GAO’s commitment to good government is reflected in its core values

of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the

Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of

current reports and testimony and an expanding archive of older products. The

Web site features a search engine to help you locate documents using key words

and phrases. You can print these documents in their entirety, including charts and

other graphics.

Each day, GAO issues a list of newly released reports, testimony, and

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site

daily. The list contains links to the full-text document files. To have GAO E-mail

this list to you every afternoon, go to our home page and complete the easy-to-use

electronic order form found under “To Order GAO Products.”

The first copy of each printed report is free. Additional copies are $2 each. A

check or money order should be made out to the Superintendent of Documents.

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a

single address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

P.O. Box 37050

Washington, D.C. 20013

To order by phone: Voice: (202) 512-6000

TDD: (301) 413-0006

Fax: (202) 258-4066

GAO Building

Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)

Washington, D.C. 20013

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm,

E-mail: fraudnet@gao.gov, or

1-800-424-5454 (automated answering system).

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G. Street NW, Room 7149,

Washington, D.C. 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Mail or Phone

Visit GAO’s Document

Distribution Center

To Report Fraud,

Waste, and Abuse in

Federal Programs

Public Affairs