a

GAO

United States General Accounting Office

Report to the Committee on Science, 

House of Representatives

May 2004 NASA

Lack of Disciplined 

Cost-Estimating 

Processes Hinders 

Effective Program 

Management

GAO-04-642
www.gao.gov/cgi-bin/getrpt?GAO-04-642.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Allen Li at 

(202) 512-4841 or lia@gao.gov. 

Highlights of GAO-04-642, a report to the 

Committee on Science, House of 

Representatives 

May 2004

NASA

Lack of Disciplined Cost-Estimating 

Processes Hinders Effective Program 

Management 

Considerable change in NASA’s program cost estimates—both increases and 

decreases—indicates that NASA lacks a clear understanding of how much its 

programs will cost and how long they will take to achieve their objectives. 

For example, the development cost estimates for more than half of the 

27 programs that GAO reviewed have increased and for some programs this 

increase was significant—as much as 94 percent. Cost estimates changed for 

each of 10 programs that GAO reviewed in detail. For 8 of the 10 programs, 

the estimates increased. Although NASA cited specific reasons for the 

changes, such as technical problems and funding shortages, the variability 

in the cost estimates indicates that the programs lacked the sufficient 

knowledge needed to establish priorities, quantify risks, and make informed 

investment decisions, and thus predict costs. 

Most notably, NASA’s basic cost-estimating processes—an important tool 

for managing programs—lack the discipline needed to ensure that program 

estimates are reasonable. Specifically, GAO found that none of the 10 NASA 

programs that GAO reviewed in detail met all of GAO’s cost-estimating 

criteria, which are based on criteria developed by Carnegie Mellon 

University’s Software Engineering Institute. Moreover, none of the 

10 programs fully met certain key criteria—including clearly defining the 

program’s life cycle to establish program commitment and manage program 

costs, as required by NASA. In addition, only three programs provided a 

breakdown of the work to be performed. Without this knowledge, the 

programs’ estimated costs could be understated and thereby subject to 

underfunding and cost overruns, putting programs at risk of being reduced 

in scope or requiring additional funding to meet their objectives. Finally, 

only two programs have a process in place for measuring cost and 

performance to identify risks. 

NASA has limited ability to collect the program cost and schedule data 

needed to meet basic cost-estimating criteria. For example, as GAO has 

previously reported, NASA does not have a system to capture reliable 

financial and performance data—key to using effectively the cost-estimating 

tools that NASA officials state that programs employ. Further, without 

adequate financial and nonfinancial data, programs cannot easily track an 

acquisition’s progress and assess whether the program can meet its cost and 

schedule goals before it incurs significant cost and schedule overruns. NASA 

identified other barriers, including limited cost-estimating staff. According to 

NASA officials, several initiatives are under way to remove such obstacles 

and improve the agency’s cost-estimating practices. 

For more than a decade, GAO has 

identified the National Aeronautics 

and Space Administration’s (NASA) 

contract management as a high-risk 

area—in part because of NASA’s 

inability to collect, maintain, and 

report the full cost of its programs 

and projects. Lacking this 

information, NASA has been 

challenged to manage its programs 

and control program costs. The 

scientific and technical 

expectations inherent in NASA’s 

mission create even greater 

challenges—especially if meeting 

those expectations requires NASA 

to reallocate funding from existing 

programs to support proposed new 

efforts. 

Because cost growth has been a 

persistent problem in a number of 

NASA programs, GAO was asked to 

examine NASA’s cost estimating 

for selected programs, assess 

NASA’s cost-estimating processes 

and methodologies, and describe 

any barriers to improving NASA’s 

cost-estimating processes. To 

conduct GAO’s work, GAO 

analyzed a total of 27 NASA 

programs—10 of which GAO 

reviewed in detail. 

GAO is recommending that NASA 

take a number of actions to better 

ensure that the agency’s planned 

and recently implemented 

initiatives to improve its costestimating practices will result in 

sound cost estimates and thereby 

enable NASA to control its 

programs better. 
Page i GAO-04-642 NASA Cost Estimating

Contents

Letter 1

Results in Brief 2

Background 5

Development Cost Estimates Frequently Changed 6

Poor Estimating Processes and Methodologies Contributed to 

Wide Variations in Baseline Cost Estimates 12

NASA Has Begun to Address Certain Barriers to Effective 

Cost Estimating 19

Conclusions 24

Recommendations for Executive Action 24

Agency Comments and Our Evaluation 26

Appendixes

Appendix I: Scope and Methodology 30

Appendix II: Assessments of 10 Programs Reviewed in Detail 32

Gravity Probe B 34

Mars Exploration Rovers 36

Space Infrared Telescope Facility 38

Landsat-7 40

Aqua 42

Aura 44

Fluids and Combustion Facility 46

Hyper-X Program 48

Checkout and Launch Control System 50

Cockpit Avionics Upgrade 52

Appendix III: Summary Descriptions of the 17 Additional Programs 54

Space Science Enterprise 54

Earth Science Enterprise 56

Space Flight Enterprise 58

Appendix IV: Description of Earned Value Management 60

Appendix V: Comments from the National Aeronautics and 

Space Administration 64

Appendix VI: GAO Contact and Staff Acknowledgments 71
Contents

Page ii GAO-04-642 NASA Cost Estimating

Tables Table 1: Initial and Current Baseline Development Cost Estimates 

and Life-Cycle Cost Estimates for 27 NASA Programs 7

Table 2: Summary of Criteria Used to Assess 10 NASA Programs 

Reviewed 14

Table 3: Summary of Extent 10 NASA Programs Met Assessment 

Criteria 16

Table 4: Summary of the Number of Programs That Met, Partially 

Met, or Did Not Meet Criterion 33

Table 5: Thirty-Two Criteria for Evaluating the Quality of 

Management Systems 61

Figure Figure 1: History of Rebaselinings of 10 Programs’ Development 

Cost Estimates 9
Contents

Page iii GAO-04-642 NASA Cost Estimating

Abbreviations

AHMS Phase 1 Advanced Health Management System Phase I

ATP Alternate Turbopump Program

CAIV cost as an independent variable

CALIPSO Cloud-Aerosol Lidar and Infrared Pathfinder 

Satellite Observations

CARD cost analysis requirements description

CAU Cockpit Avionics Upgrade

CLCS Checkout and Launch Control System

CPR cost performance report

CRV Crew Return Vehicle

DOD Department of Defense

EOS Earth Observing System

EVM earned value management

FCF Fluids and Combustion Facility

GP-B Gravity Probe B

IFMP Integrated Financial Management Program

INTEGRAL International Gamma-Ray Astrophysics Laboratory

IPAO Independent Program Assessment Office

MERs Mars Exploration Rovers

MESSENGER Mercury Surface, Space Environment,

Geochemistry, and Ranging

NASA National Aeronautics and Space Administration

NMP-EO-1 New Millennium Program Earth Observing-1

OMB Office of Management and Budget

PMA President’s Management Agenda

SEER System Evaluation and Estimation of Resources

SEI Software Engineering Institute

SIRTF Space Infrared Telescope Facility

SOFIA Stratospheric Observatory for Infrared Astronomy

STEREO Solar Terrestrial Relations Observatory

TDRS Tracking and Data Relay Satellite Replenishment

TIMED Thermosphere, Ionosphere, Mesosphere Energetics, 

and Dynamics

WBS work breakdown structure
Contents

Page iv GAO-04-642 NASA Cost Estimating

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.
Page 1 GAO-04-642 NASA Cost Estimating

United States General Accounting Office

Washington, D.C. 20548

A

May 28, 2004 Letr

The Honorable Sherwood L. Boehlert

Chairman

The Honorable Bart Gordon

Ranking Minority Member

Committee on Science

House of Representatives

The lack of reliable financial and performance information has posed 

significant challenges to the National Aeronautics and Space 

Administration’s (NASA) ability to manage its largest and most costly 

programs effectively. For nearly 15 years, NASA contract management has 

been on GAO’s high-risk list—due in part to NASA’s inability to collect, 

maintain, and report the full cost of its programs and projects.1

 Without 

such information, NASA has consistently developed unrealistic cost and 

schedule estimates, which, at least in part, are reflected in the cost growth 

and schedule increases in many of its programs.

The demanding scientific and technical expectations inherent in NASA’s 

mission create even greater challenges for the agency to control program 

costs—especially if meeting those expectations requires NASA to 

reallocate funding from existing programs to support new efforts. Because 

cost growth has been a persistent problem on a number of NASA programs, 

you asked us to (1) identify initial cost estimates in selected NASA 

programs and any changes in those cost estimates, (2) assess NASA’s costestimating processes and methodologies, and (3) describe any barriers that 

make it difficult for NASA to improve its cost-estimating processes.

Our review focused on 27 of 68 NASA programs in the development phase 

as of April 2003 or that completed development in fiscal year 2001 or 2002. 

To assess NASA’s cost-estimating processes and methodologies, we 

conducted a more in-depth review of 10 of the 27 programs, which 

generally had the highest development cost estimate within five of NASA’s

1

 U.S. General Accounting Office, Major Management Challenges and Program Risks: 

National Aeronautics and Space Administration, GAO-03-114 (Washington, D.C.: 

Jan. 2003).
Page 2 GAO-04-642 NASA Cost Estimating

seven Enterprises. 2 Our work was conducted between February 2003 and 

March 2004 in accordance with generally accepted government auditing 

standards. For a complete description of our scope and methodology, see 

appendix I.

Results in Brief Many of the NASA programs that we reviewed cost more and took longer 

than was proposed at the time of congressional approval.3

 Several factors 

continue to put NASA projects at risk of increased cost and schedule 

delays. Most notably, NASA lacks the basic cost-estimating processes 

needed to establish priorities, quantify risks, and make informed 

investment decisions for its programs. Further, NASA has limited ability to 

collect, analyze, and use program cost and schedule data to identify and 

mitigate impediments to program success.

Current baseline development cost estimates for the 27 programs we 

reviewed varied considerably from the programs’ initial baseline 

estimates.4

 More than half of the programs’ development cost estimates 

increased, and for some programs, this increase was significant—as 

much as 94 percent. In addition, the baseline development estimates for 

each of 10 programs that we reviewed in detail were rebaselined—some as 

many as four times. For 7 of the 10 programs, the new baseline 

development estimate was an increase over the previous baseline estimate. 

Although NASA cited specific reasons for the cost growth and the 

recalculated baselines, such as technical problems and funding shortages, 

the variability in the cost estimates and the rebaselinings indicate that the 

programs lacked sufficient knowledge needed to make informed 

acquisition decisions.

2

 NASA’s Enterprises, listed in the background section of this report, function as primary 

business areas for implementing NASA’s mission. Each Enterprise has its own strategic 

goals, objectives, and implementation strategies.

3

 According to NASA, congressional approval occurs when the Congress appropriates 

design and development funds for the program.

4

 NASA defines baseline as the technical performance and content, technology application, 

schedule milestones, and budget (including contingency and allowance for program 

adjustment) that are documented in the approved program and project plans. Current 

baseline development cost estimates are as of April 2003.
Page 3 GAO-04-642 NASA Cost Estimating

Although an important tool for managing programs, NASA’s cost-estimating 

processes lack the discipline needed to ensure that program estimates are 

reasonable. Specifically, we found that none of the 10 NASA programs that 

we reviewed in detail met all of the criteria that we selected to assess 

NASA’s cost-estimating processes. Moreover, none of the 10 programs met 

certain key criteria—such as clearly defining the program’s life cycle. 

NASA procedures and guidelines require programs and projects to be 

managed on the basis of life-cycle cost—which the agency clearly 

defines—and that such cost be developed to establish the program’s 

commitment.5

 In addition, only three programs provided a complete 

breakdown of the work to be performed. Without knowing the full life cycle 

and the work to be performed, the programs’ estimated costs could be 

understated and thereby subject to underfunding and cost overruns, thus 

putting programs at risk of being reduced in scope or requiring additional 

funding to meet their objectives. Finally, only two programs had a process 

in place for measuring cost and performance to identify these potential 

risks and take action to avoid them.

NASA faces a number of barriers in meeting the cost-estimating criteria 

that we used to assess the 10 programs. For example, although NASA 

officials noted that programs are using cost-estimating tools, NASA 

generally lacks the data needed to employ these tools effectively. For more 

than a decade, we have reported that, despite repeated efforts, NASA has 

failed to develop a system to capture reliable financial and performance 

information. Most recently, we reported that the agency’s current effort to 

implement a modern integrated financial management system will not, as it 

is being implemented, routinely provide program managers and other key 

stakeholders and decision makers—including the Congress—with the 

financial information needed to measure program performance and ensure

5

 NASA defines life-cycle cost as the total of the direct, indirect, recurring, nonrecurring, and 

other related expenses incurred, or estimated to be incurred, in the design, development, 

verification, production, operation, maintenance, support, and retirement of a system over 

its planned life.
Page 4 GAO-04-642 NASA Cost Estimating

accountability.6 According to NASA officials, nonfinancial data, such as 

data on technology readiness levels, have also been difficult for the NASA 

cost-estimating community to obtain. Without adequate financial and 

nonfinancial data, programs cannot easily track an acquisition’s progress 

and assess whether the program can meet its cost and schedule goals 

before the program incurs significant cost and schedule overruns. NASA 

identified other barriers, including limited cost-estimating staff. According 

to NASA officials, there are several initiatives under way to remove such 

obstacles and improve the agency’s cost-estimating practices.

We are recommending that NASA take a number of actions to better ensure 

that the agency’s initiatives result in sound cost-estimating practices and 

are integrated into the project approval process. Specifically, we are 

recommending that NASA develop an integrated plan that includes specific 

actions that ensure that guidance is established on rebaselining and that 

programs have a well-defined process in place to measure cost and 

performance and identify potential risks. We are also recommending that 

NASA establish a framework for developing life-cycle cost estimates.

In its comments on a draft of this report, NASA stated that it concurred 

with our recommendations. NASA believes that it has already made 

progress toward achieving many of the improvements intended by the 

recommendations by developing new guidance, implementing management 

controls, and instituting additional levels of project oversight. These 

reforms to NASA's project development and implementation processes are, 

in our view, positive steps in addressing some of the problems discussed in 

our report. However, planned improvements must be integrated and 

enforced on an agency wide basis; our recommendations are in line with 

that thrust. NASA’s detailed comments are included as appendix V.

6

 U.S. General Accounting Office, Business Modernization: NASA’s Challenges 

in Managing Its Integrated Financial Management Program, GAO-04-255

(Washington, D.C.: Nov. 21, 2003); Business Modernization: Disciplined Processes Needed 

to Better Manage NASA’s Integrated Financial Management Program, GAO-04-118

(Washington, D.C.: Nov. 21, 2003); Business Modernization: NASA’s Integrated 

Financial Management Program Does Not Fully Address Agency’s External Reporting 

Issues, GAO-04-151 (Washington, D.C.: Nov. 21, 2003); and Information Technology: 

Architecture Needed to Guide NASA’s Financial Management Modernization, GAO-04-43

(Washington, D.C.: Nov. 21, 2003).
Page 5 GAO-04-642 NASA Cost Estimating

Background NASA’s programs encompass a broad range of complex and technical 

activities—from investigating the composition and resources of Mars to 

providing satellite and aircraft observations of Earth for scientific and 

weather forecasting. NASA currently funds more than 100 programs and 

projects in various phases of execution in 7 strategic Enterprises: Space 

Science, Earth Science, Biological and Physical Research, Aeronautics, 

Space Flight, Education, and Exploration Systems. Two NASA offices have 

key responsibilities in ensuring the effective execution of these programs: 

the Office of the Chief Financial Officer, which is responsible for providing 

oversight and financial management of agency resources and establishing 

related policy guidance, and the Office of Chief Engineer, which is 

responsible for ensuring development efforts and mission operations are 

planned and conducted using sound engineering practices.

More than two-thirds of NASA’s work force is made up of contractors and 

grantees, and 90 percent—or roughly $13 billion—of NASA’s annual budget 

is spent on work performed by its contractors. Since 1990, we have 

identified NASA’s contract management as a high-risk area. This 

assessment has been based in part on our repeated finding that NASA does 

not have good cost-estimating processes or the financial information 

needed to develop good cost estimates for its programs, making it difficult 

for NASA to oversee its contracts and control costs. For example, in July 

2002, we reported that an independent task force convened to assess the 

management of the International Space Station concluded that the 

program’s fiscal year 2002 through fiscal 2006 budget was not credible 

because of weaknesses in its cost-estimating processes.7

 The task force 

pointed out that these problems occurred because NASA had not instituted 

or had ignored many of the program’s control and contract oversight 

procedures—such as preparing a full life-cycle cost estimate—that should 

have alerted the agency to the growing cost problem and the need for 

mitigating actions. According to the cost analysis team that supported the 

task force, NASA’s focus on staying within annual budgets instead of 

managing total program costs was perhaps the single greatest factor in the 

program’s cost growth.

NASA’s unreliable cost estimates have significant implications for potential 

future endeavors, such as those outlined by the President in January of this 

7 U.S. General Accounting Office, Space Station: Actions Under Way to Manage Cost, but 

Significant Challenges Remain, GAO-02-735 (Washington, D.C.: July 17, 2002).
Page 6 GAO-04-642 NASA Cost Estimating

year. Specifically, the President called for a shift in NASA’s long-term focus, 

envisioning that NASA will retire the shuttle program as soon as assembly 

of the International Space Station is completed, planned for the end of the 

decade; develop a new crew exploration vehicle as well as launch human 

missions to the moon between 2015 and 2020, and build a permanent lunar 

base as a stepping stone for more ambitious missions. To achieve these 

goals, the President proposed spending $12 billion over the next 5 years—

about $1 billion of which would come from an increase in NASA’s budget, 

currently $15.4 billion—with the remaining $11 billion being reallocated 

from existing NASA programs.

Developing reliable cost estimates has been difficult for agencies across 

the federal government. The need for reliable cost estimates is at the heart 

of two of the five-governmentwide initiatives in the 2002 President’s 

Management Agenda (PMA); the two are “improved financial performance” 

and “budget and performance integration.”8

 These initiatives are aimed at 

ensuring that federal financial systems produce accurate and timely 

information to support operating, budget, and policy decisions and that 

budgets are performance-based. As part of these initiatives, the President 

calls for changes to the budget process to better measure the real cost and 

performance of programs. According to the PMA, accomplishing all of the 

crosscutting initiatives will matter little without the integration of agency 

budgets with performance.

Development Cost 

Estimates Frequently 

Changed

As of April 2003, the baseline development cost estimates for the programs 

we reviewed varied considerably from the programs’ initial baseline 

estimates. More than half of the programs’ development cost estimates 

increased, and for some programs, this increase was significant. The 

baseline development cost estimates for each of the 10 programs we 

reviewed in detail were rebaselined—that is, recalculated to reflect new 

costs, time frames, or resources associated with program changes in 

program objectives, deliverables, or scope and plans. Although NASA 

provided specific reasons for the increased cost estimates and 

rebaselinings—such as delays in the development or delivery of key 

system components and funding shortages—it does not have guidance for 

determining when rebaselinings are justified. Such criteria are important to 

instilling discipline in the cost-estimating process.

8

 The other three initiatives are strategic human capital management, competitive sourcing, 

and expanded electronic government.
Page 7 GAO-04-642 NASA Cost Estimating

Most of the 27 programs we reviewed experienced a change in their 

development costs estimates. While 8 of the 27 programs experienced 

slight decreases in their development cost estimates, 17 experienced 

cost growth—as much as almost 94 percent. The remaining two programs 

had no change. Ten of the 17 programs’ cost growth was greater than 

25 percent. Table 1 shows the development cost estimate changes from 

the initial baseline to the baseline as of April 2003 and the life-cycle cost 

estimate for each of the 27 programs. The 10 programs that we reviewed 

in detail are shaded and italicized. (See app. II for assessments of the 

10 programs and app. III for descriptions of the remaining 17 programs.) 

Table 1: Initial and Current Baseline Development Cost Estimates and Life-Cycle Cost Estimates for 27 NASA Programs 

Then-year dollars in millions

Baseline development cost estimate

Program, by Enterprise Initial

Current 

(as of April 

2003)a

Percent 

change

Life-cycle cost estimate 

(as of April 2003)a

Space Science

Space Infrared Telescope Facility (SIRTF)b $472.0 $610.5 29.3 $1,170.6

2003 Mars Exploration Rovers (MERs) 657.2 767.0 16.7 806.3

Gravity Probe B (GP-B) 529.6 709.3 33.9 734.9

Strastospheric Observatory for Infrared 

Astronomy (SOFIA) 234.8 373.0 58.9 604.5

Solar Terrestrial Relations Observatory 

(STEREO) 404.7 302.1 (25.4) 423.0

Mercury Surface, Space Environment, 

Geochemistry and Ranging (MESSENGER) 325 235.1 (27.7) 337.7

Herschel 103.7 72.7 (29.9) 277.6

Thermosphere, Ionosphere, Mesosphere 

Energetics and Dynamics (TIMED) 176.8 176.2 (0) 253.5

Solar-B 99.3 80.4 (19.0) 146.4

Rosetta 28.4 40.1 41.2 106.0

International Gamma-Ray Astrophysics 

Laboratory (INTEGRAL) 8.2 11.9 45.1 51.2

Earth Science

Terra 1,309.1 1,393.2 6.4 1,451.7

Aqua 1,005.5 952.4 (5.3) 1,050.6

Aura 762.5 764.6 0.3 788.5

Landsat-7 445.8 508.8 14.1 508.8
Page 8 GAO-04-642 NASA Cost Estimating

Source: NASA.

Note: The draft NASA Cost Estimating Handbook 2002 defines then-year dollars as dollars that are 

escalated into the time period of performance of a contract. It further states that then-year dollars are 

sometimes referred to as escalated costs, inflated costs, or real-year dollars. NASA normally uses the 

term—real-year dollars.

a

Includes launch vehicle cost where applicable.

b

SIRTF was renamed the Spitzer Space Telescope in December 2003.

c

Because Hyper-X is classified as a test program, there is no life-cycle cost estimate.

d

A life-cycle cost estimate was not developed for the Tracking and Data Relay Satellite Replenishment 

program because it is currently in pre-phase A (conceptual definition). According to a NASA official, a 

life-cycle cost estimate will be determined before it enters phase C/D (design, development, test, and 

evaluation).

e

A life-cycle cost estimate was not developed for the X-38 Crew Return Vehicle program because the 

program was cancelled in 2003, and the program’s contracts remained undefinitized at termination—

that is, the final price or estimated cost and fee were not negotiated and mutually agreed to by NASA 

and the contractor.

f

A life-cycle cost estimate was not developed for the CLCS program because it was canceled due to 

excessive cost growth.

New Millennium Program Earth Observing-1 

(NMP-EO-1) $111.7 $176.4 57.9 $192.5

SeaWinds 130.2 148.8 14.3 160.1

Cloud-Aerosol Lidar and Infrared Pathfinder 

Satellite Observations (CALIPSO) 98.0 133.9 36.6 150.9

Jason-1 77.5 87.8 13.3 127.8

Biological and Physical Research

Fluids and Combustion Facility (FCF) 118.9 114.1 (4.0) 132.0

Aeronautics

Hyper-X (X-43A) 167.0 227.0 35.9 c

Space Flight

Alternate Turbopump Program (ATP) 1,056.0 764.0 (27.7) 982.0

Cockpit Avionics Upgrade (CAU) 442.0 454.0 2.7 514.0

Advanced Health Management System Phase I 

(AHMS Phase 1) 55.0 55.0 (0) 55.0

Tracking and Data Relay Satellite Replenishment 

(TDRS) 937.0 518.1 (44.7) d

X-38 Crew Return Vehicle (CRV) 792.0 1,025.0 29.4 e

Checkout and Launch Control System (CLCS) 206.0 399.0 93.7 f

(Continued From Previous Page)

Then-year dollars in millions

Baseline development cost estimate

Program, by Enterprise Initial

Current 

(as of April 

2003)a

Percent 

change

Life-cycle cost estimate 

(as of April 2003)a
Page 9 GAO-04-642 NASA Cost Estimating

The development cost estimates for each of the 10 programs that we 

reviewed in detail have been rebaselined—for some programs, as many as 

four times—and for 7 of the 10 programs, the cost estimate increased each 

time it was rebaselined (see fig. 1).

Figure 1: History of Rebaselinings of 10 Programs’ Development Cost Estimates

a

SIRTF was renamed the Spitzer Space Telescope in December 2003.

Program Baseline

in millions

FY 2000 by quarter FY 2001 by quarter FY 2002 by quarter FY 2003 by quarter

SIRTF

MERs

GP-B

Aqua

Aura

Landsat-7

FCF

Hyper-X

CAU

CLCS

$472

$657

$530

$1,006 

$763 

$446 

$119

$167

$442

$206

1 2 3 4

c

Nov

$399d

1 2 3 4 1 2 3 4 1 2 3 4

b

b

May

$615

Apr

$631

Oct

$697

Mar

$729

July

$534

May

$667

Jan

$638

Apr

$669

Aug

$712

Apr

$767

Feb

$709

Jan

$183

Jan

$217

Sept

$227

Apr

$114

Source: NASA.

a

Then-year dollars in millions

Oct

$454
Page 10 GAO-04-642 NASA Cost Estimating

b

The baseline development estimates for the Aura and Aqua projects were rebaselined once as a 

result of a restructuring of the overall Earth Observing System (EOS) program in 1995 to address 

affordability issues. Before EOS’ restructuring, the baseline was $524 million for Aura and $1.2 billion 

for Aqua. However, according to NASA officials, both the Congress and NASA recognize the revised 

baseline estimates as the initial baseline estimates.

c

Landsat-7’s initial baseline development estimate was established by the Department of Defense 

(DOD), which originally had responsibility for managing the program. A 1994 Presidential Directive 

later reassigned the program to a joint NASA, National Oceanic and Atmospheric Administration, and 

U.S. Geological Survey program, with NASA having responsibility for the development and launch of 

the satellite and development of the ground system. Landsat-7 also became a part of the EOS 

program. In 1995, NASA established a revised initial baseline development estimate for Landsat-7, 

which according to NASA officials is recognized by the Congress and NASA as the initial baseline 

development estimate. DOD’s initial baseline estimate was not available.

d

CLCS was rebaselined twice, but the second rebaselined estimate for CLCS was not established 

because NASA terminated the program due to the program’s excessive cost growth.

For the 10 programs we reviewed in detail, NASA cited specific reasons for 

changes in the baseline development cost estimates and the recalculated 

baselines—many of which were related to technical problems and 

subsequent delays in the development or delivery of key system 

components, and insufficient funding and reserves, as illustrated in the 

following examples:

• Technical problems in the MERs program required a significant redesign 

of components and the development of a new landing system. Two of 

MERs’ three rebaselinings were also the result of inadequate reserves. 

According to NASA officials, without the rebaselinings, the development 

cost “to go”9

 would have drained the program’s reserves.

• The increase in CLCS’s development cost estimate and rebaselining was 

the result of poorly defined requirements and design, software 

integration problems, and fundamental changes in the project’s 

management structure and contractors’ approach to the work. The 

project, which experienced an almost 94 percent increase in its baseline 

development cost estimate, was ultimately terminated.

• The GP-B program—which was rebaselined four times—experienced 

significant schedule slippages due to repeated technical problems, 

including failures in the probe’s heat exchanger, the need for additional 

testing, payload electronics delays, and thermal vacuum test failures.

9

 According to a NASA project manager, “to go” means from this point forward to 

completion of the project, given the current status of the project and the resources available 

to complete it.
Page 11 GAO-04-642 NASA Cost Estimating

• Schedule slippages in the SIRTF program—which contributed to 

increases in the program’s baseline development cost estimate and four 

rebaselinings of the estimate—were caused by delays in the delivery of 

components, flight software, and the mission operation system as well 

as launch delays that resulted from a handling accident involving a 

global positioning system payload and concerns of delamination on the 

launch vehicle’s solid rocket motors.

• Changes in development cost estimates for the CAU program were 

primarily the result of the program’s expanded scope, which occurred in 

October 2002, to produce modification kits that would allow the CAU 

upgrade to be installed into the orbiters.

• The Hyper-X program experienced three rebaselinings, and according to 

the project manager, the program will be rebaselined again in the near 

future. The rebaselinings were due to schedule slippages resulting from 

the need to fund an investigation of the problems experienced in the 

first Mach 7 flight vehicle—which was destroyed in flight—and related 

corrective actions to the second Mach 7 flight.10

Revised contract requirements, funding changes, or the realization that 

program goals are not achievable may require a formal rebaselining. 

However, NASA has not defined or provided guidance or restrictions on 

rebaselining to ensure that programs consistently and appropriately apply 

rebaselinings and do not adjust their baseline cost estimates whenever the 

estimates become unmanageable. Further, NASA lacks a process for 

systematically identifying and assessing programs that are not achieving 

their cost, schedule, and performance goals. Such a process has been 

employed by the Department of Defense (DOD), which also relies heavily 

on contractors to deliver complex, cutting-edge technologies to meet its 

mission. Specifically, DOD must report to the Congress programs that incur 

a cost growth of 15 percent or more in the program baseline. Moreover, 

DOD must justify the continuation of acquisition programs that incur a cost 

growth of 25 percent or more in the program baseline by certifying that 

specific criteria have been met—including that the new cost estimates are 

reasonable.11 Under such a process, 5 of the 10 programs that we reviewed 

10 The second Hyper-X flight vehicle flew successfully at Mach 7 speed in March 2004 

(see app. II).

11 10 U.S.C. 2433.
Page 12 GAO-04-642 NASA Cost Estimating

in detail would have been required to report to the Congress, and 4 of 

the 5 programs would have had to certify that their new cost estimates 

were reasonable.

Poor Estimating 

Processes and 

Methodologies 

Contributed to 

Wide Variations 

in Baseline 

Cost Estimates

NASA has yet to implement a well-defined process for estimating the 

cost of its programs—a weakness we and NASA’s Inspector General have 

repeatedly reported.12 Recognizing the need for such a process, NASA 

developed a cost-estimating handbook in 2002—the first such guidance 

provided to its cost-estimating community and program and project 

managers.13 Despite this effort, the programs we reviewed failed to follow 

key cost-estimating processes, including developing and documenting full 

life-cycle cost estimates, summarizing estimates according to the current 

breakdown of work to be performed, conducting an uncertainty analysis, 

performing an independent review of contractors’ cost estimates, and later 

using earned value management (EVM) to assess progress.14

12 See, for example, GAO-04-118; GAO-04-255; GAO-03-114; U.S. General Accounting Office, 

Space Station: Actions Under Way to Manage Cost, but Significant Challenges Remain, 

GAO-02-735 (Washington, D.C.: July 17, 2002); NASA Program Costs: Space Missions 

Require Substantially More Funding Than Initially Estimated, GAO/NSIAD-93-97

(Washington, D.C.: Dec. 31, 1992); and NASA Office of Inspector General, Final 

Management Letter on Failures in Cost Estimating and Risk Management Weaknesses 

in Prior Space Launch Initiative Assignment Numbers A-01-049-01and A-01-049-02, 

IG-03-023 (Washington, D.C.: Sept. 29, 2003).

13 The cost-estimating handbook is in draft form, but NASA made it available for official use 

by its cost-estimating community and program managers. NASA expected to complete the 

handbook by May 2004.

14 EVM compares the actual work performed at certain stages of a job to its actual costs—

rather than comparing budgeted and actual costs, the traditional management approach to 

assessing progress. By measuring the value of work that has been completed at certain 

stages in a job, EVM can alert program managers, contractors, and administrators of 

potential cost overruns and schedule delays before they occur and of problems that need 

correcting before they worsen. For a more detailed discussion of EVM, see appendix IV.
Page 13 GAO-04-642 NASA Cost Estimating

Reflecting Office of Management and Budget (OMB) guidance and best 

practices of government and industry leaders, NASA requires that full 

life-cycle cost estimates be prepared using full cost accounting,15 that 

estimates be summarized according to the current breakdown of work to 

be performed, and that major changes be tracked to the life-cycle cost. In 

its draft cost-estimating handbook, NASA lists a number steps that are 

integral to preparing a reliable life-cycle cost estimate, including 

preparing or obtaining a cost analysis requirements description (CARD),16

developing ground rules and assumptions, and developing cost range and 

risk assessments.

Carnegie Mellon University’s Software Engineering Institute (SEI) 17 echoes 

the need for reliable cost-estimating processes in managing software 

implementations—identifying tasks to be estimated, mapping the estimates 

to the breakdown of work to be performed, and identifying and explaining 

assumptions are among SEI’s requisites for producing reliable cost 

estimates. To evaluate the cost-estimating processes of the 10 NASA 

programs that we reviewed in detail, we selected 14 criteria based on SEI 

checklists (see table 2).18 Many of these criteria are included in NASA’s 

cost-estimating guidance.

15 The full cost of a project is the sum of all direct costs, service costs, and general 

administrative costs. Full cost accounting ties all NASA agency costs (including civil service 

personnel costs) to major projects.

16 A CARD provides a system technical description and programmatic information to create 

a common baseline used by the project team to develop estimates.

17 SEI is a government-funded research organization that is widely considered an authority 

on software implementation. 

18 SEI developed checklists to help evaluate software costs and schedule. However, SEI 

states that these checklists are equally applicable to hardware and systems engineering 

projects. 
Page 14 GAO-04-642 NASA Cost Estimating

Table 2: Summary of Criteria Used to Assess 10 NASA Programs Reviewed

Criterion Purpose/Significance

The objectives of the estimate are stated 

in writing.

The objectives of the program must be clearly stated in a concise document for the 

cost estimator to use to develop the cost estimate. NASA guidance states that NASA 

programs and projects are to be defined as activities that have defined objectives 

along with goals and requirements.

The life cycle to which the estimate applies is 

clearly defined.

The life cycle must be clearly defined to ensure that the full cost of the program—that 

is, all direct and indirect costs for planning, procurement, operations and 

maintenance, and disposal—are captured. The draft NASA cost-estimating 

handbook states that a life-cycle cost estimate provides “an exhaustive accounting of 

all resources necessary to develop, deploy or field, operate, maintain, and dispose of 

a system over its lifetime.” The handbook defines life cycle as the program’s or 

project’s “total life, beginning with mission feasibility and extending through operation 

and disposal or conclusion of the system or program.”

The task has been appropriately sized. This criteria asks if the appropriate metric was used in the development of the 

estimate, such as the size of a software product with expected amount of reuse.

The estimated cost and schedule are consistent 

with demonstrated accomplishments on other 

projects.

In other words, estimates have been validated by relating them back to demonstrated 

performance on completed projects.

A written summary of parameter values and their 

rationales accompany the estimate.

This criterion refers to the underlying cost-estimating methodology. If a parametric 

equation was used to generate the estimate, then the parameters that feed the 

equation must be provided along with an explanation of why they were chosen.

Assumptions have been identified and explained. The draft NASA draft cost-estimating handbook states that assumptions are a critical 

step in any estimate and should be clearly prominent in all documentation for the 

estimate. Accurate assumptions can prevent inaccurate or misleading estimates.

A structured process such as a template or 

format has been used to ensure that key factors 

have not been overlooked.

This criterion refers to whether or not the program has established a work breakdown 

structure (WBS)—that is a structure that organizes, defines, and graphically displays 

the individual work units to be performed. NASA policy guidance calls for breaking 

down work into smaller units to facilitate cost- estimating and project and contract 

management as well as to help ensure that all relevant costs are captured. The 

guidance requires that a preliminary WBS be developed during the formulation 

phase, and that a final WBS be generated following contractor selection or approval 

to implement. The guidance further requires that programs describe the overall WBS 

structure and the content of each individual element of the WBS.

Uncertainties in parameter values have been 

identified and quantified.

Again this criterion refers to the underlying cost-estimating methodology. For all major 

cost drivers, an uncertainty analysis should be performed to assess the risk 

associated with the cost estimate.

If a dictated schedule has been imposed, an 

estimate of the normal schedule has been 

compared to the additional expenditures required 

to meet the dictated schedule.a

This criterion asks whether a dictated schedule was imposed on the program, that is, 

whether the program was forced to accelerate the schedule in order to meet 

requirements. If this occurred, then the impacts to the cost estimate need to be 

calculated and provided. 

If more than one cost model or estimating 

approach has been used, any differences in 

results have been analyzed and explained.

This criterion checks to ensure that the primary methodology or cost model 

results are consistent with any secondary methodology (for example, cross checks) 

performed.
Page 15 GAO-04-642 NASA Cost Estimating

Sources: NASA and SEI.

a

Does not apply to all programs.

Despite NASA requirements and OMB and SEI guidance, few of the 

10 programs that we reviewed in detail met even a third of these criteria; 

only one met half. Further, none of the programs fully met certain key 

criteria. For example, none provided a complete life cycle with definitions 

or a complete description of the methodology used to generate the 

complete cost estimate, such as data sources and uncertainties. According 

to the draft NASA cost-estimating handbook, a reliable life-cycle cost 

estimate is critical to making realistic decisions about developing or 

producing a system and to determining the appropriate scope or size of a 

program. NASA guidance also calls for breaking down the work to be 

performed into smaller units to facilitate cost estimating and program and 

contract management and to help ensure relevant costs are not omitted. 

However, only 3 of the 10 programs provided a complete breakdown of the 

work to be performed. Table 3 shows for each program the applicable 

criteria that were met, partially met, or not met.19 (See app. II for a program 

by program assessment.) 

Estimators independent of the performing 

organization concurred with the reasonableness 

of the parameter values and estimating 

methodology.

NASA policy guidance states, “when a project under a program has an estimated 

NASA life-cycle cost greater than $150 million, an independent life-cycle cost 

analysis is required during formulation in conjunction with initiating the preliminary 

design.”

Estimates are current. Estimates should be updated whenever changes to requirements affect cost or 

schedule. NASA policy guidance requires that the life-cycle cost estimate be updated 

prior to each budget submission.

The results of the estimate have been integrated 

with project planning and tracking.

NASA policy guidance requires that a life-cycle cost be developed to establish a 

program/project commitment, assessed at major reviews, and updated for each 

budget submission and should use currently available full cost initiative guidance.

Earned value reporting has been used to 

manage the program.

NASA policy guidance requires program and project managers to “ensure that EVM 

provisions and requirements are included in requests for proposals and contracts and 

ensure that an effective surveillance program is in place to provide assurance that 

EVM data are valid and that the contractor’s integrated management system remains 

in compliance with the EVM criteria.” The guidance further requires each program 

and project to periodically generate estimates at completion, perform cost and 

schedule variance analyses based upon pre-established thresholds, and prepare 

corrective action plans where necessary.

(Continued From Previous Page)

Criterion Purpose/Significance

19 If a program provided substantiating evidence for a criterion, we determined that the 

program “fully met” the criterion. If partial evidence was provided for a criterion, we 

determined the program “partially met” the criterion. If no evidence was found, then we 

determined that the criterion was “not met.”
Page 16 GAO-04-642 NASA Cost Estimating

Table 3: Summary of Extent 10 NASA Programs Met Assessment Criteria

Space science Earth science

Biological 

and physical 

research Aeronautics Space flight

Criteria for 

cost estimating GP-B MERs SIRTF Landsat-7 Aqua Aura FCF Hyper-X CLCS CAU

Objectives stated 

in writing

NM P P NM P M P NM P M

Life cycle clearly 

defined

P P P P P P P P P P

Tasks 

appropriately 

sized

NM NM P P NM P P NM P M

Estimates based 

on demonstrated 

programs

NM P P P NM P P NM P P

Parameter values 

and rationale 

documented

NM NM P NM NM NM P NM P P

Assumptions 

identified and 

explained

P P P NM P P P NM M M

Structured format 

captures all costs

P M P P P P M P P M

Uncertainties 

identified and 

quantified

NM NM P P NM NM NM NM NM P

Accelerated 

schedules show 

cost impacts

P P N/A P P N/A N/A N/A P N/A

More than one 

estimating 

approach used

NM NM P NM NM NM P NM P P

Independent and 

program 

estimates concur

P M P P P P P P M M

Estimates reflect 

changes over 

time

P M P P P M P P P M

Key: M = Met, P = Partially met, NM = Not met
Page 17 GAO-04-642 NASA Cost Estimating

Sources: NASA (data), SEI (criteria), GAO (analysis).

Key: M = Met, P = Partially met, NM = Not met

Failing to meet these criteria puts programs at certain risk. For example, 

underestimating a program’s full life-cycle costs creates the risk that a 

program could be underfunded and subject to major cost overruns, which 

would ultimately result in the program being reduced in scope or additional 

funding being requested and appropriated to ensure the program meets its 

objectives. Conversely, overestimating life-cycle costs creates the risk that 

a program will be deemed unaffordable and would, therefore, go unfunded. 

Without a complete WBS, NASA programs cannot ensure that the life-cycle 

cost estimates have captured all relevant costs, which again can result in 

underfunding and cost overruns. Further, inconsistent WBS estimates 

across programs can create problems of double counting or, worse, 

underestimating costs when using historical program costs as a basis for 

projecting future costs on similar programs.

Despite the uncertainty inherent in estimating the cost of emerging 

technologies, all of the 10 programs we reviewed also failed to conduct 

an uncertainty analysis to assess risks associated with the cost 

estimates. Instead, the programs expressed their cost estimates as 

point values—which implies certainty—not as ranges or numbers with 

confidence levels.20 Performing an uncertainty analysis, such as a Monte

Estimates used 

for program 

tracking

M M M P P P M M P M

Earned value 

reporting used

P P P P P P M P P M

(Continued From Previous Page)

Space science Earth science

Biological 

and physical 

research Aeronautics Space flight

Criteria for 

cost estimating GP-B MERs SIRTF Landsat-7 Aqua Aura FCF Hyper-X CLCS CAU

20 For example, a cost estimate of $1 million could be presented either as a range of $900,000 

to $1.1 million or as $1 million with a confidence interval of 90 percent, indicating that there 

is a 10-percent chance that the cost will exceed the estimate.
Page 18 GAO-04-642 NASA Cost Estimating

Carlo simulation,21 quantifies the amount of cost risk within a program. 

Only by quantifying the cost risk can management make informed 

decisions about risk mitigation strategies. Quantifying cost risks also 

provides a benchmark against which future progress can be measured. 

Without this knowledge, NASA may have little specific basis to 

determine adequate financial reserves, schedule margins, and technical 

performance margins to provide managers the flexibility needed to 

address programmatic, technical, cost, and schedule risks, as required by 

NASA policy.

Seven of the 10 programs also failed to have an independent review of 

contractors’ cost estimates—as required by NASA. Instead, programs 

established their budgets based on contractor proposals—particularly 

problematic since many contractors could bid low in order to win the 

contract. To ensure contractor costs are realistic, NASA procedures and 

guidelines specifically require programs to ensure that independent 

reviews are conducted and that these reviews address project life-cycle 

costs, risk management plans, as well as technical issues. Without such 

reviews, NASA decision makers lacked the benchmarks needed to assess 

the reasonableness of the contractors’ proposed costs, limiting NASA’s 

ability to make sound investment decisions and accurately assess 

contractor performance.

Finally, only two programs used EVM—an approach used by DOD and 

leading companies to provide meaningful assessments of a program’s 

progress by comparing the value of work performed to its costs, rather 

than the traditional management approach of comparing budgeted and 

actual costs, which can provide a distorted view of a program’s progress. 

(For a detailed discussion of EVM, see app. IV.) By using the value of 

completed work as a basis for estimating the cost and time needed to 

complete the program, EVM can alert program managers to potential 

problems early in the program. NASA requires that EVM be used on all 

significant contracts—that is, research and development contracts with 

a total anticipated final value of $70 million or more, and production

21 A Monte Carlo simulation randomly generates values for uncertain variables over and over 

to simulate a model. Without the aid of simulation, a model will only reveal a single 

outcome, generally the most likely or average scenario, but after hundreds or thousands of 

trials, one can view the statistics of the results and the certainty of any outcome. 
Page 19 GAO-04-642 NASA Cost Estimating

contracts with a total anticipated final value of $300 million or more—

which includes all of the 10 programs we reviewed in detail.22 Although the 

program managers for all 10 programs stated that EVM was used in their 

projects, only two programs provided cost performance reports, indicating 

a true EVM process was in place. The remaining eight programs relied on 

NASA Form 533, which captures planned and actual obligations and 

expenditures—not the value of the work performed.23 Without a true EVM 

process, programs cannot readily determine if a program is at risk of cost 

and schedule overruns until it is too late to make programmatic changes to 

avoid these risks.

NASA Has Begun to 

Address Certain 

Barriers to Effective 

Cost Estimating

There are several impediments that NASA needs to overcome to implement 

effective cost-estimating practices. These include the lack of reliable 

financial data and other performance information; lack of trained EVM 

staff, data analysis tools, and incentive for supporting and implementing 

EVM; and ineffective use of cost analysts. NASA has initiated several 

measures to begin addressing some of these impediments.

Utility of Cost-Estimating 

Tools Depends on the 

Reliability of NASA’s 

Financial and Performance 

Data

According to NASA officials, state-of-the-art cost-estimating tools have 

been funded and implemented. For example, NASA officials told us that 

commercial-off-the-shelf models have been used to estimate hardware and 

software acquisition costs and quantify the level of uncertainty surrounding 

cost estimates. However, these cost-estimating tools are only as good as the 

data they rely on to develop the estimates. For more than a decade, we 

have reported that NASA has failed to develop a system to capture reliable 

financial and performance information, posing significant challenges to 

NASA’s ability to estimate and control program costs. Over the past year 

alone, we issued numerous reports on NASA’s Integrated Financial 

Management Program (IFMP)—the agency’s third and most recent effort to 

implement a modern, integrated financial management system. Specifically, 

we found that IFMP—which is under the responsibility of the Program 

Executive Officer for IFMP—will not, as it is being implemented, routinely 

22 See Earned Value Management, NASA Policy Directive 9501.3A (Aug. 3, 2002) and Earned 

Value Management Implementation on NASA Contracts, NASA Procedural Requirements 

9501.3 (Nov. 24, 2002).

23 Form 533 captures financial information that is used as basis for the financial 

management and budget activities within projects and NASA-wide.
Page 20 GAO-04-642 NASA Cost Estimating

provide program managers and other key stakeholders and decision 

makers—including the Congress—with the financial related information 

needed to measure program performance and ensure accountability. For 

example, the core financial module (considered the backbone of the 

system) does not appropriately capture property, plant, and equipment, as 

well as material in its general ledger at the transaction level—which is 

needed to provide independent control over these assets. In addition, 

NASA implemented the system before it had the capability to capture the 

full costs of its programs and projects. According to headquarters officials, 

collecting nonfinancial data crucial to cost estimating—such as technology 

readiness levels, parts counts, and team and management experience and 

skill ratings—has also been difficult.

Use of EVM Has Been 

Undermined by a Lack of 

Trained Staff, Data Analysis 

Tools, and Incentive

According to headquarters officials, agencywide EVM implementation 

efforts began in 1996 and are recognized by NASA management as a key 

tool in monitoring and measuring cost trends in higher risk project 

elements—a tool that serves as an early warning of the need for cost-risk 

mitigation actions to maintain control of program costs. These officials 

stated that EVM has been applied to the International Space Station 

Program24 and with varying levels of emphasis to other programs and 

projects at different NASA centers.25 While all of the program managers for 

the 10 programs that we reviewed in detail stated that they used EVM, only 

2 of the programs used a true EVM process.

NASA headquarters officials identified several challenges that have 

affected the agency’s ability to implement EVM effectively, including a lack 

of staff and data analysis tools. According to officials, resource constraints 

have prevented the agency from staffing many project offices with 

appropriate personnel to fulfill all project functions. In addition, there has 

been little or no priority to include a trained EVM analyst, even if one were 

available. Headquarters officials also noted that EVM has been hampered 

by the lack of a practical automated software data analysis tool. Without 

such a tool, analyzing the contractors’ EVM cost performance reports, 

which contain significant amounts of data, became a cumbersome 

undertaking that often resulted in incomplete and untimely analyses, 

24 The International Space Station Program was not a part of our review.

25 NASA has nine centers located around the country and owns the Jet Propulsion 

Laboratory, which is operated by the California Institute of Technology.
Page 21 GAO-04-642 NASA Cost Estimating

providing little usefulness to inform management decisions. A lack of 

incentive to support EVM has further undermined its use. Some project 

managers whom we spoke with are skeptical about the benefits of EVM 

and argue that it has failed to help them manage or control program costs. 

According to NASA headquarters officials, during proposal and contract 

negotiation phases, contractors have also suggested not using EVM as a 

way to reduce contract costs. While EVM was included in most contracts 

for the 10 programs we reviewed in detail—as required by NASA policy—it 

was used only in two programs as a cost-estimating tool. In general, EVM 

has been viewed by NASA as a financial reporting tool. Consequently, there 

is little incentive to use EVM because the data needed to report financial 

activity is captured elsewhere, such as in Form 533.

Ineffective Use and 

Placement of Cost Analysts 

across the Agency’s Cost 

Activities also Hinders 

NASA’s Efforts to Improve 

Its Cost-Estimating 

Practices

NASA’s efforts to improve its cost-estimating processes have also been 

undermined by ineffective use of its limited number of cost-estimating 

analysts. For example, headquarters officials state that as projects entered 

the formulation phase, they have typically relied on program control and 

budget specialists—not cost analysts—to provide the financial services to 

manage projects. Yet budget specialists are generally responsible for 

obligating and expending funding—not for conducting cost analyses that 

underlie the budget or ensuring budgets are based on reasonable cost 

estimates—and, therefore, tend to assume that the budget is realistic. 

While NASA officials state that its cost-estimating staff is too limited to be 

involved in day-to-day project execution activities, they agreed that the 

cost analysts could be more effectively used throughout the life cycle—

particularly when projects are rebaselined and independent cost estimates 

of project changes must be performed.

In some cases, cost analysts are not appropriately located in the 

organization, which may compromise controls NASA has in place to ensure 

reasonable cost estimates. For example, some cost analysts at NASA’s 

centers are located with senior systems engineers in systems management 

organizations, while others are not. According to NASA officials, housing 

the cost analysts with senior systems engineers has two key benefits. First, 

the systems engineers generally conduct systems analyses to help ensure 

that a program’s requirements are properly established and that the design 

and validity meet the requirements. Such analyses can greatly inform the 

development of reasonable cost estimates. Second, the systems 

engineering offices afford some measures of independence for cost 

estimating, which, according to NASA cost- estimating guidance and 

procedures, is important to the overall project management process. 
Page 22 GAO-04-642 NASA Cost Estimating

However, NASA officials stated that several of its centers’ cost analysts are 

in the advocacy chain of command—not housed with senior systems 

engineers. For example, one center’s 15 cost analysts work in the center’s 

Office of the Chief Financial Officer—which is responsible for directing the 

development and execution of the center’s budget—not in the systems 

management organization, which is independent from the rest of the 

center. As a result, the costs analysts’ estimates may not be adequately 

informed by the systems engineers and may lack the objectivity required to 

ensure that the criteria for independence have been met.

Efforts Under Way 

to Remove Some 

Barriers and Improve 

Cost Estimating

NASA has several initiatives under way to improve the agency’s 

cost-estimating processes. First, NASA has established a Cost Analysis 

Division in the Office of the Comptroller to strategically manage analyses 

related to directing and funding research, improving cost-estimating 

processes and practices, and providing cost-estimating tools and training 

throughout the agency. The division also provides, along with the 

Independent Program Assessment Office (IPAO), the last independent cost 

estimate of projects before the information is released externally. These 

efforts are being coordinated through a steering committee composed of 

the managers of the cost analysis organizations from each of the centers 

and IPAO’s deputy director.

NASA is revising the cost sections in its governing procedures and 

guidelines and is finalizing its cost-estimating handbook to reflect these 

changes.26 These documents will require the routine use of probabilistic 

cost risk analysis, a CARD document, cost as an independent variable 

(CAIV), and EVM. The CARD supports the project life-cycle cost estimate 

and a congressionally required independent cost estimate. Agency officials 

note that while there has been some use of CARD in the agency, its first 

concentrated and successful use was in the 2001 to 2002 independent 

cost estimate for the International Space Station program. According 

to headquarters officials, NASA’s revised guidance and finalized 

cost-estimating handbook will provide direction and guidance for fully 

implementing the use of CARDs for major development projects. Although 

26 According to NASA officials, revisions of NASA’s current governing program and project 

guidance—NASA Procedures and Guidelines 7120.5B, NASA Program and Project 

Management Processes and Requirements (Nov. 21, 2002)—is expected to be completed 

by August 2004, and the draft cost-estimating handbook was expected to be finalized by 

May 2004.
Page 23 GAO-04-642 NASA Cost Estimating

NASA calls for CAIV to be used routinely and notes that CAIV demonstrates 

a commitment to evolutionary acquisition, it has yet to provide guidance 

on its implementation. NASA headquarters officials stated that guidance 

relating to improvements in the collection of cost data is also being 

reflected in its revised governing procedures and guidelines.

With respect to EVM, NASA headquarters officials described several efforts 

under way to ensure agencywide implementation of true EVM. For 

example, NASA recently revised its EVM policy directives to shift 

ownership of EVM responsibilities from NASA’s Chief Financial Officer to 

NASA’s Chief Engineer, to emphasize that EVM is to be considered a project 

management tool rather than a financial management tool. NASA officials 

also noted that the agency is working to inform managers of the 

performance management capabilities available to them through EVM and 

to emphasize the importance of providing adequate resources and 

management support to ensure successful EVM implementation. 

Agencywide goals for EVM implementation include promoting the effective 

use of EVM and providing needed training and education for program and 

project staff. These efforts and proposed initiatives should help resolve 

EVM utilization problems.

Finally, NASA officials told us that the agency is planning to hire additional 

cost analysts to alleviate understaffing at all of its center cost analysis 

offices. The agency envisions a total staff of about 100 cost analysts 

along with additional support contractors. NASA officials also stated that 

it is necessary to ensure centers address the problem of having cost 

analysts located in the advocacy chain of command, which could affect 

five NASA centers.

Because NASA’s initiatives have only recently been implemented or are still 

in the drafting or planning stage, we cannot determine to what degree these 

efforts will enable NASA to provide reasonable and defensible cost 

estimates of its programs and projects.
Page 24 GAO-04-642 NASA Cost Estimating

Conclusions There are numerous scientific and technical challenges inherent in the 

successful implementation of many NASA programs. Nevertheless, the 

need to choose among competing alternatives within limited budget 

resources makes it essential that the agency and the Congress clearly 

understand the costs and uncertainties of programs proposed for 

authorization and funding. Yet, NASA does not have the disciplined 

cost-estimating process needed to make informed acquisition decisions, 

nor does the agency have processes and tools for capturing, monitoring, 

and managing program costs and schedules within an implementation plan 

on a timely basis. This makes it difficult for senior NASA officials, program 

and project managers, and other key stakeholders to measure performance 

and initiate mitigation measures when needed. Taken together, the lack of 

disciplined and established cost-estimating processes and tools can cause 

program officials to restructure projects to available resources rather than 

develop realistic cost estimates and implementation plans for projects. As 

a result, programs may have to be modified to accommodate emerging 

technical, cost, and schedule realities. Ultimately, programs cost more, fail 

to meet their schedules, or deliver less than originally envisioned. To help 

minimize project costs increases and implementation delays identified in 

this report, NASA needs to instill disciplined cost-estimating processes 

into its project development and approval activities and to ensure such 

processes are integrated with its implementation of an integrated 

financial management system. Without a process that prevents programs 

from proceeding before they have sufficiently demonstrated that key 

cost-estimating criteria have been met, NASA programs will continue to 

be at risk of cost and schedule overruns.

Recommendations for 

Executive Action

Improvements to NASA’s cost-estimating processes will partly depend on 

the agency’s ability to address recommendations that we made in 

November 2003 to help ensure NASA effectively implements a modern, 

integrated financial management system.27 Notwithstanding the need to 

address those recommendations, to better position NASA to ensure its 

recent initiatives result in sound cost-estimating practices agencywide, 

we are making three recommendations with minimum suggested courses 

of action. First, we are recommending that the NASA Administrator direct 

the Program Executive Officer for IFMP, the Chief Financial Officer, and 

27 GAO-04-118, GAO-04-151, and GAO-04-43.
Page 25 GAO-04-642 NASA Cost Estimating

the Chief Engineer to develop an integrated plan for improving cost 

estimating that, at a minimum, includes specific actions for ensuring that

• guidance is established on rebaselining and that rebaselining is 

consistently applied to provide accountability among programs,

• true earned value management is used as an organizational management 

tool to bring cost to the forefront in NASA’s management decisionmaking process,

• acquisition and earned value management policies and procedures are 

enforced, and

• staff and support for cost-estimating and earned value analyses are 

effectively used.

In addition, we recommend that the NASA Administrator direct the Chief 

Financial Officer to establish a standard framework for developing lifecycle cost estimates. At a minimum the framework should require each 

program or project to

• base its cost estimates on a full life cycle for the program—including all 

direct and indirect costs for operations and maintenance and disposal as 

well as planning and procurement—and on a work breakdown structure 

that encompass both in-house and contractor efforts,

• prepare a cost analysis requirements description,

• prepare an independent government estimate at each milestone of the 

program, and

• conduct a cost risk assessment that identifies the level of uncertainty 

inherent in the estimate.

Further, we recommend that the NASA Administrator develop procedures 

that would prohibit proposed projects from proceeding through the review 

and approval process when they do not address the elements of the 

recommended cost-estimating practices.
Page 26 GAO-04-642 NASA Cost Estimating

Agency Comments and 

Our Evaluation

In written comments on a draft of this report, NASA’s Deputy Administrator 

stated that the agency concurs with the recommendations, adding that the 

recommendations validate and reinforce the importance of activities 

underway at NASA to improve cost estimating and program management.

Notwithstanding agreement with our recommendations, the Deputy 

Administrator believes NASA has made substantive changes and achieved 

significant improvements in its cost-estimating processes. For example, 

NASA’s comments on a draft of this report cite a 1992 GAO report 

(GAO/NSIAD-93-97) that found a median 77 percent increase in NASA 

program costs. According to the Deputy Administrator, this contrasts with 

a 13 percent cost growth in this present study. While there may be 

improvements in the percent of cost growth of some projects, such 

declines in cost growth are often achieved by rescoping and rebaselining 

projects to remain within available resources, as was demonstrated in a 

number of projects discussed in this report. We do not believe other 

examples cited by the Deputy Administrator, namely termination of the 

Checkout and Launch Control System and cost control measures imposed 

on the International Space Station, demonstrate that NASA has made 

substantive changes and achieved significant improvements in its costestimating processes. Rather, we believe these examples demonstrate what 

happens when projects are undertaken without a full understanding of the 

potential costs and management challenges inherent in many of the 

programs NASA proposes and then implemented without adequate 

financial management systems in place. 

With regard to our recommendation to develop guidelines for rebaselining 

and ensure effective use of earned value management, the Deputy 

Administrator cited the development of revised direction on program and 

project management and a refocus on risk and cost-risk analysis. NASA 

also now requires the establishment of cost thresholds that, if exceeded, 

will require a rebaselining review. Further, because much of NASA’s work is 

performed through grants and contracts, NASA’s revised procedures will 

emphasize how risk and technical complexity affect contractor 

performance. New earned value management and acquisition policies and 

procedures will be implemented through program management councils 

that will review and approve programs and projects regularly through each 

step of their development. Also, a new Cost Analysis Division has been 

established, and cost-estimating staff has been added to it and NASA’s 

Independent Program Assessment Office. NASA also noted the importance 

of training needed to match the new requirements. 
Page 27 GAO-04-642 NASA Cost Estimating

NASA’s Deputy Administrator also concurred with our recommendation to 

establish a standard framework for developing life-cycle cost estimates. 

According to the Deputy Administrator, NASA’s new processes and 

procedural requirements document will define the full life-cycle cost to 

include development, operations, maintenance, disposal, and all NASA inhouse direct and indirect costs to eliminate ambiguity and ensure 

consistency. NASA’s revised cost-estimating handbook will provide further 

guidance for life-cycle cost estimates. Also, project managers will be 

responsible for developing and maintaining a cost analysis requirements 

document similar to a tool DOD uses that will include the equivalent of a 

project and technical description; key performance parameters, including 

documentation of actual work breakdown structure cost elements; and 

initial and annual updates of the life-cycle cost estimates. NASA guidance 

will also require periodic independent cost estimates on major programs 

and approval by the respective program management council to enter into 

implementation after an independent estimate has been completed.

Lastly, NASA’s Deputy Administrator concurred with our recommendation 

to prohibit proposed projects from proceeding through the review and 

approval process when they do not address the elements of the 

recommended cost-estimating practices. Accordingly, NASA’s forthcoming 

procedural requirements will define the authority of the program 

management councils that will, according to NASA, enforce the 

requirements, including the required information, documentation, and 

management methods needed for proceeding through the review and 

approval process. The Deputy Administrator also noted the availability of 

recent management information system improvements that enhance 

visibility over project and program performance. In his general comments, 

the Deputy Administrator also stated that NASA had recently taken steps to 

address issues raised in the draft report and suggested a report title that 

would better reflect that progress. 

We agree that NASA has initiated number of reforms to its project 

development and implementation processes that, if properly implemented, 

would be positive steps to addressing many of the problems noted in this 

report. However, we also note that some of these problems have been 

long-standing in the projects discussed in this report and in a number of 

other projects we and NASA’s Office of Inspector General have reviewed. 

Furthermore, planned improvements in the past have fallen short of 

agencywide implementation. For example, poor or inadequate cost 

estimates and management oversight have been central to the problems 

that plagued several programs, including those intended to develop new 
Page 28 GAO-04-642 NASA Cost Estimating

space transportation and the International Space Station programs. A 

reliable financial management structure is central to the success of many 

measures noted by the Deputy Administrator in his reply. We recently 

reported and testified on the impediments that exist in achieving such a 

capability. Finally, we note that contract management has been a longstanding problem at NASA. In 1990, we identified NASA’s contract 

management function as an area at high risk. During that time, there was 

little emphasis on end results, product performance, and cost control. 

NASA found itself procuring expensive hardware that did not work 

properly. This report shows that these types of problems still exist. 

Regarding the Deputy Administrator’s suggestion that we revise the title of 

our report to reflect recent progress that NASA has made towards 

addressing issues that we raise, we believe NASA’s improvements have 

been properly reflected in our report’s title. We considered the concerns 

expressed in the Deputy Administrator’s comments, and consistent with 

our stated position that NASA’s improvements are positive steps but that its 

problems still persist, we revised the title accordingly.

Finally, until NASA’s integrated financial management system, which is 

central to providing effective management and oversight, is fully 

implemented, performance assessments relying on cost data may be 

incomplete and full costing will be only partially achieved. And until these 

problems are resolved and the measures the Deputy Administrator noted in 

commenting on a draft of this report are fully implemented and integrated 

into the way the agency does business, NASA’s contract management 

function will continue to be an area of concern. 

As agreed with your office, unless you announce its contents earlier, we 

will not distribute this report further until 30 days from its date. At that 

time, we will send copies to the NASA Administrator and interested 

congressional committees. We will make copies available to others upon 

request. In addition, the report will be available at no charge on the 

GAO Web site at http://www.gao.gov.
Page 29 GAO-04-642 NASA Cost Estimating

If you or your staff have any question concerning this report, please contact 

me at (202) 512-4841 or lia@gao.gov. Key contributors to this report are 

acknowledged in appendix VI.

Allen Li

Director

Acquisition and Sourcing Management
Page 30 GAO-04-642 NASA Cost Estimating

Appendix I

Scope and Methodology ApApenednxidexiIs

To determine cost estimates in selected NASA programs and any changes 

in those estimates, we asked NASA to provide a list of programs that were 

currently in the development phase, and programs that had completed 

development or were launched in fiscal year 2001or 2002. We also asked 

NASA to provide the initial baseline development cost estimate and current 

cost estimate for the development phase and life of the program, and the 

reasons for changes to initial development cost estimates. NASA identified 

68 programs that were currently in development or had completed 

development in fiscal years 2001 and 2002. These included planetary 

missions and Earth observatory, aeronautical technology, and space flight 

systems. From that universe, we selected at least one program (10 in total) 

from 5 of NASA’s 7 Enterprises. This involved 6 of 9 NASA centers (and the 

Jet Propulsion Laboratory) with lead responsibility for one or more of 

these programs. Our selection was generally based on programs with the 

highest current development cost estimates within an Enterprise. We 

compared the initial development cost estimates NASA provided to the 

current development cost estimates for the programs. The initial 

development estimates generally reflect the projected costs at the time a 

new program was first approved by the Congress. The current development 

and life-cycle cost estimates reflect the latest estimates provided by NASA 

as of April 2003. We also interviewed program officials to obtain additional 

information related to NASA’s revisions to initially established baseline 

development cost estimates, including the rationale for changes to the cost 

estimates.

We also analyzed the initial and current development cost estimates for 

17 additional NASA programs, later added to the scope of our review, to 

ascertain the level of cost growth or decline as those programs progressed 

through the development phase.

To assess NASA’s cost-estimating processes and methodologies, we used 

cost-estimating criteria developed by Carnegie Mellon University’s 

Software Engineering Institute (SEI) designed to assess the reliability of 

project cost and schedule estimates. SEI is a government-funded research 

organization that is widely considered an authority on software 

implementation. SEI developed checklists with these criteria to help 

evaluate software costs and schedule; however, SEI states that these 

checklists are equally applicable to hardware and systems engineering 

projects. We first analyzed NASA’s cost-estimating procedures and 

guidelines to determine if they incorporated key components of good 

cost-estimating practices advocated by SEI and other experts.
Appendix I

Scope and Methodology

Page 31 GAO-04-642 NASA Cost Estimating

Based on that analysis, we selected 14 criteria from two SEI reports1 to use 

in assessing NASA’s cost-estimating practices for the 10 programs we 

selected to review in detail. Our selection of the 14 criteria from the SEI 

reports was based, in part, on their commonality with NASA costestimating procedures and guidelines. Finally, using the cost-estimating 

documentation provided by NASA for the 10 programs, we determined the 

extent to which the programs met the 14 criteria. If a program provided 

substantiating evidence for a criterion, we determined that the program 

“fully met” the criterion. If partial evidence was provided for a criterion, we 

determined the program “partially met” the criterion. If no evidence was 

found, then we determined that the criterion was “not met.” Table 2 

describes each of the 14 criteria and the significance of each criterion.

To identify any barriers that make it difficult to improve any weaknesses in 

NASA’s cost-estimating processes, we reviewed our recent work on NASA’s 

efforts to implement a modern integrated financial management system. 

We also provided questions to NASA headquarters that asked for 

information regarding NASA’s ability to use its cost estimates as a 

management tool for its programs. We also provided questions related to 

the SEI criteria, and NASA’s responses to these questions provided further 

insight into the agency’s cost-estimating management process at the 

organizational level. In addition, we interviewed officials in NASA 

headquarters’ Office of the Chief Financial Officer and Office of the Chief 

Engineer, and the center project managers for the 10 programs and other 

appropriate personnel to obtain further perspective on this issue.

To accomplish our work, we visited NASA headquarters, Washington, D.C., 

and Goddard Space Flight Center, Maryland. We also contacted officials at 

Marshall Space Flight Center, Alabama; Jet Propulsion Laboratory, 

California; Kennedy Space Center, Florida; Glenn Research Center, Ohio; 

Johnson Space Center, Texas; and Langley Research Center, Virginia.

We conducted our work from February 2003 to March 2004 in accordance 

with generally accepted government auditing standards.

1

 Software Engineering Institute, A Manager’s Checklist for Validating Software Cost and 

Schedule Estimates, CMU/SEI-95-SR-004 (Pittsburgh, Penn.: Jan. 1995) and Software 

Engineering Institute, Checklists and Criteria for Evaluating the Cost and Schedule 

Estimating Capabilities of Software Organizations, CMU/SEI-95-SR-005 (Pittsburgh, 

Penn.: 1995).
Page 32 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail ApendxiI

This appendix provides a program by program assessment of the 10 NASA 

programs we reviewed in detail. Each assessment provides

• a brief description of the program’s mission;

• the status of the program—that is, whether it is in development, 

operational, or terminated;

• the year the program was initiated;1

• the fiscal year in which the Congress approved the program—that is, 

when full-scale design and development funds were appropriated;

• a comparison of the initial and current (as of April 2003) baseline 

development estimates; and

• an assessment of the program’s cost-estimating processes, 

methodologies, and practices to determine the extent they met 

the 14 cost-estimating criteria that we used to measure program 

performance. (Table 4 shows for each criterion the number of programs 

that met, partially met, or did not meet the criterion.)

1

 We use the date the program was initiated to refer to the beginning of the formulation 

subprocess—the phase of a NASA program that establishes an affordable project concept 

and plan to meet mission objectives or technology goals.
Appendix II

Assessments of 10 Programs Reviewed 

in Detail

Page 33 GAO-04-642 NASA Cost Estimating

Table 4: Summary of the Number of Programs That Met, Partially Met, or Did Not Meet Criterion

Source: NASA (data), SEI (criteria), GAO (analysis).

a

This criterion did not apply to 5 of the 10 programs we reviewed. For those 5 programs to which the 

criterion did apply, none provided evidence comparing the dictated schedule to the normal schedule.

Number of programs that met criterion

Criterion Met Partially met Not met

The objectives of the estimate are stated in writing. 2 5 3

The life cycle to which the estimate applies is clearly defined. 0 10 0

The task has been appropriately sized. 1 5 4

The estimated cost and schedule are consistent with demonstrated 

accomplishments on other projects. 0 7 3

A written summary of parameter values and their rationales accompany 

the estimate. 0 4 6

Assumptions have been identified and explained. 2 6 2

A structured process such as a template or format has been used to ensure that key 

factors have not been overlooked. 3 7 0

Uncertainties in parameter values have been identified and quantified. 0 3 7

If a dictated schedule has been imposed, an estimate of the normal schedule has 

been compared to the additional expenditures required to meet the dictated 

schedule. a a a

If more than one cost model or estimating approach has been used, any differences 

in results have been analyzed and explained. 0 4 6

Estimators independent of the performing organization concurred with the 

reasonableness of the parameter values and estimating methodology. 3 7 0

Estimates are current. 3 7 0

The results of the estimate have been integrated with project planning and tracking. 6 4 0

Earned value reporting has been used to manage the program. 2 8 0
Page 34 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

SPACE SCIENCE

Gravity Probe B

The mission of the Gravity Probe B (GP-B) space vehicle—launched in 

April 2004—is to test Einstein’s theory of relativity, which states that 

space and time are very slightly distorted by the presence of massive 

objects, such as Earth. Over approximately 16 months, GP-B will 

measure very precisely, the expected tiny changes in the direction of 

the spin of four gyroscopes contained in GP-B as it orbits at a 400-mile 

altitude directly over the poles. The gyroscopes, free from 

disturbance, will provide an almost perfect space-time reference 

system.

Source: Katherine Stephenson/Stanford University/Lockheed Martin/NASA.

Program Facts

• Status: Development

• Program initiation: Fiscal year 1993

• Program approved by Congress: Fiscal year 1996

• Comparison of initial and current baseline development 

estimates: $179.7 million or 33.9 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiI

Met Partially met Not met

• Estimates used as baselines for program 

tracking

• Estimate life cycle clearly defined

• Assumptions identified and explained

• Structured format used to ensure all costs are 

captured

• Dictated schedules show cost impacts of 

acceleration

• Independent estimates concur with program 

estimates

• Estimates reflect changes over time

• Earned value reporting used to manage 

program

• Estimate objectives stated in writing

• Tasks appropriately sized

• Estimated costs based on demonstrated 

programs

• Written documentation of parameter values 

and rationale

• Parameter value uncertainties identified and 

quantified

• More than one cost model or estimating 

approach used 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$530

$709
Page 35 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The GP-B program failed to provide evidence of a documented complete cost estimate, including the lack of a 

written objective or description of the program to be estimated. While the program assessed risk, there was no 

evidence of how it determined the impact the risk elements had on the cost estimates. Our other key findings 

related to the GP-B program are summarized as follows:

• The GP-B program failed to provide evidence of a 

documented complete cost estimate, including the 

lack of a written objective or description of the 

program to be estimated. While the program 

assessed risk, there was no evidence of how it 

determined the impact the risk elements had on the 

cost estimates. Our other key findings related to the 

GP-B program are summarized as follows:

• Although a partial list of assumptions was included 

in the GP-B cost-estimating documentation, it was 

not clear to us whether those assumptions pertained 

to the entire life- cycle cost estimate.

• A high-level cost breakdown was provided; however, 

descriptions for key program elements-including 

formulation, implementation, operations, launch 

vehicle, and tracking and data-were not included. 

A NASA official stated that a more detailed work 

breakdown structure existed, but this breakdown 

was not provided.

• The delays in the launch date-originally scheduled 

for 2000 and pushed out to 2003 (the actual launch 

was in April 2004)-causing cost overruns were 

documented. However, there was no explanation of 

how the schedule delays impacted the cost estimate.

• The GP-B program submitted independent reviews 

annually between 1997 and 2001 and for 2003. The 

1998 independent review stated that program cost 

was a challenge for the program. To control costs, 

the 2001 independent review recommended that the 

program office (a) constrain the workforce, 

(b) replan the schedule, (c) have an independent 

team look again at the remaining costs, and 

(d) develop a contingency plan. However, none of 

these reviews provided details on the costestimating methodology.

• NASA submitted the history of programmatic 

changes from June 1994 to June 2004, but the 

associated increases in program costs were 

not included.

• GP-B gathers earned value type data using NASA 

Form 533. However, the form did not report 

full-earned value management data. According to a 

NASA official, a modified milestone-based earned 

value management system was used because the 

prime contractor resisted the implementation of a 

full-earned value management system.
Page 36 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

SPACE SCIENCE

Mars Exploration Rovers ApendxiIV

Launched in the summer of 2003, NASA’s twin roving exploration 

robots—Spirit and Opportunity—landed on opposite sides of Mars 

in January 2004 in search of answers about the history of water on 

the red planet. Over the course of their 90-day mission, the rovers 

were expected to perform on-site geological investigations, 

searching for and characterizing a wide range of rocks and soils. 

The robotic geologists were equipped with mast-mounted cameras 

that provide 360-degree, stereoscopic, humanlike views of the 

terrain; robotic arms capable of human-like elbow and wrist 

movements; and a mechanical “fist” with a microscopic camera and 

rock hammer.

Source: NASA.

Program Facts

• Status: Operations

• Program initiation: Fiscal year 2000

• Program approved by Congress: Fiscal year 2001

• Comparison of initial and current baseline development 

estimates: $109.8 million or 16.7 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxVi

Met Partially met Not met

• Structured format used to ensure all costs are 

captured

• Independent estimates concur with program 

estimates

• Estimates are kept current by reflecting 

changes over time

• Estimates used as baselines for program 

tracking

• Estimate objectives stated in writing

• Estimate life cycle clearly defined

• Estimated costs based on demonstrated 

programs

• Assumptions identified and explained

• Dictated schedules show cost impact of 

acceleration

• Earned value reporting used to manage 

program

• Tasks appropriately sized

• Written documentation of parameter values 

and rationale

• Parameter value uncertainties identified and 

quantified

• More than one cost model or estimating 

approach used

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$657

$767
Page 37 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The MERs program failed to provide evidence of a documented complete cost estimate. However, the main reason 

for the more than $100 million cost growth was the imposed schedule requirements. As NASA stated, there was a 

“less than optimal utilization of project funds...driven by the significant loss of leverage associated with design 

heritage from the Pathfinder mission and the extremely tight schedule which did not have any resiliency to 

accommodate the design changes and flight hardware delays.” Our other key findings related to the MERs 

program are summarized as follows:

• While the MERs cost-estimating supporting 

documentation provided to us collectively described 

the program’s objectives, no one document clearly 

and concisely described the overall objectives-key to 

developing a reliable life-cycle cost estimate.

• A high-level breakout of the life-cycle phases of the 

estimate was provided, but descriptions of the 

phases were not included in the documentation.

• NASA stated that the program relied on design 

heritage from the Pathfinder program. This 

assumption ultimately led to cost and schedule 

overruns and rebaselinings.

• Some high-level assumptions about the cost 

estimate were provided. A reference to “cost 

guidelines” was made, but those guidelines were not 

provided. One document showed the cost estimate 

assumptions for the flight system component of the 

program.

• MERs was initially given a 3-year development 

schedule to meet the May to June 2003 launch 

window-determined by the relative positions of the 

Earth and Mars. To meet this date within the initial 

budget, NASA planned to leverage existing 

technology. However, the program soon discovered 

this would not be possible. With the launch date set, 

the program embarked on multiple concurrent 

development efforts to meet the schedule, leading to 

cost overruns.

• A NASA official stated that the MERs program did 

not have an integrated earned value management 

system. Instead, the program used NASA form 533, 

which does not track earned value data since there is 

no measure of the progress of work performed.
Page 38 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

SPACE SCIENCE

Space Infrared Telescope Facility ApendxiVI

The Space Infrared Telescope Facility (now called Spitzer), 

launched in August 2003, is the fourth and final mission in NASA’s 

Great Observatories Program—a program designed to see the 

universe in different kinds of light. During its planned 21⁄2-year 

mission, SIRTF aims to detect infrared heat, which is mostly 

blocked by the Earth’s atmosphere. Infrared light penetrates gas and 

dust clouds, allowing scientists to peer into hidden regions of space, 

revealing star formations, centers of galaxies, and newly forming 

planetary systems. Infrared light also provides information about 

cooler objects, such as dim stars, extrasolar planets, and giant 

molecular clouds.

Source: NASA.

Program Facts

• Status: Operations

• Program initiation: Fiscal year 1984

• Program approved by Congress: Fiscal year 1998

• Comparison of initial and current baseline development 

estimates: $139 million or 29.3 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiVI

Met Partially met Not met

• Estimates used as baselines for program 

tracking

• Estimate objectives stated in writing

• Estimate life cycle clearly defined

• Tasks appropriately sized

• Estimated costs based on demonstrated programs

• Written documentation of parameter values and rationale

• Assumptions identified and explained

• Structured format used to ensure all costs are captured

• Parameter value uncertainties identified and quantified

• More than one cost model or estimating approach used

• Independent estimates concur with program estimates

• Estimates reflect changes over time

• Earned value reporting used to manage program

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$472

$611
Page 39 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The SIRTF program failed to provide supporting documentation-such as sources of data, estimating approach by 

work breakdown structure element, and any uncertainties that may accompany the cost elements—to the 

evidence provided on various cost elements and high level explanations of the methodology used to estimate 

them. Further, the life-cycle cost estimate was underestimated because it did not account for various operations 

and maintenance costs once SIRTF was launched—despite the prelaunch cost growth SIRTF experienced due to 

technical problems such as weight growth, thermal design issues, telescope heritage problems, and aperture door 

risks. Our other key findings related to the SIRTF program are summarized as follows:

• NASA provided elements of a cost estimate in 

1997 non-advocate briefing slides, but the objectives 

of the estimates were not concisely or clearly stated. 

And while the SIRTF program plan described the 

overall program objectives, it did not include the 

level of detail needed to generate an estimate.

• The SIRTF 1996 life-cycle cost estimate did not 

include key life-cycle costs, such as field support, 

science, mission operations, flight operations, and 

storage. To identify all costs associated with SIRTF’s 

life cycle, estimates needed to include all costs that 

support SIRTF's planned 21⁄2-year mission.

• NASA submitted an example of the parametric 

model inputs, but provided no parameters for 

software. In addition, the 1996 non-advocate review 

showed a high-level estimation and validation 

approach, but did not include detailed 

documentation of the sources of data, an estimating 

approach by work breakdown structure element, or 

any uncertainties that might accompany the cost 

elements. Finally, the 1997 nonadvocate review 

showed detailed estimates for SIRTF subelements, 

but no supportive documentation.

• The 1996 and 1997 nonadvocate reviews showed that 

cost probability simulations were conducted for 

certain SIRTF elements, but we found no evidence 

that this type of analysis was done for the whole 

estimate.

• The 1996 nonadvocate review showed that the cost 

basis had been validated by independent cost 

modeling and industry estimates. In the Fall of 2001, 

independent reviewers recommended adding 3 to 

6 months and $32 million to $55 million to the 

development program. In October 2002, 

independent reviews suggested that an additional 

$73 million to $130 million would be required for 

operations activities.

• The development estimate changed four times over 

the history of the program. The primary reason for 

the increases was delays in the schedule. These 

changes were tracked in the program plan and 

task plan.

• Despite the evidence of monthly tracking, we do not 

view this as indicative of earned value analysis since 

there was no measure of the progress of work 

performed. Furthermore, there was no description 

of the variances’ drivers. NASA stated that earned 

value was applied to the SIRTF program using a 

rudimentary Excel-based system, but submitted no 

reports from this system.
Page 40 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

EARTH SCIENCE

Landsat-7 ApendxiVI

Launched in April 1999, Landsat-7 is the latest in a series of earth 

observation satellites. Since 1972, Landsat satellites have collected 

continuous data on the earth's continental surfaces for land surface 

monitoring and global change research. Landsat-7’s combination of 

synoptic coverage, high spatial resolution, spectral range, and 

radiometric calibration is unparalleled and provides digital data in 

greater quantities, more quickly, and at lower cost than at any 

previous time in Landsat’s history.

Source: NASA.

Program Facts

• Status: Operations

• Program initiation: Fiscal year 1992

• Program approved by Congress: Fiscal year 1995

• Comparison of initial and current baseline development 

estimates: $63 million or 14.1 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiIX

Met Partially met Not met

• Estimate life cycle clearly defined

• Tasks appropriately sized

• Estimated costs based on demonstrated 

programs

• Structured format used to ensure all costs are 

captured

• Parameter value uncertainties identified and 

quantified

• Dictated schedules show cost impacts of 

acceleration

• Independent estimates concur with program 

estimates

• Estimates reflect changes over time

• Estimates used as baselines for program 

tracking

• Earned value reporting used to manage 

program 

• Estimate objectives stated in writing

• Written documentation of parameter values 

and rationale

• Assumptions identified and explained

• More than one cost model or estimating 

approach used 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$446

$509
Page 41 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The Landsat-7 program lacked documentation of a complete life-cycle cost estimate. Our other key findings 

related to the Landsat-7 program are summarized as follows:

• None of the Landsat-7 cost-estimating supporting 

documentation provided to us described the cost 

estimate for the complete life cycle (1994-2005). 

While we found a high-level work breakdown of 

costs through 2004, it is unclear if those costs 

included all life-cycle costs. For example, in 

April 2003, NASA provided us a cost estimate of 

$509 million for its portion of the life-cycle cost for 

Landsat-7. However, in a major review in 1996, 

a total life-cycle cost of $848 million was presented. 

Although the funding breakout showed the 

U.S. Geological Survey, National Oceanic and 

Atmospheric Administration, and DOD as having 

funding responsibility for about $400 million of the 

$848 million, it is unclear what the true life-cycle cost 

of Landsat-7 is from the documentation presented.

• Landsat-7’s program plan is highly dependent on 

heritage from Landsat-6 for management, budget, 

and schedule. We find this approach questionable 

because Landsat 6’s failures were never fully 

understood and NASA’s last significant involvement 

was on Landsat-5, launched in 1984. Further, internal 

NASA reviewers warned in 1998 that heritage 

without original staff was not heritage and that 

20-year old designs could not be reproduced.

• Although Landsat-7’s launch date was accelerated 

to May 1998—to accommodate closing of the 

contractor’s facility at Valley Forge—no 

corresponding impact to the cost estimate was 

provided. Ultimately, a series of design and 

production problems, uncovered during system level 

testing, delayed launch and increased cost by more 

than $50 million.

• Two independent reviews—in March 1996 and in 

April 1997—found that ground system development 

commitments had not been met and program 

contingency funds were critically low. The 1997 

review characterized the integration and testing 

schedule as optimistic and the May 1998 launch date 

as unrealistic. Despite these reviews, it is unclear 

whether the assessments concurred with the 

program cost estimate since we found no summary 

of such a comparison in the data.

• NASA submitted a time-phased estimate that 

compared 1997 and 1998 cost estimates and 

provided high-level reasons for the changes and a 

summary of contingency cost changes from 1995 to 

1998. However, we found no documentation of the 

entire life-cycle cost estimate for the initial program, 

nor the changes that occurred over time.

• A financial status report for October 1998 to 

April 1999 compared planned and actual costs as 

well as obligations and expenditures and provided 

a top-level explanation of cost variances using the 

work breakdown structure cited in the document. 

While the documentation also provided causes for 

launch delays, we found no evidence of data that 

detailed the corresponding cost impact.

• According to NASA documentation, a project 

action was initiated in March 1995 to develop a 

performance measurement system. Cost and 

schedule variances were reported for August 1996, 

but due to a lack of detail, we were unable to 

determine whether this was truly earned value data. 
Page 42 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

EARTH SCIENCE

Aqua ApendxiX

Aqua, part of the Earth Observing System (EOS), is expected to 

provide a 6-year chronology of Earth and its processes. Launched 

in May 2002, the Aqua satellite collects information on evaporation 

from the oceans, water vapor in the atmosphere, clouds, 

precipitation, soil moisture, sea and land ice, and snow cover. 

Aqua also measures radiative energy fluxes; aerosols; land 

vegetation cover; dissolved organic matter and phytoplankton in the 

oceans; and air, land, and water temperatures. Measurements taken 

by on-board instruments will allow scientists to assess long-term 

climate change, identify its human and natural causes, and advance 

the development of models for long-term forecasting. Source: NASA.

Program Facts

• Status: Operations

• Program initiation: Fiscal year 1991

• Program approved by Congress: Fiscal year 1991

• Comparison of initial and current baseline development 

estimates: $53.1 million or 5.3 percent decrease

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiXI

Met Partially met Not met

• Estimate objectives stated in writing

• Estimate life cycle clearly defined

• Assumptions identified and explained

• Structured format used to ensure all costs are 

captured

• Dictated schedules show cost impacts of 

acceleration

• Independent estimates concur with program 

estimates

• Estimates reflect changes over time

• Estimates used as baselines for program 

tracking

• Earned value reporting used to manage 

program

• Tasks appropriately sized

• Estimated costs based on demonstrated 

programs

• Written documentation of parameter values 

and rationale

• Parameter value uncertainties identified and 

quantified

• More than one cost model or estimating 

approach used 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$1006

1000

$952
Page 43 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The Aqua program lacked evidence of a well-documented life-cycle cost estimate. Delays in Aqua’s launch, 

originally scheduled for December 2000, increased the program’s cost to over $49 million, contributing to cost 

overruns. Our other key findings related to the Aqua program are summarized as follows:

• The President’s budget submission for fiscal year 

2003 indicated that the $49 million increase in the 

program baseline included costs for project support; 

imaging, sound, and sensor instruments; launch 

vehicle; and contingency. However, we found no 

evidence of operations, support, and other potential 

costs. Further, the President’s fiscal year 2002 

budget assumed project support and operations 

costs only for launch plus 120 days, although the 

program’s life cycle is planned for continuance 

beyond 120 days. Aqua was built to gather data for 

six years and for full life-cycle cost estimating; the 

estimate should represent the lifetime of operations 

expected for Aqua.

• We found under the assumptions for the President’s 

2002 Budget Request that the costs represented 

project support through launch plus 120 days. 

Furthermore, we found that the budget estimate did 

not include costs for mission operations beyond the 

initial 120 days.

• A high-level work breakdown structure was 

provided, but it did not include all costs for the 

life cycle.

• The effect of a slip in the launch readiness date, 

which caused a significant delay, on the cost 

estimate was not provided.

• While NASA provided evidence of independent 

reviews of the program, the most recent one 

provided was in 2000, and the reviewers concluded 

that there would be significant budget shortfalls in 

fiscal years 2001 and 2002.

• Although evidence of changes to life-cycle costs 

estimates was provided, we found no evidence 

showing that the entire life-cycle cost estimate was 

kept current given that the estimate did not include 

costs for operations and support.

• We found evidence that the budget estimates 

were analyzed and presented to management in a 

June 11, 2002, monthly status review; however, this 

analysis was not indicative of a true earned value 

management approach.
Page 44 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

EARTH SCIENCE

Aura ApendxiXI

Scheduled for launch in June 2004, the Aura satellite is the third in a 

series of major Earth-observing satellites to study environment and 

climate change. The first and second missions, Terra and Aqua, 

were designed to study the land, oceans, and the Earth’s radiation 

budget. Aura’s mission is to study, for at least a 5-year period, the 

Earth’s ozone, air quality, and climate, focusing exclusively on the 

composition, chemistry, and dynamics of the Earth’s upper and 

lower atmospheres.

Source: NASA.

Program Facts

• Status: Development

• Program initiation: Fiscal year 1991

• Program approved by Congress: Fiscal year 1994

• Comparison of initial and current baseline development 

estimates: $2.1 million or 0.3 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiXI

Met Partially met Not met

• Estimate objectives stated in writing

• Estimates reflect changes over time

• Estimate life cycle clearly defined

• Tasks appropriately sized

• Estimated costs based on demonstrated 

programs

• Assumptions identified and explained

• Structured format used to ensure all costs are 

captured

• Independent estimators concur with program 

estimates

• Estimates used as baselines for program 

tracking

• Earned value reporting used to manage 

program

• Written documentation of parameter values 

and rationale

• Parameter value uncertainties identified and 

quantified

• More than one cost model or estimating 

approach used 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

.

$763

$765
Page 45 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The Aura program lacked evidence of a life-cycle cost estimate with a methodology, a complete work breakdown 

structure, or other supporting documentation. Without such evidence, we were unable to determine if all 

associated program costs were included and if the cost estimate were reliable. Our other key findings related to 

the Aura program are summarized as follows:

• A 2002 budget document included funding for Aura 

development plus 90 days of support after launch, 

but it did not include costs for launch vehicle 

contingencies or funding for operations.

• Evidence of mass and wattage allocations for Aura 

power did not clearly show how these data were 

used and whether they were used to derive a cost 

estimate based on a cost/pound or cost/watt 

cost-estimating relationship.

• Expected efficiencies gained through the Aqua 

program’s experience were not clearly documented. 

Aqua’s experience was expected to reduce Aura’s 

cost, schedule, and technical risks because the 

majority of Aura’s structural drawings and 

spacecraft database are common with Aqua's; 

similar launch site and vehicle activities are also 

expected to provide additional efficiencies.

• Four independent reviews of cost estimates for 

Aura were conducted—in July 1998, October 1998, 

October 1999, and October 2000; however, we found 

no evidence of a cost-estimating methodology or 

reviewer concurrence with the estimates.

• While we also found some evidence that the 

program was using program baselines for program 

tracking—such as a financial status report that 

showed monthly trends for cumulative cost as 

well as obligation plans versus actual costs and 

obligations—the evidence was not convincing 

enough to demonstrate that baselines were 

consistently used for program tracking.

• Although NASA documented use of earned value 

management data for the Aura program in the 

third quarter of 1998, 1999, and 2000, officials 

stated that the program had not used such data 

during the past year and a half. While a March 2003 

cost performance report provided earned value 

management data for one of Aura's four ozone 

instruments, evidence that the Aura program as a 

whole was using earned value management data on 

a monthly basis was not provided in the Aura 

documentation.
Page 46 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

BIOLOGICAL AND PHYSICAL RESEARCH

Fluids and Combustion Facility ApendxiXIV

The Fluids and Combustion Facility (FCF) is designed to be a 

permanent modular facility for conducting microgravity 

experiments on the International Space Station. Through these 

experiments, scientists hope to enhance their understanding of 

gravity's role in a wide range of physical processes, including 

materials science, power, propulsion, combustion, fluid physics, 

and plasma physics. FCF is to be composed of two racks that share 

mutually necessary hardware. The fluids integration rack will be 

used to perform investigations for microscopic imaging to particle 

tracking. The combustion integration rack will be used to study the 

process of combustion in a near weightless environment with the 

aim of improving fire safety and increasing fuel efficiency. Source: NASA.

A flame in gravity (left) and in microgravity (right).

Program Facts

• Status: Development

• Program initiation: Fiscal year 1987

• Program approved by Congress: Fiscal year 2001

• Comparison of initial and current baseline development 

estimates: $4.8 million or 4 percent decrease

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiXV

Met Partially met Not met

• Structured format used to ensure all costs are 

captured

• Estimates used as baselines for program 

tracking

• Earned value reporting used to manage 

program

• Estimate objectives stated in writing

• Estimate life cycle clearly defined

• Tasks appropriately sized

• Estimated costs based on demonstrated 

programs

• Written documentation of parameter values 

and rationale

• Assumptions identified and explained

• More than one cost model or estimating 

approach used

• Independent estimates concur with program 

estimates

• Estimates reflect changes over time 

• Parameter value uncertainties identified and 

quantified

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$119

$114
Page 47 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The FCF program lacked evidence to determine the consistency and efficiency of the program’s estimating. 

Further, the life-cycle cost estimates did not provide a clear description of costs included, and the life cycle began 

in 1999—12 years after the program was initiated. Our other key findings related to the FCF program are 

summarized as follows:

• The February 2001 FCF independent cost estimate—

an estimate-to-complete in real year dollars—clearly 

defined the objectives of the estimate and the 

ground rules and assumptions of what was and what 

was not contained in the estimate. Further, it took 

into account previous development status. However, 

because detailed initial and life-cycle cost estimates 

were not provided, we could not determine if the 

objectives of the initial estimates were clearly 

defined.

• Although the 2001 FCF independent cost estimate 

clearly defined the life cycle that the estimate 

applied to, it did not cover the program from its 

initiation in 1987. Instead, development costs were 

considered sunk costs in another NASA area. In 

addition, an estimate of operations support did not 

include an analysis or boundaries to back up the 

estimate. Finally, the current total life- cycle cost for 

the FCF program was not evident in the documents 

provided.

• All cost estimates for software that were provided 

and reviewed have been appropriately sized; 

however, there were no initial estimates available 

for FCF software to review or compare against the 

preliminary design review. Further, the independent 

cost estimate did not map to the preliminary design 

review to allow comparisons.

• The government estimate for FCF’s prime 

development contract was broken down by work 

breakdown structure element and provided a basis 

of estimates for each element that showed the 

formula of how the cost was derived. However, 

supporting documentation did not show how the 

hours or cost of materials were estimated or 

explain the parameters that might have been used in 

the estimates.

• While an independent cost assessment was provided 

with a complete cost estimate stating assumptions 

and cost and schedule risks, no conclusions were 

provided. The independent assessors did not provide 

opinions on the reasonableness of the parameter 

values and estimating methodology. NASA provided 

a list of formal assessments the FCF program had 

completed, but the underlying documentation was 

not provided. Therefore, it was not clear to us 

whether or not the performing agencies of those 

assessments concurred with the parameter values 

and methodology.

• NASA provided a budget trace that showed 

the changes to the program estimates from the 

year 2000 to 2003. The budget trace provided the 

cost estimates from that period in the program 

forward. This trace explained the changes to the 

estimates at a top level from year to year. However, 

there were no details provided to show what 

specifically caused the changes in each of the 

year’s estimates. 
Page 48 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

AERONAUTICS

Hyper-X Program ApendxiXVI

The goal of NASA’s Hyper-X program is to flight validate key 

propulsion and related technologies for air-breathing hypersonic 

aircraft. The Hyper-X (X-43A) vehicle, launched in March 2004, flew 

at Mach 7—greater than the cruising speed of the SR-71, the world’s 

fastest air-breathing aircraft, which cruises slightly above Mach 3. 

The highest speed attained by NASA’s rocket-powered X-15 was 

Mach 6.7, back in 1967. NASA anticipates that the technologies 

exposed by the Hyper-X Program will increase payload capacities 

and reduce costs for future air and space vehicles.

Source: NASA.

Program Facts

• Status: Development

• Program initiation: Fiscal year 1996

• Program approved by Congress: Fiscal year 1998

• Comparison of initial and current baseline development 

estimates: $60 million or 35.9 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiXVI

Met Partially met Not met

• Estimates used as baselines for program 

tracking

• Estimate life cycle clearly defined

• Structured format used to ensure all costs are 

captured

• Independent estimates concur with program 

estimates

• Estimates are kept current by reflecting 

changes over time

• Earned value reporting used to manage 

program 

• Estimate objectives stated in writing

• Tasks appropriately sized

• Estimated costs based on demonstrated 

programs

• Written documentation of parameter values 

and rationale

• Assumptions identified and explained

• More than one cost model or estimating 

approach used

• Parameter value uncertainties identified and 

quantified 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$167 

$227
Page 49 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The Hyper-X program lacked a program plan and a detailed, activity-based work breakdown structure to manage 

schedule, funding, and staffing at multiple sites—which resulted in the program’s consistently running over its 

budget and not meeting schedule requirements despite the program’s high level of technical expertise needed to 

complete the project. Our other key findings related to the Hyper-X program are summarized as follows:

• The supporting cost-estimating documentation 

provided to us for the Hyper-X program included no 

definition of the life cycle. Information provided in 

those documents consisted of past costs with some 

projections of post-production modification and 

support costs. However, the costs and projections 

were stated at a high level and did not include an 

explanation of where they occurred in the life cycle.

• Although a partial cost estimate in a work 

breakdown structure format implied that a structure 

was established for the program by 1996, the 

structure was not explicitly provided. The 

contractor’s financial management report for 

July 2002 was also provided in a work breakdown 

structure format, but no master structure was 

included in the data to decipher the extent of the 

effort. Further, a NASA official stated that a work 

breakdown structure was used for both major 

contracts, however, it was not used to track costs.

• A nonadvocate review and a cost validation review 

stated that the Hyper-X program could not meet 

program objectives within current program funding 

and that additional funding would be required. 

However, parameter values and estimating 

methodology were not discussed in the documents.

• Updated estimates for the Hyper-X program were 

not provided in the two aforementioned reviews. 

However, information in the Hyper-X program status 

brief indicated that NASA had to use some updated 

cost information to determine that cost overruns 

were going to occur for the particular system for 

which it was proposing alternatives.

• In a monthly contractor financial management 

report containing the variance analysis, the 

earned value report, and the progress report 

(the only monthly report provided) for the Hyper-X 

contract—only actual and planned costs were 

included; no earned value data for analysis were 

provided. In addition, the total value provided in this 

report totaled $67.8 million—well below the Hyper-X 

program’s total cost. Furthermore, a NASA official 

stated that EVM was a deliverable on the contracts, 

but the contractor used NASA form 533, which does 

not provide full EVM data. Also, the data showed 

that there were no EVM specialists employed in the 

Hyper-X program office. As a result, the evidence 

provided was insufficient for us to determine 

whether the Hyper-X program was implementing 

earned value throughout the total program.
Page 50 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

SPACE FLIGHT

Checkout and Launch Control System ApendxiXVI

The Checkout and Launch Control System (CLCS) was intended to replace a central 

component in NASA’s existing launch processing system for the space shuttle. The 

original justification for CLCS was that a substantial portion of the vendors for the 

command control and monitor system no longer provided support. In addition, out-ofdate software and systems were expected to increase costs. CLCS promised to reduce 

staff, paperwork, and operations and maintenance costs by 50 percent. The program 

was canceled in September 2002 due to cost overruns, which according to NASA, were 

caused by factors such as software development delays based on poorly defined 

requirements and design, integration problems, and a lack of experienced 

development staff.

Source: NASA.

Program Facts

• Status: Canceled

• Program initiation: Fiscal year 1996

• Program approved by Congress: Fiscal year 1998

• Comparison of initial and current baseline development 

estimates: $193 million or 93.7 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiXI

Met Partially met Not met

• Assumptions identified and explained

• Independent estimators concur with program 

estimates

• Estimate objectives stated in writing

• Estimate life cycle clearly defined

• Tasks appropriately sized

• Estimated costs based on demonstrated programs

• Written documentation of parameter values and 

rationale

• Structured format used to ensure all costs are 

captured

• More than one cost model or estimating approach 

used

• Dictated schedules show cost impacts of 

acceleration

• Estimates reflect changes over time

• Estimates used as baselines for program tracking

• Earned value reporting used to manage program 

• Parameter value uncertainties identified 

and quantified 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$206

$399
Page 51 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

The CLCS program lacked complete evidence supporting the cost estimate. Our other key findings related to the 

CLCS program are summarized as follows:

• The description of the program objectives and 

overview provided in the program commitment 

agreement was not the description used to generate 

the cost estimate, and supporting documentation of 

the detailed cost estimated referred to in the 1997 

non-advocate review briefing was not provided.

• The total life cycle and WBS were not defined in the 

program’s life-cycle cost estimate.

• The 1997 nonadvocate review identified the analogy 

to be used as well as six different projects for 

parametric estimating. However, no details on the 

cost model parameters were documented.

• No evidence was provided to explain how the 

schedule slip—from June 2001 to June 2005—

impacted the cost estimate.

• Various documents discuss various estimating 

approaches, but no evidence of the differences in 

estimating approaches being analyzed was provided.

• Detailed descriptions of changes in CLCS estimates 

were not provided, although NASA stated that 

changes were tracked and estimates were updated 

accordingly.

• A briefing on CLCS software stated that progress 

was tracked against the plan for software 

development, but no documentation on such 

tracking at a total program level was provided.

• A program management review included a graphic 

displaying earned value data, and a NASA official 

stated in an interview that EVM had been used since 

2000. However, cost performance reports or other 

supporting documentation showing that EVM had 

been used were not provided. 
Page 52 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

SPACE FLIGHT

Cockpit Avionics Upgrade ApendxiX

The Cockpit Avionics Upgrade (CAU) project is redesigning the 

display formats on the liquid crystal displays of the space shuttle 

cockpit. The objective of the redesign is to enhance flight safety by 

presenting the crew with flight and vehicle critical information in a 

user-friendly format that enhances situational awareness. Because 

the new display format uses graphics and color to present complex 

information, crews are expected to have better and more rapid 

decision-making capability under off-nominal conditions than could 

be made with the legacy system, enhancing flight safety and the 

crew’s ability to meet mission objectives.

Source: NASA.

Program Facts

• Status: Development

• Program initiation: Fiscal year 2000

• Program approved by Congress: Fiscal year 2003

• Comparison of initial and current baseline development 

estimates: $12 million or 2.7 percent increase

Cost-Estimating Criteria

Sources: NASA (data), SEI (criteria), GAO (analysis). ApendxiXI

Met Partially met Not met

• Estimate objectives stated in writing

• Tasks appropriately sized

• Assumptions identified and explained

• Structured format used to ensure all costs are 

captured

• Independent estimators concur with program 

estimates

• Estimates reflect changes over time

• Estimates used as baselines for program 

tracking

• Earned value reporting used to manage 

program 

• Estimate life cycle clearly defined

• Estimated costs based on demonstrated 

programs

• Written documentation of parameter values 

and rationale

• Parameter value uncertainties identified and 

quantified

• More than one cost model or estimating 

approach used 

0 200 400 600 800

Initial

baseline

Baseline as 

of April 2003

Dollars in millions

Development Baselines

Source: GAO.

$442 

$454
Page 53 GAO-04-642 NASA Cost Estimating

Appendix II

Assessments of 10 Programs Reviewed 

in Detail

Our assessment results for the CAU program were the highest among the ten programs we reviewed. However, we 

noted some weaknesses. The contractor failed to include full life-cycle costs in its life-cycle cost estimate and 

detailed cost information for some work breakdown structure elements to support the estimate and the analogies 

to historical known programs. Further, we found no evidence that the CAU program office conducted its own cost 

estimate prior to receiving the contractor’s proposal, providing the office no objective means to assess the realism 

of the contractor's estimate before the nonadvocate reviews conducted in July 2001 and October 2002. Our other 

key findings related to the CAU program are summarized as follows:

• CAU project costs for certain life-cycle costs 

through 2008—such as design, development, and 

certification and delivery of the hardware and 

software; facility costs associated with the upgrade; 

and development of operational products associated 

with the upgrade, as well as reserves—were 

documented. However, other projects costs, such as 

installation costs, costs to sustain engineering, and 

operations and support were not. The sustaining 

engineering costs after 2008 were assumed to be 

zero because the costs would be absorbed through 

efficiencies provided by the CAU system. However, 

the assumption is unproven and could lead to 

higher costs after 2008. Further, although NASA 

requires that each estimate include specific 

information—including scope, definitions of terms, 

ground rules and assumptions, detailed description 

of the estimating methodology and the rationale 

for the approach, time-phased dollar estimates, 

pricing factors, and results of quantitative risk 

analyses—we found no evidence of this detailed 

information. Finally, the CAU program office said 

that impacts to the project due to full cost 

accounting have not been defined.

• The nonadvocate review costs were based on 

applied parametric estimating tools using CAU 

development process and technical descriptions of 

products including PRICE-H and NAFCOM for 

hardware development and SEER SEM for software 

development. Detailed assumptions for hardware 

and software development were provided, but there 

was no detail supporting the cost estimates for a 

number of elements, including ground facilities and 

integrated logistics, to facilitate re-creation of 

the estimate.

• PRICE-H and NAFCOM cost models were used to 

estimate hardware development; however, the 

documentation did not address whether there were 

any differences in model results; therefore, it is 

unclear whether one model was used for the primary 

estimate and the other was used to validate that 

estimate as a crosscheck. For software, we only 

found evidence of the SEER-SEM model.
Page 54 GAO-04-642 NASA Cost Estimating

Appendix III

Summary Descriptions of the 17 Additional 

Programs ApendxiI

In addition to the 10 programs that we reviewed in detail, we analyzed 

the initial and current development cost estimates for 17 other 

NASA programs.

Space Science 

Enterprise

Thermosphere, Ionosphere, 

Mesosphere Energetics and 

Dynamics (TIMED)

NASA’s TIMED satellite is conducting the first global study of the Earth’s 

mesosphere, lower thermosphere, and ionosphere—segments of the 

Earth’s atmosphere located between 40 and 110 miles above the planet. 

Initially, TIMED’s mission was to last 2 years, beginning with its launch in 

December 2001, but NASA extended the satellite’s orbital operations 

through 2006. TIMED’s goal is to improve our understanding of the 

influences the sun and humans have on this “gateway region” as well as the 

effects of its atmospheric variability on satellites and spacecraft reentering 

the Earth’s atmosphere.

International Gamma-Ray 

Astrophysics Laboratory 

(INTEGRAL)

INTEGRAL is a European Space Agency mission, with Russian and 

U.S. involvement. Launched in October 2002, the INTEGRAL satellite is 

equipped with two telescopes designed to register elusive gamma rays—

some of the universe’s most energetic radiation—and give insight into the 

most violent processes in our universe. Through INTEGRAL, scientists plan 

to study black holes’ interaction with their surroundings, the explosion of 

supernovae and their role in forming chemical elements, the nature of 

powerful gamma-ray bursts, and transient sources that suddenly change 

brightness. U.S. participation consists of co-investigators providing 

hardware and software components to the spectrometer and imager 

instruments, a co-investigator for the data center, a mission scientist, and a 

provision for ground tracking and data collection.

Rosetta Rosetta is a European Space Agency mission whose objectives are to study 

the origin of and the relationship between comets and interstellar material 

and to improve our knowledge of the origins of the Solar System. The 

Rosetta satellite was launched in March 2004 and, after a long cruise phase, 

is planned to rendezvous with comet Churyumov-Gerasimenko in 2014. 

Plans call for Rosetta to orbit the comet while taking scientific 
Appendix III

Summary Descriptions of the 17 Additional 

Programs

Page 55 GAO-04-642 NASA Cost Estimating

measurements and to position a probe on the comet surface to take in-situ 

measurements. U.S. involvement includes developing three remote-sensing 

instruments and a subsystem for a fourth instrument.

Mercury Surface, Space 

Environment, Geochemistry 

and Ranging 

(MESSENGER)

Currently scheduled to launch during a 15-day period that opens July 30, 

2004, the MESSENGER spacecraft is intended to collect images of Mercury. 

Through these images, NASA scientists hope to determine Mercury’s 

geological history and the nature of its surface composition, core, poles, 

exosphere and magnetosphere, and magnetic field. This information is 

expected to provide scientists with a better understanding of how Earth 

was formed, how it evolved, and how it interacts with the sun.

Solar Terrestrial Relations 

Observatory (STEREO)

Through STEREO—an international collaboration involving France, 

Germany, the United Kingdom, and the United States—NASA plans to trace 

the flow of energy and matter from the sun to Earth by studying the 

solar origin of coronal mass ejections, their evolution in the heliosphere, 

and their effects on geospace. Twin STEREO observatories, scheduled to 

be launched in November 2005, will be used to develop a threedimensional, time-dependent model of the magnetic topology, temperature, 

density, and velocity structure of the ambient solar wind. Because coronal 

mass ejections are the prime drivers of major space weather hazards, 

STEREO is expected to greatly improve our understanding of the most 

severe disturbances of the Sun-Earth system. The observatories will also 

provide a continuous data stream for the purpose of real-time space 

weather forecasts.

Stratospheric Observatory 

for Infrared Astronomy 

(SOFIA)

The SOFIA observatory—a modified Boeing 747 aircraft with a 

permanently installed telescope, which NASA plans to begin flying in 

2005—will be used to study different astronomical objects and phenomena, 

including star births and deaths; solar system formations; complex 

molecules in space; planets, comets, and asteroids in our solar system; 

nebulae and dust in galaxies; and black holes at the centers of galaxies. The 

telescope, provided through a partnership with the German Aerospace 

Center, is designed to provide routine access to nearly all of the visual, 

infrared, far-infrared, and submillimeter parts of the spectrum. As such, 

SOFIA is expected to extend the range of astrophysical observations 

significantly beyond that of previous infrared airborne observatories 

through increases in sensitivity and angular resolution. NASA plans to 
Appendix III

Summary Descriptions of the 17 Additional 

Programs

Page 56 GAO-04-642 NASA Cost Estimating

incorporate new or upgraded technologies over the aircraft’s lifetime to 

allow additional scientific exploration. Because most of the instruments 

are to be designed and built by graduate students and post-doctoral 

scientists in universities throughout the United States, SOFIA will serve as 

a training ground for the next generation of instrument builders.

Solar-B Observatory The Solar-B program’s objectives are to investigate the interaction between 

the Sun’s magnetic field and its corona and to understand the sources of 

solar variability. Solar-B is a Japanese Institute of Space and Astronautical 

Science mission, with significant U.S. involvement, and follows the Solar-A 

collaboration among Japan, the United Kingdom, and the United States. 

The observatory is designed to consist of a set of optical, extreme 

ultraviolet, and X-ray instruments, and NASA is expected to provide 

components for each. The Solar-B observatory is scheduled to be launched 

on a Japanese M-V rocket out of Kagoshima, Japan, in September 2006.

Herschel Space Observatory The European Space Agency’s Herschel Space Observatory (formerly the 

Far Infrared and Submillimetre Telescope, or FIRST) houses an infrared 

telescope that is expected to observe virtually unexplored spectrum 

wavelengths that cannot be observed from the ground. Scheduled for 

launch in February 2007, Herschel is expected to enable scientists to better 

understand galaxy formation, evolution in the early universe, and the 

nature of active galaxy power sources; star-forming regions and interstellar 

medium physics in the Milky Way and other galaxies; and the molecular 

chemistry of cometary, planetary, and satellite atmospheres in our solar 

system. NASA is providing components for two of the three instruments 

that will be flown on Herschel: the Heterodyne Instrument for Far Infrared 

and the Spectral and Photometric Imaging Receiver.

Earth Science 

Enterprise

Terra Launched in February 2000, Terra is providing measurements that, 

according to NASA, are significantly contributing to the understanding of 

the total Earth system. Specifically, Terra is collecting 200 gigabytes of data 

each day on the earth’s physical and radiative properties of clouds, air-land 
Appendix III

Summary Descriptions of the 17 Additional 

Programs

Page 57 GAO-04-642 NASA Cost Estimating

and air-sea exchanges of energy, carbon, and water as well as 

measurements of trace gases and volcanology. One of the first operational 

uses of Terra was to provide imagery to support the U.S. Forest Service’s 

efforts to combat forest fires in the western United States. Through Terra, 

fire fighters were able to identify the locations of active fires, instead of 

locations of smoke, providing them with the data needed to better control 

spreading fires. Terra data were also used by the Geography Department of 

Dartmouth College in New Hampshire to assist in flood hazard reduction 

programs.

New Millennium Program’s 

Earth Observing-1 (EO-1)

NASA’s New Millennium Program (NMP) is designed to identify, develop, 

and flight-validate key instrument and spacecraft technologies that can 

enable new or more cost-effective approaches to conducting science 

missions. EO-1—the first NMP mission, launched in November 2000—

includes three land imaging instruments that are expected to lead to a 

new generation of lighter weight, higher performance, and lower cost 

Landsat-type Earth surface imaging instruments.

Jason-1 The mission of the Jason-1 program, a cooperative effort with the French 

Space Agency, is to study the global oceans. Launched in December 2001, 

the Jason-1 satellite was expected to monitor ocean circulation and events 

such as El Nino and ocean eddies and to improve global climate forecasts 

and predictions. The Jason-1 satellite was positioned to orbit the earth in 

tandem with TOPEX/Poseidon, an earlier generation satellite launched in 

1992, to provide data to the National Oceanic and Atmospheric 

Administration.

SeaWinds The SeaWinds satellite, launched in December 2002, is providing highresolution, ocean surface wind data used for studies of ocean circulation, 

climate, and air-sea interaction to understand global climate changes and 

weather patterns better. By using long-term wind data in numerical 

weather and wave prediction models, SeaWinds is expected to improve 

weather forecasts near coastlines and storm warning and monitoring.
Appendix III

Summary Descriptions of the 17 Additional 

Programs

Page 58 GAO-04-642 NASA Cost Estimating

Cloud-Aerosol Lidar and 

Infrared Pathfinder Satellite 

Observations (Calipso)

The Calipso satellite, scheduled for launch in 2005, is being designed to 

study the effect that aerosols and clouds have on the Earth’s radiation 

balance, which ultimately controls the temperature of the Earth. Calipso is 

expected to provide scientists with data to construct three-dimensional 

structures of the atmosphere, enabling new observationally based 

assessments of the radiative effects of aerosol and clouds that will greatly 

improve our ability to predict future climate change. NASA plans to fly 

Calipso in formation with Aqua and CloudSat, a satellite being designed to 

measure the vertical structure of clouds from space and contribute to a 

better understanding of the role of clouds in the Earth’s climate system. 

The Calipso program is a cooperative effort with France.

Space Flight Enterprise

X-38 Crew Return Vehicle 

(CRV)

The X-38 Crew Return Vehicle was cancelled in April 2002, due to its single 

purpose design and the potentially high costs identified by an independent 

assessment. The purpose of the CRV project was to initiate work toward an 

independent U.S. crew return capability for the International Space Station. 

As envisioned, CRV was expected to serve as a back-up to the space shuttle 

orbiters by providing resupply to the station or change-out crew, or 

accommodating safe return for up to seven crew members who may be ill 

or injured or in the event that a catastrophic failure of the station made it 

unable to support life.

Alternate Turbopump 

Program (ATP)

ATP’s primary objectives were to significantly improve the safety and 

operating margins of the high-pressure turbopump in the space shuttle’s 

main engine and to eliminate the need to remove the turbopump for 

postflight maintenance. An alternative turbopump was successfully 

implemented in the shuttle launched in April 2002. According to NASA, 

ATP’s development contract, signed in December 1986, specifically 

addressed shortcomings of the previous turbopumps; took advantage of 

the latest technologies; and applied lessons learned. The contract called for 

the parallel development of two high-pressure turbopumps—one that 

operates on oxidization and one on fuel. However, 5 years into the 

program, technical problems prompted NASA to end parallel development 

and concentrate first on developing the oxidizer turbopump, which was 

first flown in July 1995. Although development of the fuel turbopump 
Appendix III

Summary Descriptions of the 17 Additional 

Programs

Page 59 GAO-04-642 NASA Cost Estimating

resumed in 1994, extreme high temperatures, pressures, and rotor speeds 

resulted in significant design challenges and the design certification review 

was not completed until March 2001. The full implementation of the fuel 

turbopump into flight was completed beginning with the April 2002 shuttle 

flight.

Tracking and Data Relay 

Satellite (TDRS) 

Replenishment

In December 2002, the TDRS Replenishment project achieved its goal: 

launch three geosynchronous satellites to replace the existing aging 

satellite constellation, and thereby continue to provide space network 

tracking, data, voice, and video services to NASA scientific satellites, the 

Space Shuttle program, the International Space Station, and other NASA 

customers. According to NASA, the functional and technical performance 

requirements for the replacement satellites—launched in June 2000, 

March 2002, and December 2002—are virtually identical to those of the 

previous satellites.

Advanced Health 

Management System 

(AHMS) Phase 1

AHMS is expected to provide safe shutdown of the space shuttle main 

engine during potentially catastrophic high-pressure turbopump failures 

through improved monitoring of engine vibration and anomaly response 

capabilities. According to NASA, AHMS modifications include (1) adding a 

vibration redline monitor for high pressure turbopumps, (2) doubling 

memory capacity and employing radiation tolerant memory, (3) adding an 

external communication interface for a potential phase-two health 

management computer, and (4) eliminating existing memory retention 

batteries and replacing them with nonvolatile memory. While NASA stated 

the AHMS will be available for launch in January 2005, the shuttle fleet’s 

return to flight date is planned for March or April 2005.
Page 60 GAO-04-642 NASA Cost Estimating

Appendix IV

Description of Earned Value Management ApendxiIV

Earned value management (EVM) goes beyond the two-dimensional 

approach of comparing budgeted costs to actuals. Instead, it attempts to 

compare the value of work accomplished during a given period with the 

work scheduled for that period. By using the value of completed work as a 

basis for estimating the cost and time needed to complete the program, 

earned value can alert program managers to potential problems early in 

the program.

An accurate, valid, and current performance management baseline is 

needed to perform useful analyses using EVM. In 1996, in response to 

acquisition reform initiatives, the Department of Defense (DOD) adopted 

32 criteria for evaluating the quality of management systems. In general 

terms, the 32 criteria require contractors to (1) define the contractual 

scope of work using a work breakdown structure; (2) identify 

organizational responsibility for the work; (3) integrate internal 

management subsystems; (4) schedule and budget authorized work; 

(5) measure the progress of work based on objective indicators; (6) collect 

the cost of labor and materials associated with the work performed; 

(7) analyze any variances from planned cost and schedules; (8) forecast 

costs at contract completion; and (9) control changes. The criteria have 

become the standard for EVM and have been adopted by major 

U.S. government agencies, industry, and the governments of Canada and 

Australia. The full application of EVM system criteria is appropriate for 

large cost reimbursable contracts where the government bears the cost 

risk. For such contracts, management discipline prescribed by the criteria 

is essential. In addition, data from an EVM system have been proved to 

provide objective reports of contract status, allowing numerous indices 

and performance measures to be calculated. These can then be used to 

develop accurate estimates of anticipated costs at completion, providing 

early warning of impending schedule delays and cost overruns.

Table 5 lists the 32 criteria, organized into five basic categories: 

organization, planning and budgeting, accounting considerations, analysis 

and management reports, and revisions and data maintenance.
Appendix IV

Description of Earned Value Management

Page 61 GAO-04-642 NASA Cost Estimating

Table 5: Thirty-Two Criteria for Evaluating the Quality of Management Systems

Category Criteria

Organization 1. Define the authorized work elements for the program. A work breakdown structure, tailored for effective 

internal management control, is commonly used in this process.

2. Identify the program organizational structure, including the major subcontractors responsible for 

accomplishing the authorized work, and define the organizational elements in which work will be planned 

and controlled.

3. Provide for the integration of the company’s planning, scheduling, budgeting, work authorization, and 

cost accumulation processes with each other and, as appropriate, the program work breakdown structure 

and the program organizational structure.

4. Identify the company organization or function responsible for controlling overhead (indirect costs).

5. Provide for integration of the program work breakdown structure and the program organizational 

structure in a manner that permits cost and schedule performance measurement by elements of either or 

both structures as needed.

Planning and budgeting 6. Schedule the authorized work in a manner that describes the sequence of work and identifies significant 

task interdependencies required to meet the requirements of the program.

7. Identify physical products, milestones, technical performance goals, or other indicators that will be used 

to measure progress.

8. Establish and maintain a time-phased budget baseline, at the control account level, against which 

program performance can be measured. Budget for far-term efforts may be held in higher-level accounts 

until an appropriate time for allocation at the control account level. Initial budgets established for 

performance measurement will be based on either internal management goals or the external customernegotiated target cost, including estimates for authorized but undefinitized work. On government contracts, 

if an over target baseline is used for performance measurement reporting purposes, prior notification must 

be provided to the customer.

9. Establish budgets for authorized work with identification of significant cost elements (labor and material, 

for example) as needed for internal management and for control of subcontractors.

10. To the extent it is practical to identify the authorized work in discrete work packages, establish budgets 

for this work in terms of dollars, hours, or other measurable units. Where the entire control account is not 

subdivided into work packages, identify the far term effort in larger planning packages for budget and 

scheduling purposes.

11. Provide that the sum of all work package budgets plus planning package budgets within a control 

account equals the control account budget.

12. Identify and control level of effort activity by time-phased budgets established for this purpose. Only that 

effort which is unmeasurable or for which measurement is impractical may be classified as level of effort.

13. Establish overhead budgets for each significant organizational component of the company for expenses 

that will become indirect costs. Reflect in the program budgets, at the appropriate level, the amounts in 

overhead pools that are planned to be allocated to the program as indirect costs.

14. Identify management reserves and undistributed budget.

15. Provide that the program target cost goal is reconciled with the sum of all internal program budgets and 

management reserves.

Accounting considerations 16. Record direct costs in a manner consistent with the budgets in a formal system controlled by the 

general books of account.
Appendix IV

Description of Earned Value Management

Page 62 GAO-04-642 NASA Cost Estimating

Source: Interim Defense Acquisition Guide Book, Appendix 4.

17. When a work breakdown structure is used, summarize direct costs from control accounts into the work 

breakdown structure without allocation of a single control account to two or more work breakdown structure 

elements.

18. Summarize direct costs from the control accounts into the contractor’s organizational elements without 

allocation of a single control account to two or more organizational elements.

19. Record all indirect costs that will be allocated to the contract.

20. Identify unit costs, equivalent units costs, or lot costs when needed.

21. For EVM, the material accounting system will provide (1) accurate cost accumulation and assignment 

of costs to control accounts in a manner consistent with the budgets using recognized, acceptable, costing 

techniques; (2) cost performance measurement at the point in time most suitable for the category of 

material involved, but no earlier than the time of progress payments or actual receipt of material; and (3) full 

accountability of all material purchased for the program, including the residual inventory.

Analysis and management 

reports

22. At least on a monthly basis, generate the following information at the control account and other levels 

as necessary for management control using actual cost data from, or reconcilable with, the accounting 

system: (1) Comparison of the amount of planned budget and the amount of budget earned for work 

accomplished. This comparison provides the schedule variance. (2) Comparison of the amount of the 

budget earned and the actual (applied where appropriate) direct costs for the same work. This comparison 

provides the cost variance.

23. Identify, at least monthly, the significant differences between both planned and actual schedule 

performance and planned and actual cost performance, and provide the reasons for the variances in the 

detail needed by program management.

24. Identify budgeted and applied (or actual) indirect costs at the level and frequency needed by 

management for effective control, along with the reasons for any significant variances.

25. Summarize the data elements and associated variances through the program organization and/or work 

breakdown structure to support management needs and any customer reporting specified in the contract.

26. Implement managerial actions taken as the result of earned value information.

27. Develop revised estimates of cost at completion based on performance to date, commitment values for 

material, and estimates of future conditions. Compare this information with the performance measurement 

baseline to identify variances at completion important to company management and any applicable 

customer reporting requirements, including statements of funding requirements.

Revisions and data 

maintenance

28. Incorporate authorized changes in a timely manner, recording the effects of such changes in budgets 

and schedules. In the directed effort prior to negotiation of a change, base such revisions on the amount 

estimated and budgeted to the program organizations.

29. Reconcile current budgets to prior budgets in terms of changes to the authorized work and internal 

replanning in the detail needed by management for effective control.

30. Control retroactive changes to records pertaining to work performed that would change previously 

reported amounts for actual costs, earned value, or budgets. Adjustments should be made only for 

correction of errors, routine accounting adjustments, effects of customer or management directed changes, 

or to improve the baseline integrity and accuracy of performance measurement data.

31. Prevent revisions to the program budget except for authorized changes.

32. Document changes to the performance measurement baseline.

(Continued From Previous Page)

Category Criteria
Appendix IV

Description of Earned Value Management

Page 63 GAO-04-642 NASA Cost Estimating

The standard format for tracking earned value is through a cost 

performance report (CPR). The CPR is a monthly compilation of 

cost, schedule, and technical data, which displays the performance 

measurement baseline, any cost and schedule variances from that baseline, 

the amount of management reserve used to date, the portion of the 

contract that is authorized unpriced work, and the contractor’s latest 

revised estimate to complete the program. As a result, the CPR can be used 

as an effective management tool because it provides the program manager 

with early warning of potential cost and schedule overruns.

Using data from the CPR, a program manager can assess trends in cost and 

schedule performance. This information is useful because trends tend to 

continue and can be difficult to reverse. Studies have shown that once 

programs are 15 percent complete, the performance indicators are 

indicative of the final outcome. For example, a CPR showing a negative 

trend for schedule status would indicate that the program is behind 

schedule. By analyzing the CPR, one could determine the cause of the 

schedule problem such as delayed flight tests, changes in requirements, or 

test problems because the CPR contains a section that describes the 

reasons for the negative status. A negative schedule can be a predictor of 

later cost problems because additional spending is often necessary to 

resolve problems. CPR data also provide the basis for independent 

assessments of a program’s cost and schedule status and can be used to 

project final costs at completion in addition to determining when a 

program should be completed.

Examining a program’s management reserves is another way that a 

program can use a CPR to determine potential issues early on. Management 

reserves, which are funds that may be used as needed, provide flexibility to 

cope with problems or unexpected events. EVM experts agree that 

transfers of management reserves should be tracked and reported because 

they are often problem indicators. An alarming situation arises if the CPR 

shows that the management reserves are being used at a faster pace than 

the program is progressing toward completion. For example, a problem 

would be indicated if a program has used 80 percent of its management 

reserves, but only completed 40 percent of its work. A program’s 

management reserves should contain at least 10 percent of the cost to 

complete a program so that funds will always be available to cover future 

unexpected problems that are more likely to surface as the program moves 

into the testing and evaluation phase.
Page 64 GAO-04-642 NASA Cost Estimating

Appendix V

Comments from the National Aeronautics and 

Space Administration ApendxiV
Appendix V

Comments from the National Aeronautics 

and Space Administration

Page 65 GAO-04-642 NASA Cost Estimating
Appendix V

Comments from the National Aeronautics 

and Space Administration

Page 66 GAO-04-642 NASA Cost Estimating
Appendix V

Comments from the National Aeronautics 

and Space Administration

Page 67 GAO-04-642 NASA Cost Estimating
Appendix V

Comments from the National Aeronautics 

and Space Administration

Page 68 GAO-04-642 NASA Cost Estimating
Appendix V

Comments from the National Aeronautics 

and Space Administration

Page 69 GAO-04-642 NASA Cost Estimating
Appendix V

Comments from the National Aeronautics 

and Space Administration

Page 70 GAO-04-642 NASA Cost Estimating
Page 71 GAO-04-642 NASA Cost Estimating

Appendix VI

GAO Contact and Staff Acknowledgments ApendxiVI

GAO Contact Allen Li (202) 512-4841

Acknowledgments Staff making key contributions to this report were Jerry Herley, Shirley 

Johnson, Charles Malphurs, Karen Sloan, Madhav Panwar, Karen Richey, 

Jennifer Echard, and Deborah Lott.

(120208)
GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal government 

for the American people. GAO examines the use of public funds; evaluates federal 

programs and policies; and provides analyses, recommendations, and other 

assistance to help Congress make informed oversight, policy, and funding 

decisions. GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail this 

list to you every afternoon, go to www.gao.gov and select “Subscribe to 

e-mail alerts” under the “Order GAO Products” heading.

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check 

or money order should be made out to the Superintendent of Documents. GAO 

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single 

address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00