GAO Cost Estimating

and Assessment Guide

Best Practices for Developing and Managing 

Capital Program Costs

United States Government Accountability Office

GAO Applied Research and Methods

GAO-09-3SP

March 2009

GAO-09-3SP Preface i

Preface

The U.S. Government Accountability Office is responsible for, among other things, assisting the 

Congress in its oversight of the federal government, including agencies’ stewardship of public funds. 

To use public funds effectively, the government must meet the demands of today’s changing world by 

employing effective management practices and processes, including the measurement of government 

program performance. In addition, legislators, government officials, and the public want to know whether 

government programs are achieving their goals and what their costs are. To make those evaluations, 

reliable cost information is required and federal standards have been issued for the cost accounting that 

is needed to prepare that information.1

 We developed the Cost Guide in order to establish a consistent 

methodology that is based on best practices and that can be used across the federal government for 

developing, managing, and evaluating capital program cost estimates. 

For the purposes of this guide, a cost estimate is the summation of individual cost elements, using 

established methods and valid data, to estimate the future costs of a program, based on what is known 

today.2 The management of a cost estimate involves continually updating the estimate with actual data 

as they become available, revising the estimate to reflect changes, and analyzing differences between 

estimated and actual costs—for example, using data from a reliable earned value management (EVM) 

system.3

The ability to generate reliable cost estimates is a critical function, necessary to support the Office of 

Management and Budget’s (OMB) capital programming process.4 Without this ability, agencies are at risk 

of experiencing cost overruns, missed deadlines, and performance shortfalls—all recurring problems that 

our program assessments too often reveal. Furthermore, cost increases often mean that the government 

1Federal Accounting Standards Advisory Board, Statement of Federal Financial Accounting Standards No. 4: Managerial Cost 

Accounting Standards and Concepts (Washington, D.C.: July 1995).

2 In the context of the Cost Guide, a program refers to all phases in a capital asset’s life cycle—that is, concept analysis, technology 

definition, requirements planning, acquisition, and operations and maintenance.

3EVM is a project management tool that integrates the technical scope of work with schedule and cost elements for investment 

planning and control. It compares the value of work accomplished in a given period with the value of the work expected in that 

period. Differences in expectations are measured in both cost and schedule variances. The Office of Management and Budget 

(OMB) requires agencies to use EVM in their performance-based management systems for the parts of an investment in which 

development effort is required or system improvements are under way.

4Office of Management and Budget, Preparation, Submission, and Execution of the Budget, Circular No. A-11 (Washington, 

D.C.: Executive Office of the President, June 2006); Management of Federal Information Resources, Circular No. A-130 Revised 

(Washington, D.C.: Executive Office of the President, Nov. 28, 2000); and Capital Programming Guide: Supplement to Circular 

A-11, Part 7, Preparation, Submission, and Execution of the Budget (Washington, D.C.: Executive Office of the President, June 2006). 

www.whitehouse.gov/omb/circulars/index.html.
ii Preface GAO-09-3SP

cannot fund as many programs as intended or deliver them when promised. The methodology outlined in 

this guide is a compilation of best practices that federal cost estimating organizations and industry use to 

develop and maintain reliable cost estimates throughout the life of a government acquisition program. By 

default, the guide will also serve as a guiding principle for our auditors to evaluate the economy, efficiency, 

and effectiveness of government programs. 

The U.S. Government Accountability Office, the Congressional Budget Office (CBO), and others 

have shown through budget simulations that the nation is facing a large and growing structural 

deficit in the long term, primarily because the population is aging and health care costs are rising. As 

Comptroller General David Walker noted, “Continuing on this unsustainable path will gradually erode, 

if not suddenly damage, our economy, our standard of living and ultimately our national security.”5

New budgetary demands and demographic trends will place serious budgetary pressures on federal 

discretionary spending, as well as on other federal policies and programs, in the coming years. 

As resources become scarce, competition for them will increase. It is imperative, therefore, that 

government acquisition programs deliver as promised, not only because of their value to their users but 

also because every dollar spent on one program will mean one less available dollar to fund other efforts. 

To get better results, programs will need higher levels of knowledge when they start and standardized 

monitoring metrics such as EVM so that better estimates can be made of total program costs at 

completion. 

5GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: February 2005), p. 1.
GAO-09-3SP Contents iii

Contents

Preface i

Abbreviations xv

Introduction 1

The Guide’s Case Studies 3

The Cost Guide in Relation to Established Standards 3

The Guide’s Readers 3

Acknowledgments 4

Chapter 1

The Characteristics of Credible Cost Estimates and a Reliable Process for Creating Them 5

Basic Characteristics of Credible Cost Estimates 6

A Reliable Process for Developing Credible Cost Estimates 8

Chapter 2

Why Government Programs Need Cost Estimates and the Challenges in Developing Them 15

Cost Estimating Challenges 17

Earned Value Management Challenges 23

Chapter 3

Criteria for Cost Estimating, EVM, and Data Reliability 25

Determining Data Reliability 29

Chapter 4

Cost Analysis Overview 31

Differentiating Cost Analysis and Cost Estimating 31

Main Cost Estimate Categories 31

The Overall Significance of Cost Estimates 36

Cost Estimates in Acquisition 37

The Importance of Cost Estimates in Establishing Budgets 38

Cost Estimates and Affordability 39

Evolutionary Acquisition and Cost Estimation 42
iv Contents GAO-09-3SP

Chapter 5

The Cost Estimate’s Purpose, Scope, and Schedule 47

Purpose 47

Scope 47

Schedule 48

Chapter 6

The Cost Assessment Team 51

Team Composition and Organization 52

Cost Estimating Team Best Practices 53

Certification and Training for Cost Estimating and EVM Analysis 54

Chapter 7

Technical Baseline Description Definition and Purpose 57

Process 57

Schedule 58

Contents 58

Key System Characteristics and Performance Parameters 61

Chapter 8

Work Breakdown Structure 65

Best Practice: Product-Oriented WBS 65

Common WBS Elements 68

WBS Development 71

Standardized WBS 71

WBS and Scheduling 75

WBS and EVM 75

WBS and Risk Management 75

WBS Benefits 76

Chapter 9

Ground Rules and Assumptions 79

Ground Rules 79

Assumptions 79

Global and Element-Specific Ground Rules and Assumptions 80

Assumptions, Sensitivity, and Risk Analysis 82

Chapter 10

Data 89

Data Collection 89

Types of Data 91

Sources of Data 92
GAO-09-3SP Contents v

Data Applicability 95

Validating and Analyzing the Data 96

EVM Data Reliability 98

Data Normalization 100

Recurring and Nonrecurring Costs 101

Inflation Adjustments 102

Selecting the Proper Indexes 103

Data Documentation 104

Chapter 11

Developing a Point Estimate 107

Cost Estimating Methods 107

Production Rate Effects on Learning 121

Pulling the Point Estimate Together 122

Chapter 12

Estimating Software Costs 125

Unique Components of Software Estimation 126

Estimating Software Size 127

Estimating Software Development Effort 131

Scheduling Software Development 133

Software Maintenance 134

Parametric Software Estimation 135

Commercial Off-the-Shelf Software 136

Enterprise Resource Planning Software 137

Software Costs Must also Account for Information Technology Infrastructure and Services 139

Unique Components of IT Estimation 139

Chapter 13

Sensitivity Analysis 147

Sensitivity Factors 148

Steps in Performing a Sensitivity Analysis 149

Sensitivity Analysis Benefits and Limitations 150

Chapter 14

Cost Risk and Uncertainty 153

The Difference Between Risk and Uncertainty 153

Point Estimates Alone Are Insufficient for Good Decisions 154

Budgeting to a Realistic Point Estimate 155

Developing a Credible S Curve of Potential Program Costs 159

Risk Management 175
vi Contents GAO-09-3SP

Chapter 15

Validating the Estimate 179

The Cost Estimating Community’s Best Practices for Validating Estimates 179

Chapter 16

Documenting the Estimate 191

Elements of Cost Estimate Documentation 193

Other Considerations 195

Chapter 17

Presenting the Estimate to Management 197

Chapter 18

Managing Program Costs: Planning 199

Linking Cost Estimation as the Foundation for EVM Analysis 199

The Nature and History of EVM 202

Implementing EVM 204

Federal and Industry Guidelines for Implementing EVM 211

The Thirteen Steps in the EVM Process 215

Integrated Baseline Reviews 231

Award Fees 233

Progress and Performance-Based Payments Under Fixed-Price Contracts 234

Validating the EVM System 236

Chapter 19

Managing Program Costs: Execution 243

Validating the Performance Measurement Baseline with an IBR 243

Contract Performance Reports 249

Monthly EVM Analysis 255

Project Future Performance 265

Provide Analysis to Management 269

Continue EVM until the Program Is Complete 269

Chapter 20

Managing Program Costs: Updating 273

Incorporating Authorized Changes into the Performance Measurement Baseline 273

Using EVM System Surveillance to Keep the Performance Measurement Baseline Current 276

Overtarget Baselines and Schedules 284

Update the Program Cost Estimate with Actual Costs 293

Keep Management Updated 294
GAO-09-3SP Contents vii

Appendixes

Appendix I

Auditing Agencies and Their Web Sites 297

Appendix II

Case Study Backgrounds 298

Appendix III

Experts Who Helped Develop This Guide 305

Appendix IV

The Federal Budget Process 308

Appendix V

Federal Cost Estimating and EVM Legislation, Regulations, Policies, and Guidance 310

Appendix VI

Data Collection Instrument 321

Appendix VII

Data Collection Instrument: Data Request Rationale 323

Appendix VIII

SEI Checklist 327

Appendix IX

Examples of Work Breakdown Structures 328

Appendix X

Schedule Risk Analysis 353

Appendix XI

Learning Curve Analysis 367

Appendix XII

Technology Readiness Levels 373

Appendix XIII

EVM-Related Award Fee Criteria 374

Appendix XIV

Integrated Baseline Review Case Study and Other Supplemental Tools 377

Exhibit A – Program X: Subject Contractor 382

Exhibit B – IBR Discussion Assessment Forms 383

Exhibit C – IBR Action Item Forms 385

Exhibit D – Sample IBR Discussion Questions 387

Appendix XV

Common Risks to Consider in Software Cost Estimating 393

Appendix XVI

Contacts and Acknowledgments 404

References 405

Image Sources 420
viii Contents GAO-09-3SP

List of Figures

Figure 1: The Cost Estimating Process 8

Figure 2: Challenges Cost Estimators Typically Face 17

Figure 3: Life-Cycle Cost Estimate for a Space System 33

Figure 4: Cone of Uncertainty 38

Figure 5: An Affordability Assessment 40

Figure 6: Typical Capital Asset Acquisition Funding Profiles by Phase 41

Figure 7: Evolutionary and Big Bang Acquisition Compared 43

Figure 8: Incremental Development 44

Figure 9: Disciplines and Concepts in Cost Analysis 51

Figure 10: A Product-Oriented Work Breakdown Structure 66

Figure 11: A Work Breakdown Structure with Common Elements 70

Figure 12: A Contract Work Breakdown Structure 70

Figure 13: A Learning Curve 120

Figure 14: A Sensitivity Analysis That Creates a Range around a Point Estimate 150

Figure 15: Changes in Cost Estimate Uncertainty across the Acquisition Life Cycle 155

Figure 16: A Cumulative Probability Distribution, or S Curve 157

Figure 17: A Risk Cube Two-Dimensional Matrix 164

Figure 18: The Distribution of Sums from Rolling Two Dice 167

Figure 19: A Point Estimate Probability Distribution Driven by WBS Distributions 168

Figure 20: Integrating Cost Estimation, Systems Development Oversight, and Risk 

Management 199

Figure 21: Integrating EVM and Risk Management 201

Figure 22: Inputs and Outputs for Tracking Earned Value 207

Figure 23: WBS Integration of Cost, Schedule, and Technical Information 216

Figure 24: Identifying Responsibility for Managing Work at the Control Account 217

Figure 25: An Activity Network 219

Figure 26: Activity Durations as a Gantt Chart 221

Figure 27: Earned Value, Using the Percent Complete Method, Compared to 

Planned Costs 227

Figure 28: The Genesis of the Performance Measurement Baseline 228

Figure 29: The Time-Phased Cumulative Performance Measurement Baseline 229

Figure 30: A Performance-Based Payments Structured Contract 235

Figure 31: The EVM System Acceptance Process 236

Figure 32: IBR Control Account Manager Discussion Template 248

Figure 33: Monthly Program Assessment Using Earned Value 250

Figure 34: Overall Program View of EVM Data 251
GAO-09-3SP Contents ix

Figure 35: A Contract Performance Report’s Five Formats 252

Figure 36: Understanding Program Cost Growth by Plotting Budget at Completion 

Trends 261

Figure 37: Understanding Program Performance by Plotting Cost and Schedule 

Variances 262

Figure 38: Understanding Expected Cost Overruns by Plotting Estimate at 

Completion 263

Figure 39: Rolling Wave Planning 269

Figure 40: The Effect on a Contract of Implementing an Overtarget Budget 285

Figure 41: Steps Typically Associated with Implementing an Overtarget Budget 289

Figure 42: Establishing a New Baseline with a Single Point Adjustment 291

Figure 43: MasterFormatTM Work Breakdown Structure 351

Figure 44: Network Diagram of a Simple Schedule 354

Figure 45: Example Project Schedule 355

Figure 46: Estimated Durations for Remaining WBS Areas in the Schedule 356

Figure 47: Cumulative Distribution of Project Schedule, Including Risk 357

Figure 48: Identified Risks on a Spacecraft Schedule: An Example 357

Figure 49: A Risk Register for a Spacecraft Schedule 358

Figure 50: Spacecraft Schedule Results from a Monte Carlo Simulation 358

Figure 51: A Schedule Showing Critical Path through Unit 2 359

Figure 52: Results of a Monte Carlo Simulation for a Schedule Showing Critical 

Path through Unit 2 360

Figure 53: Sensitivity Index for Spacecraft Schedule 360

Figure 54: Evaluation of Correlation in Spacecraft Schedule 361

Figure 55: An Example of Probabilistic Branching Contained in the Schedule 361

Figure 56: Probability Distribution Results for Probabilistic Branching in Test Unit 362

Figure 57: A Project Schedule Highlighting Correlation Effects 363

Figure 58: Risk Results Assuming No Correlation between Activity Durations 363

Figure 59: Risk Results Assuming 90 Percent Correlation between Activity 

Durations 364

Figure 60: Schedule Analysis Results with and without Correlation 364

Figure 61: IBR Team’s Program Summary Assessment Results for Program X 379

Figure 62: Program X IBR Team’s Assessment Results by Program Area 379

Figure 63: Program X IBR Team’s Detailed Assessment Results for an Individual 

Program Area 380
x Contents GAO-09-3SP

List of Tables

Table 1: GAO’s 1972 Version of the Basic Characteristics of Credible Cost Estimates 6

Table 2: The Twelve Steps of a High-Quality Cost Estimating Process 9

Table 3: Cost Estimating and EVM Criteria for Federal Agencies: Legislation, 

Regulations, Policies, and Guidance 26

Table 4: Life-Cycle Cost Estimates, Types of Business Case Analyses, and Other 

Types of Cost Estimates 34

Table 5: Certification Standards in Business, Cost Estimating, and Financial 

Management in the Defense Acquisition Education, Training, and Career 

Development Program 54

Table 6: Typical Technical Baseline Elements 58

Table 7: General System Characteristics 61

Table 8: Common Elements in Work Breakdown Structures 68

Table 9: Cost Element Structure for a Standard DOD Automated Information 

System 72

Table 10: Basic Primary and Secondary Data Sources 92

Table 11: Three Cost Estimating Methods Compared 108

Table 12: An Example of the Analogy Cost Estimating Method 109

Table 13: An Example of the Engineering Build-Up Cost Estimating Method 111

Table 14: An Example of the Parametric Cost Estimating Method 113

Table 15: The Twelve Steps of High-Quality Cost Estimating Summarized 126

Table 16: Sizing Metrics and Commonly Associated Issues 127

Table 17: Common Software Risks That Affect Cost and Schedule 133

Table 18: Best Practices Associated with Risks in Implementing ERP 138

Table 19: Common IT Infrastructure Risks 142

Table 20: Common Labor Categories Described 143

Table 21: Potential Sources of Program Cost Estimate Uncertainty 160

Table 22: A Hardware Risk Scoring Matrix 165

Table 23: A Software Risk Scoring Matrix 166

Table 24: Eight Common Probability Distributions 168

Table 25: The Twelve Steps of High-Quality Cost Estimating, Mapped to the 

Characteristics of a High-Quality Cost Estimate 180

Table 26: Questions for Checking the Accuracy of Estimating Techniques 183

Table 27: Eight Types of Independent Cost Estimate Reviews 188

Table 28: What Cost Estimate Documentation Includes 193

Table 29: Key Benefits of Implementing EVM 206

Table 30: Ten Common Concerns about EVM 207

Table 31: ANSI Guidelines for EVM Systems 212
GAO-09-3SP Contents xi

Table 32: EVM Implementation Guides 214

Table 33: Typical Methods for Measuring Earned Value Performance 225

Table 34: Integrated Baseline Review Risk Categories 246

Table 35: Contract Performance Report Data Elements: Format 1 253

Table 36: EVM Performance Indexes 259

Table 37: Best Predictive EAC Efficiency Factors by Program Completion Status 266

Table 38: Basic Program Management Questions That EVM Data Help Answer 270

Table 39: ANSI Guidelines Related to Incorporating Changes in an EVM System 274

Table 40: Elements of an Effective Surveillance Organization 277

Table 41: Key EVM Processes across ANSI Guidelines for Surveillance 278

Table 42: Risk Factors That Warrant EVM Surveillance 278

Table 43: A Program Surveillance Selection Matrix 279

Table 44: A Color-Category Rating System for Summarizing Program Findings 283

Table 45: Overtarget Budget Funding Implications by Contract Type 286

Table 46: Common Indicators of Poor Program Performance 287

Table 47: Options for Treating Variances in Performing a Single Point Adjustment 290

Table 48: Case Studies Drawn from GAO Reports Illustrating This Guide 298

Table 49: Phases of the Budget Process 308

Table 50: The Budget Process: Major Steps and Time Periods 309

Table 51: Aircraft System Work Breakdown Structure 328

Table 52: Electronic/Automated Software System Work Breakdown Structure 329

Table 53: Ground Software Work Breakdown Structure 330

Table 54: Missile System Work Breakdown Structure 331

Table 55: Ordnance System Work Breakdown Structure 333

Table 56: Sea System Work Breakdown Structure 334

Table 57: Space System Work Breakdown Structure 335

Table 58: Surface Vehicle System Work Breakdown Structure 336

Table 59: Unmanned Air Vehicle System Work Breakdown Structure 337

Table 60: Department of Energy Project Work Breakdown Structure 339

Table 61: General Services Administration Construction Work Breakdown 

Structure 340

Table 62: Automated Information System: Enterprise Resource Planning 

Program Level Work Breakdown Structure 341

Table 63: Environmental Management Work Breakdown Structure 342

Table 64: Pharmaceutical Work Breakdown Structure 342

Table 65: Process Plant Construction Work Breakdown Structure 344

Table 66: Telecon Work Breakdown Structure 344
xii Contents GAO-09-3SP

Case Studies

Case Study 1: Basic Estimate Characteristics, from NASA, GAO-04-642 7

Case Study 2: Basic Estimate Characteristics, from Customs Service Modernization, 

GAO/AIMD-99-41 8

Case Study 3: Following Cost Estimating Steps, from NASA, GAO-04-642 12

Case Study 4: Cost Analysts’ Skills, from NASA, GAO-04-642 18

Case Study 5: Recognizing Uncertainty, from Customs Service Modernization, 

GAO/AIMD-99-41 18

Case Study 6: Using Realistic Assumptions, from Space Acquisitions, GAO-07-96 19

Case Study 7: Program Stability Issues, from Combating Nuclear Smuggling, 

GAO-06-389 20

Case Study 8: Program Stability Issues, from Defense Acquisitions, GAO-05-183 20

Case Study 9: Development Schedules, from Defense Acquisitions, GAO-06-327 21

Case Study 10: Risk Analysis, from Defense Acquisitions, GAO-05-183 21

Case Study 11: Risk Analysis, from NASA, GAO-04-642 22

Case Study 12: Applying EVM, from Cooperative Threat Reduction, GAO-06-692 23

Case Study 13: Rebaselining, from NASA, GAO-04-642 24

Case Study 14: Realistic Estimates, from Defense Acquisitions, GAO-05-183 39

Case Study 15: Importance of Realistic LCCEs, from Combating Nuclear Smuggling, 

GAO-07-133R 41

Case Study 16: Importance of Realistic LCCEs, from Space Acquisitions, GAO-07-96 42

Case Study 17: Evolutionary Acquisition and Cost Estimates, from Best Practices, 

GAO-03-645T 43

Case Study 18: Incremental Development, from Customs Service Modernization, 

GAO/AIMD-99-41 44

Case Study 19: The Estimate’s Context, from DOD Systems Modernization, 

GAO-06-215 49

Case Study 20: Defining Requirement, from United States Coast Guard, 

GAO-06-623 60

Case Study 21: Managing Requirements, from DOD Systems Modernization, 

GAO-06-215 63

Case Study 22: Product-Oriented Work Breakdown Structure, from Air Traffic 

Control, GAO-08-756 67

Table 67: Software Implementation Project Work Breakdown Structure 345

Table 68: Major Renovation Project Work Breakdown Structure 346

Table 69: Sample IT Infrastructure and Service Work Breakdown Structure 347

Table 70: CSI MasterFormatTM 2004 Structure Example: Construction Phase 352

Table 71: The Anderlohr Method for the Learning Lost Factor 370

Table 72: IBR Leadership Roles and Responsibilities 377
GAO-09-3SP Contents xiii

Case Study 23: Developing Work Breakdown Structure, from NASA, GAO-04-642 77

Case Study 24: Developing Work Breakdown Structure, from Homeland Security, 

GAO-06-296 77

Case Study 25: The Importance of Assumptions, from Space Acquisitions, 

GAO-07-96 80

Case Study 26: Testing Ground Rules for Risk, from Space Acquisitions, 

GAO-07-96 84

Case Study 27: The Industrial Base, from Defense Acquisition, GAO-05-183 85

Case Study 28: Technology Maturity, from Defense Acquisitions, GAO-05-183 86

Case Study 29: Technology Maturity, from Space Acquisitions, GAO-07-96 86

Case Study 30: Informing Management of Changed Assumptions, from 

Customs Service Modernization, GAO/AIMD-99-41 87

Case Study 31: Fitting the Estimating Approach to the Data, from 

Space Acquisitions, GAO-07-96 91

Case Study 32: Data Anomalies, from Cooperative Threat Reduction, GAO-06-692 99

Case Study 33: Inflation, from Defense Acquisitions, GAO-05-183 102

Case Study 34: Cost Estimating Methods, from Space Acquisitions, GAO-07-96 110

Case Study 35: Expert Opinion, from Customs Service Modernization, 

GAO/AIMD-99-41 118

Case Study 36: Production Rate, from Defense Acquisitions, GAO-05-183 122

Case Study 37: Underestimating Software, from Space Acquisitions, GAO-07-96 125

Case Study 38: Sensitivity Analysis, from Defense Acquisitions, GAO-05-183 149

Case Study 39: Point Estimates, from Space Acquisitions, GAO-07-96 156

Case Study 40: Point Estimates, from Defense Acquisitions, GAO-05-183 156

Case Study 41: Validating the Estimate, from Chemical Demilitarization, 

GAO-07-240R 181

Case Study 42: Independent Cost Estimates, from Space Acquisitions, GAO-07-96 187

Case Study 43: Documenting the Estimate, from Telecommunications, 

GAO-07-268 192

Case Study 44: Validating the EVM System, from Cooperative Threat Reduction, 

GAO-06-692 238

Case Study 45: Validating the EVM System, from DOD Systems Modernization, 

GAO-06-215 239

Case Study 46: Cost Performance Reports, from Defense Acquisitions, GAO-05-183 255

Case Study 47: Maintaining Performance Measurement Baseline Data, from 

National Airspace System, GAO-03-343 275

Case Study 48: Maintaining Realistic Baselines, from Uncertainties Remain, 

GAO-04-643R 285
xiv Contents GAO-09-3SP

This is a work of the U.S. government and is not subject to copyright protection in the United States. 

The published product may be reproduced and distributed in its entirety without further permission 

from GAO. However, because this work may contain copyrighted images or other material, permission 

from the copyright holder may be necessary if you wish to reproduce this material separately.

Best Practices Checklists

1. Best Practices Checklist: The Estimate 45

2. Best Practices Checklist: Purpose, Scope, and Schedule 49

3. Best Practices Checklist: Cost Assessment Team 56

4. Best Practices Checklist: Technical Baseline Description 64

5. Best Practices Checklist: Work Breakdown Structure 78

6. Best Practices Checklist: Ground Rules and Assumptions 88

7. Best Practices Checklist: Data 105

8. Best Practices Checklist: Developing a Point Estimate 123

9. Best Practices Checklist: Estimating Software Costs 145

10. Best Practices Checklist: Sensitivity Analysis 151

11. Best Practices Checklist: Cost Risk and Uncertainty 176

12. Best Practices Checklist: Validating the Estimate 189

13. Best Practices Checklist: Documenting the Estimate 196

14. Best Practices Checklist: Presenting the Estimate to Management 198

15. Best Practices Checklist: Managing Program Costs: Planning 240

16. Best Practices Checklist: Managing Program Costs: Execution 270

17. Best Practices Checklist: Managing Program Costs: Updating 295
GAO-09-3SP Abbreviations xv

Abbreviations

ACWP actual cost of work performed

ANSI American National Standards Institute

AOA analysis of alternatives

BAC budget at completion

BCA business case analysis 

BCWP budgeted cost for work performed 

BCWS budgeted cost for work scheduled 

CAIG Cost Analysis Improvement Group

CBO Congressional Budget Office

CEA cost-effectiveness analysis 

CER cost estimating relationship 

COSMIC Common Software Measurement 

International Consortium

CPI cost performance index 

CPR contract performance report 

C/SCSC Cost/Schedule and Control System 

Criteria

CSDR cost and software data report

DAU Defense Acquisition University 

DCAA Defense Contract Audit Agency 

DCMA Defense Contract Management 

Agency 

DOD Department of Defense 

EA economic analysis 

EAC estimate at completion 

EIA Electronic Industries Alliance 

ERP enterprise resource planning 

EVM earned value management 

FAR Federal Acquisition Regulation 

GR&A ground rules and assumptions 

IBR integrated baseline review 

ICA independent cost assessment 

ICE independent cost estimate

IGCE independent government cost estimate 

IMS integrated master schedule 

IT information technology

LCCE life-cycle cost estimate 

NAR nonadvocate review 

NASA National Aeronautics and Space 

Administration 

NDIA National Defense Industrial 

Association 

OMB Office of Management and Budget

OTB overtarget baseline

OTS overtarget schedule

PMB performance measurement baseline 

PMI Project Management Institute

SCEA Society of Cost Estimating and 

Analysis

SEI Software Engineering Institute

SLOC source line of code

SPI schedule performance index

TCPI to complete performance index

WBS work breakdown structure 
xvi GAO-09-3SP
GAO-09-3SP Introduction 1

Introduction

Because federal guidelines are limited on processes, procedures, and practices for ensuring credible cost 

estimates, the Cost Guide is intended to fill that gap. Its purpose is twofold—to address generally accepted 

best practices for ensuring credible program cost estimates (applicable across government and industry) 

and to provide a detailed link between cost estimating and EVM. Providing that link is especially 

critical, because it demonstrates how both elements are needed for setting realistic program baselines and 

managing risk. 

As a result, government managers and auditors should find in the Cost Guide principles to guide them 

as they assess (1) the credibility of a program’s cost estimate for budget and decision making purposes 

and (2) the program’s status using EVM. Throughout this guide, we refer to program cost estimates that 

encompass major system acquisitions, as well as government in-house development efforts for which a cost 

estimate must be developed to support a budget request. 

The basic information in the Cost Guide includes the purpose, scope, and schedule of a cost estimate; a 

technical baseline description; a work breakdown structure (WBS); ground rules and assumptions; how 

to collect data; estimation methodologies; software cost estimating; sensitivity and risk analysis; validating 

a cost estimate; documenting and briefing results; updating estimates with actual costs; EVM; and the 

composition of a competent cost estimating team.6

 The guide discusses pitfalls associated with cost 

estimating and EVM that can lead government agencies to accept unrealistic budget requests—as when 

risks are embedded in an otherwise logical approach to estimating costs. Since the Department of Defense 

(DOD) is considered the leader in government cost estimating, the guide relies heavily on DOD for 

terminology and examples that may not be used by, or even apply to, other federal agencies.

Chapters 1–17 of the Cost Guide discuss the importance of cost estimating and best practices associated 

with creating credible cost estimates. They describe how cost estimates predict, analyze, and evaluate a 

program’s cost and schedule and serve as a critical program control planning tool. Once cost estimates 

have been presented to and approved by management, the chapters also establish the basis for measuring 

actual performance against the approved baseline plan using an EVM system. 

Those chapters explain how EVM, if it is to work, must have a cost estimate that identifies the effort 

that is needed—the work breakdown structure—and the period of time over which the work is to be 

performed—the program schedule.7

 In essence, the cost estimate is the basis for establishing the program’s 

6Experienced and well trained staff are crucial to developing high-quality cost estimates.

7There is at this time no standard work breakdown structure for major automated information systems; there is only a generic cost 

element structure that DOD requires for major automated information system acquisition decisions.
2 Introduction GAO-09-3SP

detailed schedule, and it identifies the bounds for how much program costs can be expected to vary, 

depending on the uncertainty analysis. When all these tasks are complete, the cost estimate can be used to 

lay the foundation for the performance measurement baseline (PMB), which will measure actual program 

performance. 

Since sound acquisition management requires more than just a reliable cost estimate at a project’s outset, 

chapters 18–20 provide guidance on converting the cost estimate into an executable program and a means 

for managing program costs. Our program assessments have too often revealed that not integrating cost 

estimation, system development oversight, and risk management—three key disciplines, interrelated and 

essential to effective acquisition management—has resulted in programs costing more than planned and 

delivering less than promised. Therefore, chapters 18–20 address best practices in implementing and 

integrating these disciplines and using them to manage costs throughout the life of a program.

OMB has set the expectation that programs will maintain current estimates of cost. This requires rigorous 

performance-based program management, which can be satisfied with EVM. Chapters 18–20 address 

the details of EVM, which is designed to integrate cost estimation, system development oversight, and 

risk management. Additionally, for programs classified as major acquisitions—regardless of whether 

the development work is completed in-house or under contract—the use of EVM is a requirement 

for development, as specified by OMB.8

 The government may also require the use of EVM for other 

acquisitions, in accordance with agency procedures. 

Since linking cost estimating and EVM results in a better view of a program and allows for greater 

understanding of program risks, cost estimators and EVM analysts who join forces can use each other’s 

data to update program costs and examine differences between estimated and actual costs. This way, 

scope changes, risks, and other opportunities can be presented to management in time to plan for and 

mitigate their impact. In addition, program status can be compared to historical data to better understand 

variances. Finally, cost estimators can help EVM analysts calculate a cumulative probability distribution to 

determine the level of confidence in the baseline. 

But bringing a program to successful completion requires knowing potential risks and identifying 

ways to respond to them before they happen—using risk management to identify, mitigate, and assign 

resources to manage risks so that their impact can be minimized. This requires the support of many 

program management and engineering staff and it results in better performance and more reliable 

predictions of program outcomes. By integrating EVM data and risk management, program managers can 

develop current estimates at completion (EAC) for all levels of management, including OMB reporting 

requirements. Therefore, chapters 18–20 expand on these concepts by examining program cost planning, 

execution, and updating. 

8Major acquisition and investment means that a system or project requires special management attention because (1) of its 

importance to the mission or function of the agency, a component of the agency, or another organization; (2) it supports financial 

management and obligates more than $500,000 annually; (3) it has significant program or policy implications; (4) it has high 

executive visibility; (5) it has high development, operating, or maintenance costs; or (6) it is defined as major by the agency’s capital 

planning and investment control process.
GAO-09-3SP Introduction 3

The Guide’s Case Studies

The Cost Guide contains a number of case studies drawn from GAO program reviews. The case studies 

highlight problems typically associated with cost estimates and augment the key points and lessons learned 

that the chapters discuss. For example, GAO has found that in many programs cost growth results from 

optimistic assumptions about technological enhancements. Experts on cost estimating have also found 

that many program managers believe they can deliver state-of-the-art technology upgrades within a 

constrained budget before proof is available that the requirements are feasible. Studies have shown that it 

costs more to develop technology from scratch than to develop it incrementally over time.9 Appendix II

gives some background information for each program used in the case studies. (Appendix I is a list of 

auditing agencies.) 

The Cost Guide in Relation to Established Standards

Our intent is to use this Cost Guide in conjunction with Government Auditing Standards and Standards 

for Internal Control in the Federal Government, commonly referred to as the yellow book and the green 

book, respectively.10 If auditors cite compliance with these standards and internal controls and find 

inconsistencies between them and the Cost Guide, they should defer to the yellow and green books for the 

prevailing rules.

This guide’s reference list identifies cost estimating guides and sources available from other government 

agencies and organizations that we relied on to determine the processes, practices, and procedures most 

commonly recommended in the cost estimating community. Users of the guide may wish to refer 

to those references for more information. In addition, we relied on information from the Society of 

Cost Estimating and Analysis (SCEA), which provides standards for cost estimating, and the Project 

Management Institute (PMI), which provides EVM standards.11

The Guide’s Readers

The federal audit community is the primary audience for this guide. In addition, agencies that do not have 

a formal policy for conducting or reviewing cost estimates will benefit from it, because it will inform them 

of the criteria GAO uses in assessing a cost estimate’s credibility. Besides GAO, auditing agencies include 

Inspectors General and audit services such as the Naval Audit Service and the Army Audit Agency. 

Appendix I lists other auditing agencies that GAO may contact at the start of an audit. The list may help 

ease the burden on agencies as they work to meet the needs of various oversight offices and should help 

speed up delivery of data request items.

We intend to periodically update the Cost Guide. Comments and suggestions from experienced users are 

always welcome, as are recommendations from experts in the cost estimating and EVM disciplines. 

9For more information on these studies, see GAO, Best Practices: Successful Application to Weapon Acquisitions Requires Changes in 

DOD’s Environment, GAO/NSIAD-98-56 (Washington, D.C.: Feb. 24, 1998), pp. 8 and 62.

10 See Comptroller General of the United States, Government Auditing Standards: January 2007 Revision, GAO-07-162G

(Washington, D.C.: January 2007), and GAO, Standards for Internal Control in the Federal Government: Exposure Draft, GAO/

AIMD-98-21.3.1 (Washington, D.C.: December 1997).

11Further information on SCEA and PMI is at www.sceaonline.org and www.pmi.org.
4 Introduction GAO-09-3SP

Acknowledgments

The Cost Guide team thanks the many members of the cost community who helped make the guide 

a reality. After we discussed our plans for developing the guide with members of the cost community, 

several experts expressed interest in working with us. The number of experts who helped us create this 

guide grew over time, beginning with our first meeting in June 2005. Their contributions were invaluable. 

Together with these experts, GAO has developed a guide that clearly outlines its criteria for assessing 

cost estimates and EVM data during audits and that we believe will benefit all agencies in the federal 

government. We would like to thank everyone who gave their time by attending meetings, giving us 

valuable documentation, and providing comments. Those who worked with us on this guide are listed in 

appendix III. Additional acknowledgments are in appendix XVI.
GAO-09-3SP Chapter 1 5

The Characteristics of Credible Cost 

Estimates and a Reliable Process for 

Creating Them

Chapter 1

More than 30 years ago, we reported that realistic cost estimating was imperative when making wise 

decisions in acquiring new systems. In 1972, we published a report called Theory and Practice of Cost 

Estimating for Major Acquisitions, in which we stated that estimates of the cost to develop and produce 

weapon systems were frequently understated, with cost increases on the order of $15.6 billion from early 

development estimates.12 In that report, we identified factors in the cost estimating function that were 

causing this problem and offered suggestions for solving or abating the problem of unexpected cost 

growth. 

We found that uniform guidance on cost estimating practices and procedures that would be the basis 

for formulating valid, consistent, and comparable estimates was lacking within DOD. In fact, evidence 

showed that each military service issued its own guidance for creating cost estimates and that the guidance 

ranged from a detailed estimating manual to a few general statements. In addition, we reported that cost 

estimators often ignored this guidance.13

In that 1972 report, we also stated that cost estimates for specific systems were frequently revisions of 

previously developed estimates and that accurate revisions of both the original and updated cost estimates 

required documentation showing data sources, assumptions, methods, and decisions basic to the estimates. 

However, we discovered that in virtually every system we reviewed for the report, documentation 

supplying such information was inaccurate or lacking. Among the resulting difficulties were that 

■ known costs had been excluded without adequate or valid justification; 

■ historical cost data used for computing estimates were sometimes invalid, unreliable, or 

unrepresentative;

■ inflation was not always included or was not uniformly treated when it was included; and

■ understanding the proper use of the estimates was hindered because the estimates were too low.14

Another finding was that readily retrievable cost data that could serve in computing cost estimates for new 

weapon systems were generally lacking. Additionally, organized and systematic efforts were not made to 

12Comptroller General of the United States, Theory and Practice of Cost Estimating for Major Acquisitions, B-163058 (Washington, 

D.C.: July 24, 1972), p. 1.

13Comptroller General of the United States, Theory and Practice of Cost Estimating for Major Acquisitions, pp. 26–27.

14Comptroller General of the United States, Theory and Practice of Cost Estimating for Major Acquisitions, pp. 28–32.
6 Chapter 1 GAO-09-3SP

gather actual cost information to achieve comparability between data collected on various weapon systems 

or to see whether the cost data the contractors reported were accurate and consistent.15

Our conclusion was that without realism and objectivity in the cost estimating process, bias and 

overoptimism creep into estimates that advocates of weapon systems prepare, and the estimates tend to 

be too low. Therefore, staff not influenced by the military organization’s determination to field a weapon 

system, or by the contractor’s intention to develop and produce the system, should review every weapon 

system at major decision points in the acquisition.16

Basic Characteristics of Credible Cost Estimates

The basic characteristics of effective estimating have been studied and highlighted many times. Their 

summary, in table 1, is from our 1972 report, Theory and Practice of Cost Estimating for Major Acquisitions. 

These characteristics are still valid today and should be found in all sound cost analyses. 

Table 1: GAO’s 1972 Version of the Basic Characteristics of Credible Cost Estimates

Characteristic Description

Clear identification of task Estimator must be provided with the system description, ground rules and 

assumptions, and technical and performance characteristics

Estimate’s constraints and conditions must be clearly identified to ensure 

the preparation of a well-documented estimate

Broad participation in preparing 

estimates

All stakeholders should be involved in deciding mission need and 

requirements and in defining system parameters and other characteristics

Data should be independently verified for accuracy, completeness, and 

reliability

Availability of valid data Numerous sources of suitable, relevant, and available data should be used

Relevant, historical data should be used from similar systems to project 

costs of new systems; these data should be directly related to the system’s 

performance characteristics

Standardized structure for the 

estimate

A standard work breakdown structure, as detailed as possible, should be 

used, refining it as the cost estimate matures and the system becomes 

more defined

The work breakdown structure ensures that no portions of the estimate are 

omitted and makes it easier to make comparisons to similar systems and 

programs

Provision for program 

uncertainties

Uncertainties should be identified and allowance developed to cover the 

cost effect

Known costs should be included and unknown costs should be allowed for

Recognition of inflation The estimator should ensure that economic changes, such as inflation, are 

properly and realistically reflected in the life-cycle cost estimate

Recognition of excluded costs All costs associated with a system should be included; any excluded costs 

should be disclosed and given a rationale

Independent review of 

estimates

Conducting an independent review of an estimate is crucial to establishing 

confidence in the estimate; the independent reviewer should verify, modify, 

and correct an estimate to ensure realism, completeness, and consistency

15Comptroller General of the United States, Theory and Practice of Cost Estimating for Major Acquisitions, pp. 31–32.

16Comptroller General of the United States, Theory and Practice of Cost Estimating for Major Acquisitions, p. 32.
GAO-09-3SP Chapter 1 7

Characteristic Description

Revision of estimates for 

significant program changes 

Estimates should be updated to reflect changes in a system’s design 

requirements. Large changes that affect costs can significantly influence 

program decisions

Source: GAO.

In a 2006 survey to identify the characteristics of a good estimate, participants from a wide variety of 

industries—aerospace, automotive, energy—as well as consulting firms and the U.S. Navy and Marine 

Corps corroborated the continuing validity of the characteristics in table 1. 

Despite the fact that these basic characteristics have been published and known for decades, we find that 

many agencies still lack the ability to develop cost estimates that can satisfy them. Case studies 1 and 2, 

drawn from GAO reports, show the kind of cross-cutting findings we have reported in the past. 

Because of findings like those in case studies 1 and 2, the Cost Guide provides best practice processes, 

standards, and procedures for developing, implementing, and evaluating cost estimates and EVM systems 

and data. By satisfying these criteria, agencies should be able to better manage their programs and inform 

decision makers of the risks involved.

Case Study 1: Basic Estimate Characteristics, from NASA, 

GAO-04-642

GAO found that the National Aeronautics and Space Administration’s (NASA) basic cost 

estimating processes—an important tool for managing programs—lacked the discipline 

needed to ensure that program estimates were reasonable. Specifically, none of the 10 

NASA programs GAO reviewed in detail met all GAO’s cost estimating criteria, which are 

based on criteria Carnegie Mellon University’s Software Engineering Institute developed. 

Moreover, none of the 10 programs fully met certain key criteria—including clearly 

defining the program’s life cycle to establish program commitment and manage program 

costs, as required by NASA. 

In addition, only 3 programs provided a breakdown of the work to be performed. Without 

this knowledge, the programs’ estimated costs could be understated and thereby subject 

to underfunding and cost overruns, putting programs at risk of being reduced in scope 

or requiring additional funding to meet their objectives. Finally, only 2 programs had a 

process in place for measuring cost and performance to identify risks.

GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective 

Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004). 
8 Chapter 1 GAO-09-3SP

Case Study 2: Basic Estimate Characteristics, from Customs Service 

Modernization, GAO/AIMD-99-41

GAO analyzed the U.S. Customs Service approach to deriving its $1.05 billion Automated 

Commercial Environment life-cycle cost estimate with Software Engineering Institute (SEI) 

criteria. SEI had seven questions for decision makers to use in assessing the reliability of a 

project’s cost estimate and detailed criteria to help evaluate how well a project satisfies 

each question. Among the criteria were several very significant and closely intertwined 

requirements that are at the core of effective cost estimating. Specifically, embedded in 

several of the questions were requirements for using (1) formal cost models; (2) structured 

and documented processes for determining the software size and reuse inputs to the 

models; and (3) relevant, measured, and normalized historical cost data (estimated and 

actual) to calibrate the models. 

GAO found that Customs did not satisfy any of these requirements. Instead of using 

a cost model, it used an unsophisticated spreadsheet to extrapolate the cost of each 

Automated Commercial Environment increment. Its approach to determining software 

size and reuse was not documented and was not well supported or convincing. Customs 

had no historical project cost data when it developed the $1.05 billion estimate and did 

not account for relevant, measured, and normalized differences in the increments. Clearly, 

such fundamental changes can dramatically affect system costs and should have been 

addressed explicitly in Customs’ cost estimates.

GAO, Customs Service Modernization: Serious Management and Technical 

Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 

1999).

A Reliable Process for Developing Credible Cost Estimates

Certain best practices should be followed if accurate and credible cost estimates are to be developed. These 

best practices represent an overall process of established, repeatable methods that result in high-quality 

cost estimates that are comprehensive and accurate and that can be easily and clearly traced, replicated, 

and updated. Figure 1 shows the cost estimating process.

Figure 1: The Cost Estimating Process

Source: GAO.

Your audience, what you 

are estimating, and why 

you are estimating it are 

of the utmost importance

Cost assessment steps are 

iterative and can be

accomplished in varying order 

or concurrently 

The confidence in the point or range 

of the estimate is crucial to the 

decision maker

Documentation and 

presentation make or 

break a cost estimating 

decision outcome

Initiation and research Assessment Analysis Presentation

Present

estimate to

management

for approval

Document

the

estimate

Conduct a

risk and

uncertainty

analysis

Conduct

sensitivity

Analysis, presentation, and updating the estimate steps

can lead to repeating previous assessment steps

Develop the

estimating

plan

Define the

estimate’s

purpose

Determine

the

estimating

structure

Identify

ground

rules and

assumptions

Develop the point

estimate and compare 

it to an independent

cost estimate

Obtain

the

data

Define

the

program Update the

estimate to

reflect actual

costs/changes
GAO-09-3SP Chapter 1 9

We have identified 12 steps that, followed correctly, should result in reliable and valid cost estimates that 

management can use for making informed decisions. Table 2 identifies all 12 steps and links each one to 

the chapter in this guide where it is discussed.

Table 2: The Twelve Steps of a High-Quality Cost Estimating Process

Step Description Associated task Chapter

1 Define estimate’s 

purpose

■ Determine estimate’s purpose, required level of detail, and 

overall scope;

■ Determine who will receive the estimate

5

2 Develop 

estimating plan

■ Determine the cost estimating team and develop its master 

schedule;

■ Determine who will do the independent cost estimate;

■ Outline the cost estimating approach;

■ Develop the estimate timeline

5 and 6

3 Define program 

characteristics

■ In a technical baseline description document, identify 

the program’s purpose and its system and performance 

characteristics and all system configurations;

■ Any technology implications;

■ Its program acquisition schedule and acquisition strategy;

■ Its relationship to other existing systems, including predecessor 

or similar legacy systems;

■ Support (manpower, training, etc.) and security needs and risk 

items;

■ System quantities for development, test, and production;

■ Deployment and maintenance plans

7

4 Determine 

estimating 

structure

■ Define a work breakdown structure (WBS) and describe each 

element in a WBS dictionary (a major automated information 

system may have only a cost element structure);

■ Choose the best estimating method for each WBS element;

■ Identify potential cross-checks for likely cost and schedule 

drivers;

■ Develop a cost estimating checklist

8

5 Identify ground 

rules and 

assumptions

■ Clearly define what the estimate includes and excludes; 

■ Identify global and program-specific assumptions, such as 

the estimate’s base year, including time-phasing and life cycle;

■ Identify program schedule information by phase and program 

acquisition strategy;

■ Identify any schedule or budget constraints, inflation 

assumptions, and travel costs;

■ Specify equipment the government is to furnish as well as the 

use of existing facilities or new modification or development; 

■ Identify prime contractor and major subcontractors; 

■ Determine technology refresh cycles, technology assumptions, 

and new technology to be developed;

■ Define commonality with legacy systems and assumed heritage 

savings;

■ Describe effects of new ways of doing business

9
10 Chapter 1 GAO-09-3SP

Step Description Associated task Chapter

6 Obtain data ■ Create a data collection plan with emphasis on collecting current 

and relevant technical, programmatic, cost, and risk data;

■ Investigate possible data sources;

■ Collect data and normalize them for cost accounting, inflation, 

learning, and quantity adjustments;

■ Analyze the data for cost drivers, trends, and outliers and 

compare results against rules of thumb and standard factors 

derived from historical data;

■ Interview data sources and document all pertinent information, 

including an assessment of data reliability and accuracy;

■ Store data for future estimates

10

7 Develop point 

estimate and 

compare it to an 

independent cost 

estimate

■ Develop the cost model, estimating each WBS element, using 

the best methodology from the data collected,a and including all 

estimating assumptions;

■ Express costs in constant year dollars; 

■ Time-phase the results by spreading costs in the years they are 

expected to occur, based on the program schedule;

■ Sum the WBS elements to develop the overall point estimate;

■ Validate the estimate by looking for errors like double counting 

and omitted costs;

■ Compare estimate against the independent cost estimate and 

examine where and why there are differences;

■ Perform cross-checks on cost drivers to see if results are similar;

■ Update the model as more data become available or as changes 

occur and compare results against previous estimates

11, 12, 

and 15

8 Conduct sensitivity 

analysis

■ Test the sensitivity of cost elements to changes in estimating 

input values and key assumptions;

■ Identify effects on the overall estimate of changing the program 

schedule or quantities; 

■ Determine which assumptions are key cost drivers and which 

cost elements are affected most by changes

13

9 Conduct risk 

and uncertainty 

analysis

■ Determine and discuss with technical experts the level of cost, 

schedule, and technical risk associated with each WBS element; 

■ Analyze each risk for its severity and probability; 

■ Develop minimum, most likely, and maximum ranges for each 

risk element;

■ Determine type of risk distributions and reason for their use;

■ Ensure that risks are correlated;

■ Use an acceptable statistical analysis method (e.g., Monte Carlo 

simulation) to develop a confidence interval around the point 

estimate;

■ Identify the confidence level of the point estimate;

■ Identify the amount of contingency funding and add this to the 

point estimate to determine the risk-adjusted cost estimate;

■ Recommend that the project or program office develop a risk 

management plan to track and mitigate risks

14
GAO-09-3SP Chapter 1 11

Step Description Associated task Chapter

10 Document the 

estimate

■ Document all steps used to develop the estimate so that a cost 

analyst unfamiliar with the program can recreate it quickly and 

produce the same result;

■ Document the purpose of the estimate, the team that prepared 

it, and who approved the estimate and on what date;

■ Describe the program, its schedule, and the technical baseline 

used to create the estimate;

■ Present the program’s time-phased life-cycle cost;

■ Discuss all ground rules and assumptions;

■ Include auditable and traceable data sources for each cost 

element and document for all data sources how the data were 

normalized;

■ Describe in detail the estimating methodology and rationale 

used to derive each WBS element’s cost (prefer more detail over 

less);

■ Describe the results of the risk, uncertainty, and sensitivity 

analyses and whether any contingency funds were identified;

■ Document how the estimate compares to the funding profile;

■ Track how this estimate compares to any previous estimates

16

11 Present estimate to 

management for 

approval

■ Develop a briefing that presents the documented life-cycle cost 

estimate;

■ Include an explanation of the technical and programmatic 

baseline and any uncertainties; 

■ Compare the estimate to an independent cost estimate (ICE) and 

explain any differences; 

■ Compare the estimate (life-cycle cost estimate (LCCE)) or 

independent cost estimate to the budget with enough detail 

to easily defend it by showing how it is accurate, complete, and 

high in quality;

■ Focus in a logical manner on the largest cost elements and cost 

drivers;

■ Make the content clear and complete so that those who are 

unfamiliar with it can easily comprehend the competence that 

underlies the estimate results;

■ Make backup slides available for more probing questions;

■ Act on and document feedback from management; 

■ Request acceptance of the estimate

17

12 Update the 

estimate to reflect 

actual costs and 

changes 

■ Update the estimate to reflect changes in technical or program 

assumptions or keep it current as the program passes through 

new phases or milestones;

■ Replace estimates with EVM EAC and independent estimate at 

completion (EAC) from the integrated EVM system; 

■ Report progress on meeting cost and schedule estimates;

■ Perform a post mortem and document lessons learned for 

elements whose actual costs or schedules differ from the 

estimate;

■ Document all changes to the program and how they affect the 

cost estimate

16, 18, 19, 

and 20

Source: GAO, DHS, DOD, DOE, NASA, SCEA, and industry. 

aIn a data-rich environment, the estimating approach should precede the investigation of data sources; in reality, a lack of data often 

determines the approach.
12 Chapter 1 GAO-09-3SP

Each of the 12 steps is important for ensuring that high-quality cost estimates are developed and delivered 

in time to support important decisions.17 Unfortunately, we have found that some agencies do not 

incorporate all the steps and, as a result, their estimates are unreliable. For example, in 2003, we completed 

a cross-cutting review at the National Aeronautics and Space Administration (NASA) that showed that 

the lack of an overall process affected NASA’s ability to create credible cost estimates (case study 3).

Case Study 3: Following Cost Estimating Steps, from NASA, 

GAO-04-642

NASA’s lack of a quality estimating process resulted in unreliable cost estimates throughout 

each program’s life cycle. As of April 2003, the baseline development cost estimates for 27 

NASA programs varied considerably from their initial baseline estimates. More than half 

the programs’ development cost estimates increased. For some of these programs, the 

increase was as much as 94 percent. In addition, the baseline development estimates for 10 

programs that GAO reviewed in detail were rebaselined—some as many as four times. 

The Checkout and Launch Control System (CLCS) program—whose baseline had increased 

from $206 million in fiscal year 1998 to $399 million by fiscal year 2003—was ultimately 

terminated. CLCS’ cost increases resulted from poorly defined requirements and design 

and fundamental changes in the contractors’ approach to the work. GAO also found that

■ the description of the program objectives and overview in the program commitment 

agreement was not the description used to generate the cost estimate; 

■ the total life cycle and WBS were not defined in the program’s life-cycle cost estimate; 

■ the 1997 nonadvocate review identified the analogy to be used as well as six different 

projects for parametric estimating, but no details on the cost model parameters were 

documented; and

■ no evidence was given to explain how the schedule slip, from June 2001 to June 2005, 

affected the cost estimate.

GAO recommended that NASA establish a framework for developing life-cycle cost 

estimates that would require each program to base its cost estimates on a WBS that 

encompassed both in-house and contractor efforts and also to prepare a description 

of cost analysis requirements. NASA concurred with the recommendation; it intended 

to revise its processes and its procedural requirements document and cost estimating 

handbook accordingly.

GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective 

Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004). 

NASA has since developed a cost estimating handbook that reflects a “renewed appreciation within the 

Agency for the importance of cost estimating as a critical part of project formulation and execution.” It has 

also stated that “There are newly formed or regenerated cost organizations at NASA Headquarters . . . . 

The field centers cost organizations have been strengthened, reversing a discouraging trend of decline.” 

17The 12 steps outlined in table 2 are appropriate for estimating the costs of large, complex programs. We note, however, that 

planning trade-offs, initial rough-order estimations, and other less visible analyses can be accomplished in less time than with the 

process outlined in the table.
GAO-09-3SP Chapter 1 13

Finally, NASA reported in its cost handbook that “Agency management, from the Administrator and 

Comptroller on down, is visibly supportive of the cost estimating function.”18

While these are admirable improvements, even an estimate that meets all these steps may be of little use 

or may be overcome by events if it is not ready when needed. Timeliness is just as important as quality. 

In fact, the quality of a cost estimate may be hampered if the time to develop it is compressed. When 

this happens, there may not be enough time to collect historical data. Since data are the key drivers of an 

estimate’s quality, their lack increases the risk that the estimate may not be reliable. In addition, when time 

is a factor, an independent cost estimate (ICE) may not be developed, further adding to the risk that the 

estimate may be overly optimistic. This is not an issue for DOD’s major defense acquisition programs, 

because an ICE is required for certain milestones. 

Relying on a standard process that emphasizes pinning down the technical scope of the work, 

communicating the basis on which the estimate is built, identifying the quality of the data, determining 

the level of risk, and thoroughly documenting the effort should result in cost estimates that are defensible, 

consistent, and trustworthy. Furthermore, this process emphasizes the idea that a cost estimate should be a 

“living document,” meaning that it will be continually updated as actual costs begin to replace the original 

estimates. This last step links cost estimating with data that are collected by an EVM system, so that 

lessons learned can be examined for differences and their reasons. It also provides valuable information for 

strengthening the credibility of future cost estimates, allowing for continuous process improvement.

18NASA, Cost Analysis Division, 2004 NASA Cost Estimating Handbook (Washington, D.C.: 2004), p. i. www.nasa.gov/offices/pae/

organization/cost_analysis_division.html.
14 Chapter 1 GAO-09-3SP
GAO-09-3SP Chapter 2 15

Why Government Programs Need 

Cost Estimates and the Challenges in 

Developing Them

Chapter 2

Cost estimates are necessary for government acquisition programs for many reasons: to support decisions 

about funding one program over another, to develop annual budget requests, to evaluate resource 

requirements at key decision points, and to develop performance measurement baselines. Moreover, 

having a realistic estimate of projected costs makes for effective resource allocation, and it increases the 

probability of a program’s success. Government programs, as identified here, include both in-house and 

contract efforts.

For capital acquisitions, OMB’s Capital Programming Guide helps agencies use funds wisely in achieving 

their missions and serving the public. The Capital Programming Guide stresses the need for agencies to 

develop processes for making investment decisions that deliver the right amount of funds to the right 

projects. It also highlights the need for agencies to identify risks associated with acquiring capital assets 

that can lead to cost overruns, schedule delays, and assets that fail to perform as expected. 

OMB’s guide has made developing accurate life-cycle cost estimates a priority for agencies in properly 

managing their portfolios of capital assets that have an estimated life of 2 years or more. Examples of 

capital assets are land; structures such as office buildings, laboratories, dams, and power plants; equipment 

like motor vehicles, airplanes, ships, satellites, and information technology hardware; and intellectual 

property, including software. 

Developing reliable cost estimates has been difficult for agencies across the federal government. Too often, 

programs cost more than expected and deliver results that do not satisfy all requirements. According to the 

2002 President’s Management Agenda:

Everyone agrees that scarce federal resources should be allocated to programs and 

managers that deliver results. Yet in practice, this is seldom done because agencies rarely 

offer convincing accounts of the results their allocations will purchase. There is little 

reward, in budgets or in compensation, for running programs efficiently. And once 

money is allocated to a program, there is no requirement to revisit the question of whether 

the results obtained are solving problems the American people care about. 19

The need for reliable cost estimates is at the heart of two of the five governmentwide initiatives in that 

agenda: improved financial performance and budget and performance integration. These initiatives are 

19President George W. Bush, The President’s Management Agenda: Fiscal Year 2002 (Washington, D.C.: Executive Office of the 

President, OMB, 2002), p. 27.
16 Chapter 2 GAO-09-3SP

aimed at ensuring that federal financial systems produce accurate and timely information to support 

operating, budget, and policy decisions and that budgets are based on performance. With respect to these 

initiatives, President Bush called for changes to the budget process to better measure the real cost and 

performance of programs. 

In response to the 2002 President’s Management Agenda, OMB’s Capital Programming Guide requires 

agencies to have a disciplined capital programming process that sets priorities between new and existing 

assets.20 It also requires agencies to perform risk management and develop cost estimates to improve the 

accuracy of cost, schedule, and performance management. These activities should help mitigate difficult 

challenges associated with asset management and acquisition. In addition, the Capital Programming Guide

requires an agency to develop a baseline assessment for each major program it plans to acquire. As part 

of this baseline, a full accounting of life-cycle cost estimates, including all direct and indirect costs for 

planning, procurement, operations and maintenance, and disposal, is expected. 

The capital programming process, as promulgated in OMB’s Capital Programming Guide, outlines 

how agencies should use long-range planning and a disciplined budget process to effectively manage a 

portfolio of capital assets that achieves program goals with the least life-cycle costs and risks. It outlines 

three phases: (1) planning and budgeting, (2) acquisition, and (3) management in use, often referred to 

as operations and maintenance. For each phase, reliable cost estimates are necessary to establish realistic 

baselines from which to measure future progress. 

Regarding the planning and budgeting phase, the federal budget process is a cyclical event. Each year in 

January or early February, the president submits budget proposals for the year that begins October 1. They 

include data for the most recently completed year, the current year, the budget year, and at least the 4 years 

following the budget year. The budget process has four phases: 

1. executive budget formulation, 

2. congressional budget process, 

3. budget execution and control, and 

4. audit and evaluation. 

Budget cycles overlap—the formulation of one budget begins before action has been completed on the 

previous one. (Appendix IV gives an overview of the federal budget process, describing its phases and the 

major steps and time periods for each phase.) 

For the acquisition and management in use phases, reliable cost estimates are also important for program 

approval and for the continued receipt of annual funding. However, cost estimating is difficult. To 

develop a sound cost estimate, estimators must possess a variety of skills and have access to high-quality 

data. Moreover, credible cost estimates take time to develop; they cannot be rushed. Their many 

challenges increase the possibility that estimates will fall short of cost, schedule, and performance goals. If 

cost analysts recognize these challenges and plan for them early, this can help organizations mitigate these 

risks. 

20OMB first issued the Capital Programming Guide as a supplement to the 1997 version of Circular A-11, Part 3. We refer to the 

2006 version. See under Circulars at OMB’s Web site, www.whitehouse.gov/omb.
GAO-09-3SP Chapter 2 17

Cost Estimating Challenges

Developing a good cost estimate requires stable program requirements, access to detailed documentation 

and historical data, well-trained and experienced cost analysts, a risk and uncertainty analysis, the 

identification of a range of confidence levels, and adequate contingency and management reserves.21 Even 

with the best of these circumstances, cost estimating is difficult. It requires both science and judgment. 

And, since answers are seldom if ever precise, the goal is to find a “reasonable” answer. However, the 

cost estimator typically faces many challenges. These challenges often lead to bad estimates—that is, 

estimates that contain poorly defined assumptions, have no supporting documentation, are accompanied 

by no comparisons to similar programs, are characterized by inadequate data collection and inappropriate 

estimating methodologies, are sustained by irrelevant or out-of-date data, provide no basis or rationale for 

the estimate, and can show no defined process for generating the estimate. Figure 2 illustrates some of the 

challenges a cost estimator faces and some of the ways to mitigate them.

Figure 2: Challenges Cost Estimators Typically Face

Program

instability

Adequate

budget

Historical

data

available

Well trained

and experienced

analysts

Detailed

documentation

available

Well

defined

Risk analysis

conducted Stable

program

Adequate cost

reserve

Unrealistic

assumptions

Unreliable

data

Inexperienced

analyst

Overoptimism

Unreasonable

program

baselines

Data

not

normalized

Historical cost

databases not

available

Obtaining

data

Cutting

edge

technology

First-time

integration

Complex

technology

New

processes

Unrealistic

projected

savings

Diminishing

industrial

base

Source: GAO.

Some cost estimating challenges are widespread. Deriving high-quality cost estimates depends on the 

quality of, for example, historical databases. It is often not possible for the cost analyst to collect the kinds 

of data needed to develop cost estimating relationships (CER), analysis of development software cost, 

engineering build-up, and many other practices. In most cases, the better the data are, the better the 

resulting estimate will be. Since much of a cost analyst’s time is spent obtaining and normalizing data, 

experienced and well-trained cost analysts are necessary. Too often, individuals without these skills are 

thrown into performing a cost analysis to meet a pressing need (see case study 4). In addition, limited 

program resources (funds and time) often constrain broad participation in cost estimation processes and 

force the analyst (or cost team) to reduce the extent to which trade-off, sensitivity, and even uncertainty 

analyses are performed.

21For our purposes in this Cost Guide, contingency reserve represents funds held at or above the government program office for 

“unknown unknowns” that are outside a contractor’s control. In this context, contingency funding is added to an estimate to 

allow for items, conditions, or events for which the state, occurrence, or effect is uncertain and experience shows are likely to result 

in additional costs. Management reserve funds, in contrast, are for “known unknowns” that are tied to the contract’s scope and 

managed at the contractor level. Unlike contingency reserve, which is funding related, management reserve is budget related. The 

value of the contract includes these known unknowns in the budget base, and the contractor decides how much money to set aside. 

We recognize that other organizations may use the terms differently.
18 Chapter 2 GAO-09-3SP

Case Study 4: Cost Analysts’ Skills, from NASA, GAO-04-642

GAO found that NASA’s efforts to improve its cost estimating processes were undermined 

by ineffective use of its limited number of cost estimating analysts. For example, 

headquarters officials stated that as projects entered the formulation phase, they typically 

relied on program control and budget specialists—not cost analysts—to provide the 

financial services to manage projects. Yet budget specialists were generally responsible 

for obligating and spending funds—not for conducting cost analyses that underlay 

the budget or ensuring that budgets were based on reasonable cost estimates—and, 

therefore, they tended to assume that the budget was realistic.

GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective 

Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004). 

Many cost estimating challenges can be traced to overoptimism. Cost analysts typically develop their 

estimates from technical baselines that program offices provide. Since program technical baselines come 

with uncertainty, recognizing this uncertainty can help form a better understanding of where problems 

will occur in the execution phase. For example, if a program baseline states that its total source lines of 

code will be 100,000 but the eventual total is 200,000, the cost will be underestimated. Or if the baseline 

states that the new program will reuse 80,000 from a legacy system but can eventually reuse only 10,000, 

the cost will be underestimated. This is illustrated in case study 5.

Case Study 5: Recognizing Uncertainty, from Customs Service 

Modernization, GAO/AIMD-99-41

Software and systems development experts agree that early project estimates are 

imprecise by definition and that their inherent imprecision decreases during a project’s life 

cycle as more information becomes known. The experts emphasize that to be useful, each 

cost estimate should indicate its degree of uncertainty, possibly as an estimated range or 

qualified by some factor of confidence. The U.S. Customs Service did not reveal the degree 

of uncertainty of its cost estimate for the Automated Commercial Environment (ACE) 

program to managers involved in investment decisions. For example, Customs did not 

disclose that it made the estimate before fully defining ACE functionality. Instead, Customs 

presented its $1.05 billion ACE life-cycle cost estimate as an unqualified point estimate. This 

suggests an element of precision that cannot exist for such an undefined system, and it 

obscures the investment risk remaining in the project.

GAO, Customs Service Modernization: Serious Management and Technical 

Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 

1999).

Program proponents often postulate the availability of a new technology, only to discover that it is not 

ready when needed and program costs have increased. Proponents also often make assumptions about the 

complexity or difficulty of new processes, such as first-time integration efforts, which may end up to be 

unrealistic. More time and effort lead directly to greater costs, as case study 6 demonstrates.
GAO-09-3SP Chapter 2 19

Case Study 6: Using Realistic Assumptions, from Space Acquisitions, 

GAO-07-96

In five of six space system acquisition programs GAO reviewed, program officials and 

cost estimators assumed when cost estimates were developed that critical technologies 

would be mature and available. They made this assumption even though the programs 

had begun without complete understanding of how long they would run or how much 

it would cost to ensure that the technologies could work as intended. After the programs 

began, and as their development continued, the technology issues ended up being more 

complex than initially believed. 

For example, for the National Polar-orbiting Operational Satellite System (NPOESS), DOD 

and the U.S. Department of Commerce committed funds for developing and producing 

satellites before the technology was mature. Only 1 of 14 critical technologies was mature 

at program initiation, and it was found that 1 technology was less mature after the 

contractor conducted more verification testing. 

GAO found that the program was later beset by significant cost increases and schedule 

delays, partly because of technical problems such as the development of key sensors. 

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Collecting historical data and dedicating the time needed to do this continuously is another challenge 

facing cost estimators. Certain acquisition policy changes and pressured scheduling have had the 

unintended consequence of curtailing the generation of a great deal of historical data used for cost 

estimating. Outside of highly specific technology areas, it is often difficult for the cost analyst to collect 

the kinds of data needed to develop software cost estimates, valid CERs, and detailed engineering 

build-up estimates. 

In addition, limited program resources in terms of both funds and time often constrain broad 

participation in cost estimation processes and force the analyst or cost team to reduce the extent to which 

trade-off, sensitivity, and even uncertainty analyses are performed. Addressing these critical shortfalls is 

important and requires policy and cultural adjustments to fix. 

Program stability presents another serious challenge to cost analysts. A risk to the program also arises 

when the contractor knows the program’s budget. The contractor is pressured into presenting a cost 

estimate that fits the budget instead of a realistic estimate. Budget decisions drive program schedules 

and procurement quantities. If development funding is reduced, the schedule can stretch and costs can 

increase; if production funding is reduced, the number of quantities to be bought will typically decrease, 

causing unit procurement costs to increase. For example, projected savings from initiatives such as 

multiyear procurement—contracting for purchase of supplies or services for more than one program 

year—may disappear, as can be seen in case study 7.
20 Chapter 2 GAO-09-3SP

Case Study 7: Program Stability Issues, from Combating Nuclear 

Smuggling, GAO-06-389

According to officials of Customs and Border Protection (CBP) and the Pacific Northwest 

National Laboratory (PNNL), recurrent difficulties with project funding were the most 

important explanations of schedule delays. Specifically, according to Department 

of Homeland Security and PNNL officials, CBP had been chronically late in providing 

appropriated funds to PNNL, hindering its ability to meet program deployment goals. 

For example, PNNL did not receive its fiscal year 2005 funding until September 2005, 

the last month of the fiscal year. According to PNNL officials, because of this delay, some 

contracting activities in all deployment phases had had to be delayed or halted; the 

adverse effects on seaports were especially severe. For example, PNNL reported in August 

2005 that site preparation work at 13 seaports had ceased because PNNL had not received 

its fiscal year 2005 funding allocation.

GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying 

Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain, 

GAO-06-389 (Washington, D.C.: Mar. 22, 2006).

Stability issues can also arise when expected funding is cut. For example, if budget pressures cause breaks 

in production, highly specialized vendors may no longer be available or may have to restructure their prices 

to cover their risks. When this happens, unexpected schedule delays and cost increases usually result. A 

quantity change, even if it does not result in a production break, is a stability issue that can increase costs 

by affecting workload. Case study 8, from a GAO report on Navy shipbuilding, illustrates this point.

Case Study 8: Program Stability Issues, from Defense Acquisitions, 

GAO-05-183

Price increases contributed to growth in materials costs. For example, the price of array 

equipment on Virginia class submarines rose by $33 million above the original price 

estimate. In addition to inflation, a limited supplier base for highly specialized and unique 

materials made ship materials susceptible to price increases. According to the shipbuilders, 

the low rate of ship production affected the stability of the supplier base. Some businesses 

closed or merged, leading to reduced competition for their services and higher prices. In 

some cases, the Navy lost its position as a preferred customer and the shipbuilder had to 

wait longer to receive materials. With a declining number of suppliers, more ship materials 

contracts went to single and sole source vendors. Over 75 percent of the materials for 

Virginia class submarines—reduced from 14 ships to 9 over a 10-year period—were 

produced by single source vendors.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005). 

Significantly accelerating (sometimes called crashing) development schedules also present risks. In such 

cases, technology tends to be incorporated before it is ready, tests are reduced or eliminated, or logistics 

support is not in place. As case study 9 shows, the result can be a reduction in costs in the short term 

but significantly increased long-term costs as problems are discovered, technology is back-fit, or logistics 

support is developed after the system is in the field. 
GAO-09-3SP Chapter 2 21

Case Study 9: Development Schedules, from Defense Acquisitions, 

GAO-06-327

Time pressures caused the Missile Defense Agency (MDA) to stray from a knowledge-based 

acquisition strategy. Key aspects of product knowledge, such as technology maturity, are 

proven in a knowledge-based strategy before committing to more development. MDA 

followed a knowledge-based strategy without fielding elements such as the Airborne 

Laser and Kinetic Energy Interceptor. But it allowed the Ground-Based Midcourse Defense 

program to concurrently become mature in its technology, complete design activities, 

and produce and field assets before end-to-end system testing—all at the expense of 

cost, quantity, and performance goals. For example, the performance of some program 

interceptors was questionable because the program was inattentive to quality assurance. 

If the block approach continued to feature concurrent activity as a means of acceleration, 

MDA’s approach might not be affordable for the considerable amount of capability that 

was yet to be developed and fielded.

GAO, Defense Acquisitions: Missile Defense Agency Fields Initial Capability but 

Falls Short of Original Goals, GAO-06-327 (Washington, D.C.: Mar. 15, 2006).

In developing cost estimates, analysts often fail to adequately address risk, especially risks that are outside 

the estimator’s control or that were never conceived to be possible. This can result in point estimates that 

give decision makers no information about their likelihood of success or give them meaningless confidence 

intervals. A risk analysis should be part of every cost estimate, but it should be performed by experienced 

analysts who understand the process and know how to use the appropriate tools. On numerous occasions, 

GAO has encountered cost estimates with meaningless confidence intervals because the analysts did not 

understand the underlying mathematics or tools. An example is given in case study 10.

Case Study 10: Risk Analysis, from Defense Acquisitions, 

GAO-05-183

In developing cost estimates for eight case study ships, U.S. Navy cost analysts did not 

conduct uncertainty analyses to measure the probability of cost growth. Uncertainty 

analyses are particularly important, given uncertainties inherent in ship acquisition, such 

as the introduction of new technologies and the volatility of overhead rates. Despite 

the uncertainties, the Navy did not test the validity of the cost analysts’ assumptions 

in estimating construction costs for the eight case study ships, and it did not identify a 

confidence level for estimates. 

Specifically, it did not conduct uncertainty analyses, which generate values for parameters 

that are less than precisely known around a specific set of ranges. For example, if the 

number of hours to integrate a component into a ship is not precisely known, analysts may 

put in low and high values. The estimate will generate costs for these variables, along with 

other variables such as weight, experience, and degree of rework. The result will be a range 

of estimates that enables cost analysts to make better decisions on likely costs. Instead, the 

Navy presented its cost estimates as unqualified point estimates, suggesting an element 

of precision that cannot exist early in the process. Other military services qualify their cost 

estimates by determining a confidence level of 50 percent.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).
22 Chapter 2 GAO-09-3SP

A risk analysis should be used to determine a program’s contingency funding. All development programs 

should have contingency funding because it is simply unreasonable to expect a program not to encounter 

problems. Problems always occur, and program managers need ready access to funding in order to 

resolve them without adversely affecting programs (for example, stretching the schedule). Unfortunately, 

budget cuts often target contingency funding, and in some cases such funding is not allowed by policy. 

Decision makers and budget analysts should understand that eliminating contingency funding is 

counterproductive. (See case study 11.)

Case Study 11: Risk Analysis, from NASA, GAO-04-642

Only by quantifying cost risk can management make informed decisions about risk 

mitigation strategies. Quantifying cost risk also provides a benchmark for measuring future 

progress. Without this knowledge, NASA may have little specific basis for determining 

adequate financial reserves, schedule margins, and technical performance margins. 

Managers may thus not have the flexibility they need to address program, technical, cost, 

and schedule risks, as NASA policy requires.

GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective 

Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004). 

Too often, organizations encourage goals that are unattainable because there is overoptimism that their 

organizations can reach them. These decisions follow a thought process that accentuates the positive 

without truly understanding the pitfalls being faced—in other words, the decision makers are avoiding 

risk. Recognizing and understanding risk is an important program management discipline, but most 

program managers believe they are dealing with risks when in fact they have created risk by their 

assumptions. History shows that program managers tend to be too optimistic. They believe that lessons 

learned from past programs will apply to their program and everything will work out fine. But a plan is 

by its nature meant to be optimistic, to ensure that the results will be successful. While program managers 

believe they build risk into their plan, they often do not put in enough. This is because they believe in 

the original estimates for the plan without allowing for additional changes in scope, schedule delays, or 

other elements of risk. In addition, in today’s competitive environment, contractor program managers may 

overestimate what their company can do compared to their competition, since they want to win. 

Since most organizations have a limited amount of money for addressing these issues, optimism is 

prevalent. To properly overcome this optimism, it is important to have an independent view. Through 

the program planning process, overoptimism can be tempered by challenging the assumptions the plan 

was based on. This can be done by independently assessing the outcomes, by using comparative data or 

experts in accomplishing the efforts planned. While this function can be performed by either inside or 

outside analysts, if the organization is not willing to address and understand the risks its program faces, it 

will have little hope of effectively managing and mitigating them. Having this “honest broker” approach 

to working these programs helps bring to light actions that can potentially limit the organization’s ability 

to succeed. Therefore, program managers and their organizations must understand the value and need for 

risk management by addressing risk proactively and having a plan should risks be realized. Doing so will 

enable the program management team to use this information to succeed in the future.
GAO-09-3SP Chapter 2 23

Earned Value Management Challenges

OMB recommends that programs manage risk by applying EVM, among other ways. Reliable EVM 

data usually indicate monthly how well a program is performing in terms of cost, schedule, and technical 

matters. This information is necessary for proactive program management and risk mitigation. Such 

systems represent a best practice if implemented correctly, but qualified analytic staff are needed to validate 

and interpret the data. (See case study 12.)

Case Study 12: Applying EVM, from Cooperative Threat Reduction, 

GAO-06-692

In December 2005, a contractor’s self-evaluation stated that the EVM system for the 

chemical weapons destruction facility at Shchuch’ye, Russia, was fully implemented. DOD 

characterized the contractor’s EVM implementation as a “management failure,” citing a 

lack of experienced and qualified contractor staff. DOD withheld approximately $162,000 

of the contractor’s award fee because of its concern about the EVM system. In March 

2006, DOD officials stated that EVM was not yet a usable tool in managing the Shchuch’ye 

project. They stated that the contractor needed to demonstrate that it had incorporated 

EVM into project management rather than simply fulfilling contractual requirements. DOD 

expected the contractor to use EVM to estimate cost and schedule effects and their causes 

and, most importantly, to help eliminate or mitigate identified risks. The contractor’s EVM 

staff stated that they underestimated the effort needed to incorporate EVM data into 

the system, train staff, and develop EVM procedures. The contractor’s officials were also 

surprised by the number of man-hours required to accomplish these tasks, citing high staff 

turnover as contributing to the problem. According to the officials, working in a remote 

and isolated area caused many of the non-Russian employees to leave the program rather 

than extend their initial tour of duty.

GAO, Cooperative Threat Reduction: DOD Needs More Reliable Data to Better 

Estimate the Cost and Schedule of the Shchuch’ye Facility, GAO-06-692

(Washington, D.C.; May 31, 2006).

Case study 12 shows that using EVM requires a cultural change. As with any initiative, an agency’s 

management must show an interest in EVM if its use is to be sustained. Executive personnel should 

understand EVM terms and analysis products if they expect program managers and teams to use 

them. Additionally, at the program level, EVM requires qualified staff to independently assess what was 

accomplished. EVM training should be provided and tracked at all levels of personnel. This does not 

always happen, and government agencies struggle with how to obtain qualified and experienced personnel.

Perhaps the biggest challenge in using EVM is the trend to rebaseline programs. This happens when the 

current baseline is not adequate to complete all the work, causing a program to fall behind schedule or 

run over cost (see case study 13). A new baseline serves an important management control purpose when 

program goals can no longer be achieved: it gives perspective on the program’s current status. However, 

auditors should be aware that comparing the latest cost estimate with the most recent approved baseline 

provides an incomplete perspective on a program’s performance, because a rebaseline shortens the period 

of performance reported and resets the measurement of cost growth to zero.
24 Chapter 2 GAO-09-3SP

Case Study 13: Rebaselining, from NASA, GAO-04-642

Baseline development cost estimates for the programs GAO reviewed varied considerably 

from the programs’ initial baseline estimates. Development cost estimates of more 

than half the programs increased; for some programs, the increase was significant. 

The baseline development cost estimates for the 10 programs GAO reviewed in detail 

were rebaselined—that is, recalculated to reflect new costs, time periods, or resources 

associated with changes in program objectives, deliverables, or scope and plans. Although 

NASA provided specific reasons for the increased cost estimates and rebaselinings—such 

as delays in development or delivery of key system components and funding shortages—it 

did not have guidance for determining when rebaselinings were justified. Such criteria are 

important for instilling discipline in the cost estimating process.

GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective 

Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004). 

These challenges make it difficult for cost estimators to develop accurate estimates. Therefore, it is very 

important that agencies’ cost estimators have adequate guidance and training to help mitigate these 

challenges. In chapter 3, we discuss audit criteria related to cost estimating and EVM. We also identify 

some of the guidance we relied on to develop this guide. 
GAO-09-3SP Chapter 3 25

Criteria for Cost Estimating, EVM, and 

Data Reliability

Chapter 3

Government auditors use criteria as benchmarks for how well a program is performing. Criteria provide 

auditors with a context for what is required, what the program’s state should be, or what it was expected to 

accomplish. Criteria are the laws, regulations, policies, procedures, standards, measures, expert opinions, 

or expectations that define what should exist. When auditors conduct an audit, they should select criteria 

by whether they are reasonable, attainable, and relevant to the program’s objectives. 

Criteria include the

■ purpose or goals that statutes or regulations have prescribed or that the audited entity’s officials 

have set,

■ policies and procedures the audited entity’s officials have established, 

■ technically developed norms or standards, 

■ expert opinions, 

■ earlier performance, 

■ performance in the private sector, and 

■ leading organizations’ best practices.

In developing this guide, we researched legislation, regulations, policy, and guidance for the criteria 

that most pertained to cost estimating and EVM. Our research showed that while DOD has by far 

the most guidance on cost estimating and EVM in relation to civil agencies, other agencies are starting 

to develop policies and guidance. Therefore, we intend this guide as a starting point for auditors to 

identify criteria. 

For each new engagement, however, GAO auditors should exercise diligence to see what, if any, new 

legislation, regulation, policy, and guidance exists. Auditors also need to decide whether criteria are valid. 

Circumstances may have changed since they were established and may no longer conform to sound 

management principles or reflect current conditions. In such cases, GAO needs to select or develop criteria 

that are appropriate for the engagement’s objectives. 

Table 3 lists criteria related to cost estimating and EVM. Each criterion is described in more detail in 

appendix V. 
26 Chapter 3 GAO-09-3SP

Table 3: Cost Estimating and EVM Criteria for Federal Agencies: Legislation, Regulations, Policies, and 

Guidance

Type 

and 

date Title

Applicable 

agency Notes

Legislation or regulation

1968 SAR: Selected Acquisition Reports, 10 

U.S.C. § 2432 (2006)

DOD Became permanent law in 1982; applies 

only to DOD’s major defense acquisition 

programs

1982 Unit Cost Reports (“Nunn-McCurdy”); 10 

U.S.C. § 2433 (2006)

DOD Applies only to DOD’s major defense 

acquisition programs

1983 Independent Cost Estimates; Operational 

Manpower Requirements, 10 U.S.C. § 2434 

(2006)

DOD Applies only to DOD’s major defense 

acquisition programs

1993 GPRA: Government Performance and 

Results Act, Pub. L. No. 103-62 (1993)

All Requires agencies to prepare (1) multiyear 

strategic plans describing mission goals 

and methods for reaching them and (2) 

annual program performance reports 

to review progress toward annual 

performance goals

1994 The Federal Acquisition Streamlining Act 

of 1994, § 5051(a), 41 U.S.C. § 263 (2000).

All civilian 

agencies

Established congressional policy that 

agencies should achieve, on average, 

90 percent of cost, performance, and 

schedule goals established for their 

major acquisition programs; requires 

an agency to approve or define cost, 

performance, and schedule goals and to 

determine whether there is a continuing 

need for programs that are significantly 

behind schedule, over budget, or not in 

compliance with performance or capability 

requirements and to identify suitable 

actions to be taken

1996 CCA: Clinger-Cohen Act of 1996, 40 U.S.C. 

§§ 11101–11704 (Supp. V 2005)

All Requires agencies to base decisions about 

information technology investments 

on quantitative and qualitative factors 

associated with their costs, benefits, 

and risks and to use performance data 

to demonstrate how well expenditures 

support program improvements

2006 Federal Acquisition Regulation (FAR), 

Major Systems Acquisition, 48 C.F.R. 

part 34, subpart 34.2, Earned Value 

Management System

All Earned Value Management System policy 

was added by Federal Acquisition Circular 

2005-11, July 5, 2006, Item I—Earned Value 

Management System (EVMS) (FAR Case 

2004-019)
GAO-09-3SP Chapter 3 27

Type 

and 

date Title

Applicable 

agency Notes

2006 Major Automated Information System 

Programs, 10 U.S.C. §§ 2445a–2445d 

(2006)

DOD Oversight requirements for DOD’s 

major automated information system 

(MAIS) programs, including estimates of 

development costs and full life-cycle costs 

as well as program baseline and variance 

reporting requirements

2008 Defense Federal Acquisition Regulation 

Supplement; Earned Value Management 

Systems (DFARS Case 2005–D006), 73 

Fed. Reg. 21,846 (April 23, 2008), primarily 

codified at 48 C.F.R. subpart 234.2 and 

part 252 (sections 252.234-7001 and 7002)

DOD DOD’s final rule (1) amending the Defense 

Federal Acquisition Regulation Supplement 

(DFARS) to update requirements for DOD 

contractors to establish and maintain EVM 

systems and (2) eliminating requirements 

for DOD contractors to submit cost/

schedule status reports

Policy

1976 OMB, Major Systems Acquisitions, Circular 

A-109 (Washington, D.C.: Apr. 5, 1976)

All 

1992 OMB, Guidelines and Discount Rates for 

Benefit-Cost Analysis of Federal Programs, 

Circular No. A-94 Revised (Washington, 

D.C.: Oct. 29, 1992)

All

1995 DOD, Economic Analysis for 

Decisionmaking, Instruction No. 7041.3 

(Washington, D.C.: USD, Nov. 7, 1995)

DOD

2003 DOD, The Defense Acquisition System, 

Directive No. 5000.1 (Washington, D.C.: 

USD, May 12, 2003). Redesignated 5000.01 

and certified current as of Nov. 20, 2007

DOD States that every program manager 

must establish program goals for the 

minimum number of cost, schedule, and 

performance parameters that describe the 

program over its life cycle and identify any 

deviations

2003 DOD, Operation of the Defense Acquisition 

System, Instruction No. 5000.2 

(Washington, D.C.: USD, May 12, 2003). 

Canceled and reissued by Instruction No. 

5000.02 on Dec. 8, 2008

DOD Describes the standard framework for 

defense acquisition systems: defining the 

concept, analyzing alternatives, developing 

technology, developing the system and 

demonstrating that it works, producing 

and deploying the system, and operating 

and supporting it throughout its useful life

2004 National Security Space Acquisition Policy, 

Number 03-01, Guidance for DOD Space 

System Acquisition Process (Washington, 

D.C.: revised Dec. 27, 2004)

DOD

2005 DOD, “Revision to DOD Earned Value 

Management Policy,” memorandum, 

Under Secretary of Defense, Acquisition, 

Technology, and Logistics (Washington, 

D.C.: Mar. 7, 2005)

DOD
28 Chapter 3 GAO-09-3SP

Type 

and 

date Title

Applicable 

agency Notes

2005 OMB, “Improving Information Technology 

(IT) Project Planning and Execution,” 

memorandum for Chief Information 

Officers No. M-05-23 (Washington, D.C.: 

Aug. 4, 2005)

All

2006 OMB, Capital Programming Guide, 

Supplement to Circular A-11, Part 7, 

Preparation, Submission, and Execution of 

the Budget (Washington, D.C.: Executive 

Office of the President, June 2006)

All 

2006 DOD, Cost Analysis Improvement Group 

(CAIG), Directive No. 5000.04 (Washington, 

D.C.: Aug. 16, 2006)

DOD

Guidance

1992 CAIG, Operating and Support CostEstimating Guide (Washington, D.C.: DOD, 

Office of the Secretary, May 1992)

DOD

1992 DOD, Cost Analysis Guidance and 

Procedures, DOD Directive 5000.4-M 

(Washington, D.C.: OSD, Dec. 11, 1992)

DOD

2003 DOD, The Program Manager’s Guide to 

the Integrated Baseline Review Process

(Washington, D.C.: OSD, April 2003) 

DOD

2004 NDIA, National Defense Industrial 

Association (NDIA) Program Management 

Systems Committee (PMSC) Surveillance 

Guide (Arlington, Va.: October 2004)

All 

2005 NDIA, National Defense Industrial 

Association (NDIA) Program Management 

Systems Committee (PMSC) Earned Value 

Management Systems Intent Guide 

(Arlington, Va.: January 2005)

All 

2006 Defense Contract Management Agency, 

Department of Defense Earned Value 

Management Implementation Guide

(Alexandria, Va.: October 2006)

DOD, FAA, 

NASA

2006 National Defense Industrial Association, 

Program Management Systems 

Committee, “NDIA PMSC ANSI/EIA 748 

Earned Value Management System 

Acceptance Guide,” draft, working 

release for user comment (Arlington, Va.: 

November 2006) 

All
GAO-09-3SP Chapter 3 29

Type 

and 

date Title

Applicable 

agency Notes

2007 National Defense Industrial Association, 

Program Management Systems 

Committee, “NDIA PMSC Earned Value 

Management Systems Application Guide,” 

draft, working release for user comment 

(Arlington, Va.: March 2007)

All 

2007 American National Standards Institute, 

Information Technology Association of 

America, Earned Value Management 

Systems (ANSI/EIA 748-B) (Arlington, Va.: 

July 9, 2007)

All

Source: GAO, DOD, and OMB.

Determining Data Reliability

Auditors need to collect data produced from both a program’s cost estimate and its EVM system. They 

can collect these data by questionnaires, structured interviews, direct observations, or computations, 

among other methods. (Appendix VI is a sample data collection instrument; appendix VII gives reasons 

why auditors need the information.) After auditors have collected their data, they must judge the data for 

integrity as well as for quality in terms of validity, reliability, and consistency with fact. 

For cost estimates, auditors must confirm that, at minimum, internal quality control checks show that the 

data are reliable and valid. To do this, they must have source data and must estimate the rationale for each 

cost element, to verify that

■ the parameters (or input data) used to create the estimate are valid and applicable,22

■ labor costs include a time-phased breakdown of labor hours and rates,

■ the calculations for each cost element are correct and the results make sense,

■ the program cost estimate is an accurate total of subelement costs, and

■ escalation was properly applied to account for differences in the price of goods and services over 

time.

Auditors should clarify with cost estimators issues about data and methodology. For example, they might 

ask what adjustments were made to account for differences between the new and existing systems with 

respect to design, manufacturing processes, and types of materials. In addition, auditors should look for 

multiple sources of data that converge toward the same number, in order to gain confidence in the data 

used to create the estimate.

It is particularly important that auditors understand problems associated with the historical data—such 

as program redesign, schedule slips, and budget cuts—and whether the cost estimators “cleansed the 

22The auditor must ask the cost estimator if the technical assumptions for a new program have been tested for reasonableness. A 

program whose technical assumptions are not supported by historical data may be a high-risk program or its data may not be valid. 

Closing the gap between what a program wants to achieve and what has been achieved in the past is imperative for proper data 

validation.
30 Chapter 3 GAO-09-3SP

data” to remove their effects. According to experts in the cost community, program inefficiencies should 

not be removed from historical data, since the development of most complex systems usually encounters 

problems. The experts stress that removing data associated with past problems is naïve and introduces 

unnecessary risk. (This topic is discussed in chapter 10.) 

With regard to EVM, auditors should request a copy of the system compliance or validation letter 

that shows the contractor’s ability to satisfy the 32 EVM guidelines (discussed in chapter 18).23 These 

guidelines are test points to determine the quality of a contractor’s EVM system. Contract performance 

reports (CPR) formally submitted to the agency should be examined for reasonableness, accuracy, and 

consistency with other program status reports as a continuous measure of the EVM system quality 

and robustness. Auditors should also request a copy of the integrated baseline review (IBR) results (also 

discussed in chapter 18) to see what risks were identified and whether they were mitigated. Auditors 

should request copies of internal management documents or reports that use EVM data to ensure that 

EVM is being used for management, not just for external reporting. Finally, to ensure that EVM data are 

valid and accurate, auditors should look for evidence that EVM analysis and surveillance are performed 

regularly by staff trained in this specialty. 

23 For DOD programs, the Defense Contract Management Agency (DCMA) should have a copy of the EVM validation letter.
GAO-09-3SP Chapter 4 31

Cost Analysis Overview 

Chapter 4

Although “cost estimating” and “cost analysis” are often used interchangeably, cost estimating is a specific 

activity within cost analysis. Cost analysis is a powerful tool, because it requires a rigorous and systematic 

analysis that results in a better understanding of the program being acquired. This understanding, in turn, 

leads to improved program management in applying resources and mitigating program risks.

Differentiating Cost Analysis and Cost Estimating

Cost analysis, used to develop cost estimates for such things as hardware systems, automated information 

systems, civil projects, manpower, and training, can be defined as

■ the effort to develop, analyze, and document cost estimates with analytical approaches and 

techniques;

■ the process of analyzing, interpreting, and estimating the incremental and total resources required 

to support past, present, and future systems—an integral step in selecting alternatives; and

■ a tool for evaluating resource requirements at key milestones and decision points in the acquisition 

process.

Cost estimating involves collecting and analyzing historical data and applying quantitative models, 

techniques, tools, and databases to predict a program’s future cost. More simply, cost estimating combines 

science and art to predict the future cost of something based on known historical data that are adjusted to 

reflect new materials, technology, software languages, and development teams.

Because cost estimating is complex, sophisticated cost analysts should combine concepts from such 

disciplines as accounting, budgeting, computer science, economics, engineering, mathematics, and 

statistics and should even employ concepts from marketing and public affairs. And because cost 

estimating requires such a wide range of disciplines, it is important that the cost analyst either be familiar 

with these disciplines or have access to an expert in these fields. 

Main Cost Estimate Categories

Auditors are likely to encounter two main cost estimate categories:

■ a life-cycle cost estimate (LCCE) that may include independent cost estimates, independent cost 

assessments, or total ownership costs, and

■ a business case analysis (BCA) that may include an analysis of alternatives or economic analyses. 
32 Chapter 4 GAO-09-3SP

Auditors may also review other types of cost estimates, such as independent cost assessments (ICA), 

nonadvocate reviews (NAR), and independent government cost estimates (IGCE). These types of 

estimates are commonly developed by civilian agencies.

Life-Cycle Cost Estimate

A life-cycle cost estimate provides an exhaustive and structured accounting of all resources and associated 

cost elements required to develop, produce, deploy, and sustain a particular program. Life cycle can 

be thought of as a “cradle to grave” approach to managing a program throughout its useful life. This 

entails identifying all cost elements that pertain to the program from initial concept all the way through 

operations, support, and disposal. An LCCE encompasses all past (or sunk), present, and future costs for 

every aspect of the program, regardless of funding source. 

Life-cycle costing enhances decision making, especially in early planning and concept formulation of 

acquisition. Design trade-off studies conducted in this period can be evaluated on a total cost basis, as well 

as on a performance and technical basis. A life-cycle cost estimate can support budgetary decisions, key 

decision points, milestone reviews, and investment decisions.

The LCCE usually becomes the program’s budget baseline. Using the LCCE to determine the budget 

helps to ensure that all costs are fully accounted for so that resources are adequate to support the program. 

DOD identifies four phases that an LCCE must address: research and development, procurement 

and investment, operations and support, and disposal. Civilian agencies may refer to the first two as 

development, modernization, and enhancement and may include in them acquisition planning and 

funding. Similarly, civilian agencies may refer to operations and support as “steady state” and include them 

in operations and maintenance activities. Although these terms mean essentially the same thing, they can 

differ from agency to agency. DOD’s four phases are described below. 

1. Research and development include development and design costs for system engineering and 

design, test and evaluation, and other costs for system design features. They include costs for 

development, design, startup, initial vehicles, software, test and evaluation, special tooling and test 

equipment, and facility changes.

2. Procurement and investment include total production and deployment costs (e.g., site activation, 

training) of the prime system and its related support equipment and facilities. Also included are any 

related equipment and material furnished by the government, initial spare and repair parts, interim 

contractor support, and other efforts. 

3. Operations and support are all direct and indirect costs incurred in using the prime system—

manpower, fuel, maintenance, and support—through the entire life cycle. Also included are 

sustaining engineering and other collateral activities.

4. Disposal, or inactivation, includes the costs of disposing of the prime equipment after its useful life.

Because they encompass all possible costs, LCCEs provide a wealth of information about how much 

programs are expected to cost over time. This information can be displayed visually to show what funding 

is needed at a particular time and when the program is expected to move from one phase to another. For 

example, figure 3 is a life-cycle cost profile for a hypothetical space system.
GAO-09-3SP Chapter 4 33

Figure 3: Life-Cycle Cost Estimate for a Space System

Cost

0

Phase B

ATP

Final

design

O&M support

start

Launch 1

Launch 2

Launch 3

Launch N

IOC FOC

RDT&E

Includes development

and production of first

two vehicles

Space system

life cycle

Follow-on buys occur

after final design verification

O&M staff in place

before launch 1

Procurement

Includes production

of follow-on buys

(typically lots of 2

or 3 SVs) O&M

Operators and controllers

through system EOL

Source: DOD.

RDT&E

Procurement

O&M

Note: O&M = operations and maintenance; RDT&E = research, development, test, and evaluation; SV = space vehicle; EOL = end of 

life; IOC = initial operational capacity; FOC = full operational capacity.

Figure 3 illustrates how space systems must invest heavily in research and development because once a 

system is launched into space, it cannot be retrieved for maintenance. Other systems such as aircraft, ships, 

and information technology systems typically incur hefty operations costs in relation to development and 

production costs. Such mission operations costs are very large because the systems can be retrieved and 

maintained and therefore require sophisticated logistics support and recurring broad-based training for 

large user populations. Thus, having full life-cycle costs is important for successfully planning program 

resource requirements and making wise decisions.

Business Case Analysis 

A business case analysis, sometimes referred to as a cost benefit analysis, is a comparative analysis that 

presents facts and supporting details among competing alternatives. A BCA considers not only all the 

life-cycle costs that an LCCE identifies but also quantifiable and nonquantifiable benefits. It should 

be unbiased by considering all possible alternatives and should not be developed solely for supporting 

a predetermined solution. Moreover, a BCA should be rigorous enough that independent auditors can 

review it and clearly understand why a particular alternative was chosen. 
34 Chapter 4 GAO-09-3SP

A BCA seeks to find the best value solution by linking each alternative to how it satisfies a strategic 

objective. Each alternative should identify the

■ relative life-cycle costs and benefits;

■ methods and rationale for quantifying the life-cycle costs and benefits;

■ effect and value of cost, schedule, and performance tradeoffs;

■ sensitivity to changes in assumptions; and

■ risk factors.

On the basis of this information, the BCA then recommends the best alternative. In addition to 

supporting an investment decision, the BCA should be considered a living document and should be 

updated often to reflect changes in scope, schedule, or budget. In this way, the BCA is a valuable tool for 

validating decisions to sustain or enhance the program. 

Auditors may encounter other estimates that fall into one of the two main categories of cost estimates. 

For example, an auditor may examine an independent cost estimate, independent cost assessment, 

independent government cost estimates, total ownership cost, or rough order of magnitude estimate—all 

variations of a life-cycle cost estimate. Similarly, instead of reviewing a business case analysis, an auditor 

may review an analysis of alternatives (AOA), a cost-effectiveness analysis (CEA), or an economic analysis 

(EA). Each of these analyses is a variation, in one form or another, of a BCA. Table 4 looks more closely at 

the different types of cost estimates that can be developed. 

Table 4: Life-Cycle Cost Estimates, Types of Business Case Analyses, and Other Types of Cost 

Estimates

Estimate type Level of effort Description

Life-cycle cost estimate

Independent cost 

estimate 

Usually requires a large 

team, may take many 

months to accomplish, 

and addresses the full 

LCCE

An ICE, conducted by an organization independent of 

the acquisition chain of command, is based on the same 

detailed technical and procurement information used 

to make the baseline estimate—usually the program or 

project LCCE. ICEs are developed to support new programs 

or conversion, activation, modernization, or service life 

extensions and to support DOD milestone decisions for 

major defense acquisition programsa

An estimate might cover a program’s entire life cycle, 

one program phase, or one high-value, highly visible, or 

high-interest item within a phase. ICEs are used primarily 

to validate program or project LCCEs and are typically 

reconciled with them

Because the team performing the ICE is independent, it 

provides an unbiased test of whether the program office 

cost estimate is reasonable. It is also used to identify risks 

related to budget shortfalls or excesses
GAO-09-3SP Chapter 4 35

Estimate type Level of effort Description

Total ownership 

cost estimate 

Requires a large team, 

may take many months 

to accomplish, and 

addresses the full LCCE

Related to LCCE but broader in scope, a total ownership cost 

estimate consists of the elements of life-cycle cost plus some 

infrastructure and business process costs not necessarily 

attributable to a program

Infrastructure includes acquisition and central logistics 

activities; nonunit central training; personnel administration 

and benefits; medical care; and installation, communications, 

and information infrastructure to support military bases. It is 

normally found in DOD programs

Business case analysis

Analysis of 

alternatives and

cost effectiveness 

analysis 

Requires a large team, 

may take many months 

to accomplish, and 

addresses the full LCCE

AOA compares the operational effectiveness, suitability, and 

LCCE of alternatives that appear to satisfy established capability 

needs. Its major components are a CEA and cost analysis

AOAs try to identify the most promising of several conceptual 

alternatives; analysis and conclusions are typically used to 

justify initiating an acquisition program. An AOA also looks at 

mission threat and dependencies on other programs

When an AOA cannot quantify benefits, a CEA is more 

appropriate. A CEA is conducted whenever it is unnecessary 

or impractical to consider the dollar value of benefits, as when 

various alternatives have the same annual monetary benefits

Both the AOA and CEA should address each alternative’s 

advantages, disadvantages, associated risks, and uncertainties 

and how they might influence the comparison

Economic analysis 

and cost benefit 

analysis

Requires a large team, 

may take many months 

to accomplish, and 

addresses the full LCCE

EA is a conceptual framework for systematically investigating 

problems of choice. Posing various alternatives for reaching an 

objective, it analyzes the LCCE and benefits of each one, usually 

with a return on investment analysis

Present value is also an important concept: Since an LCCE 

does not consider the time value of money, it is necessary to 

determine when expenditures for alternatives will be made

EA expands cost analysis by examining the effects of the time 

value of money on investment decisions. After cost estimates 

have been generated, they must be time-phased to allow for 

alternative expenditure patterns. Assuming equal benefits, 

the alternative with the least present value cost is the most 

desirable: it implies a more efficient allocation of resources 

Other 

Rough order of 

magnitude 

May be done by a 

small group or one 

person; can be done in 

hours, days, or weeks; 

and may cover only a 

portion of the LCCE

Developed when a quick estimate is needed and few details 

are available. Usually based on historical ratio information, it 

is typically developed to support what-if analyses and can be 

developed for a particular phase or portion of an estimate to the 

entire cost estimate, depending on available data. It is helpful for 

examining differences in high-level alternatives to see which are 

the most feasible. Because it is developed from limited data and 

in a short time, a rough order of magnitude analysis should never 

be considered a budget-quality cost estimate
36 Chapter 4 GAO-09-3SP

Estimate type Level of effort Description

Independent cost 

assessment 

Requires a small group; 

may take months to 

accomplish, depending 

on how much of the 

LCCE is being reviewed

An ICA is an outside, nonadvocate’s evaluation of a cost 

estimate’s quality and accuracy, looking specifically at a 

program’s technical approach, risk, and acquisition strategy 

to ensure that the program’s cost estimate captures all 

requirements

Typically requested by a program manager or outside source, 

it may be used to determine whether the cost estimate 

reflects the program of record. It is not as formal as an ICE 

and does not have to be performed by an organization 

independent of the acquisition chain of command, although it 

usually is

An ICA usually does not address a program’s entire life cycle

Independent 

government cost 

estimate 

Requires a small group, 

may take months to 

accomplish, and covers 

only the LCCE phase 

under contract

An IGCE is conducted to check the reasonableness of a 

contractor’s cost proposal and to make sure that the offered 

prices are within the budget range for a particular program

The program manager submits it as part of a request for 

contract funding. It documents the government’s assessment 

of the program’s most probable cost and ensures that enough 

funds are available to execute it

It is also helpful in assessing the feasibility of individual tasks 

to determine if the associated costs are reasonable

Estimate at 

completion 

Requires nominal effort 

once all EVM data are 

on hand and have been 

determined reliable; 

covers only the LCCE 

phase under contract

An EAC is an independent assessment of the cost to complete 

authorized work based on a contractor’s historical EVM 

performance

It uses various EVM metrics to forecast the expected final cost: 

EAC = actual costs incurred + (budgeted cost for work 

remaining / EVM performance factor)

The performance factor can be based on many different EVM 

metrics that capture cost and schedule status to date

Source: GAO, DOD, NIH, OMB, and SCEA.

a

For more detail, see app. V, ICEs, 10 U.S.C. § 2434.

The Overall Significance of Cost Estimates

Not an end in itself, cost estimating is part of a total systems analysis. It is a critical element in any 

acquisition process and helps decision makers evaluate resource requirements at milestones and other 

important decision points. 

Cost estimates 

■ establish and defend budgets and 

■ drive affordability analysis. 

Cost estimates are integral to determining and communicating a realistic view of likely cost and schedule 

outcomes that can be used to plan the work necessary to develop, produce, install, and support a program.
GAO-09-3SP Chapter 4 37

Cost estimating also provides valuable information to help determine whether a program is feasible, how 

it should be designed, and the resources needed to support it. Further, cost estimating is necessary for 

making program, technical, and schedule analyses and to support other processes such as

■ selecting sources;

■ assessing technology changes, analyzing alternatives, and performing design trade-offs; and 

■ satisfying statutory and oversight requirements.

Cost Estimates in Acquisition

An acquisition program focuses on the cost of developing and procuring an end item and whether enough 

resources and funding are available. The end product of the acquisition process is a program capability 

that meets its users’ needs at a reasonable price. During the acquisition process, decisions must be made on 

how best to consume labor, capital, equipment, and other finite resources. A realistic cost estimate allows 

better decision making, in that an adequate budget can accomplish the tasks that ultimately increase a 

program’s probability of success. 

Acquisition is an event-driven process, in that programs must typically pass through various milestones or 

investment reviews in which they are held accountable for their accomplishments. Cost estimates play an 

important role in these milestone or investment decisions. For example, in government programs, a cost 

estimate should be validated if a major program is to continue through its many acquisition reviews and 

other key decision points. 

Validation involves testing an estimate to see if it is reasonable and includes all necessary costs. Testing can 

be as simple as comparing results with historical data from similar programs or using another estimating 

method to see if results are similar. Industry requires similar scrutiny throughout development, in what is 

commonly referred to as passing through specific gates. 

Once a cost estimate has been accepted and approved, it should be updated periodically as the program 

matures and as schedules and requirements change. Updated estimates help give management control over 

a project’s resources when new requirements are called for under tight budget conditions. This is especially 

important early in a project, when less is known about requirements and the opportunity for change 

(and cost growth) is greater. As more knowledge is gained, programs can retire some risk and reduce the 

potential for unexpected cost and schedule growth. 

Cost estimates tend to become more certain as actual costs begin to replace earlier estimates. This happens 

when risks are either mitigated or realized. If risks actually occur, the resulting cost growth becomes 

absorbed by the cost estimate. 

For this reason, it is important to continually update estimates with actual costs, so that management 

has the best information available for making informed decisions. In addition, narrow risk ranges 

should be viewed as suspect, because more cost estimates tend to overrun than underrun. These 

processes are illustrated in what is commonly called the “cone of uncertainty,” which are depicted in 

figure 4. 
38 Chapter 4 GAO-09-3SP

Figure 4: Cone of Uncertainty

Estimate becomes more certain as program progresses

Estimate tends to grow over time as risks are realized

Concept refinement gate

Cost estimate

baseline

Technology development gate Start of program and start

of system integration gate

Uncertainty about cost estimate is high

Uncertainty is low

Source: GAO.

It is important to have a track record of the estimate so one can measure growth from what the estimate 

should have been. Therefore, tying growth and risk together is critical because the risk distribution 

identifies the range of anticipated growth. 

The Importance of Cost Estimates in Establishing Budgets

A program’s approved cost estimate is often used to create the budget spending plan. This plan outlines 

how and at what rate the program funding will be spent over time. Since resources are not infinite, 

budgeting requires a delicate balancing act to ensure that the rate of spending closely mirrors available 

resources and funding. And because cost estimates are based on assumptions that certain tasks will 

happen at specific times, it is imperative that funding be available when needed so as to not disrupt the 

program schedule. 

Because a reasonable and supportable budget is essential to a program’s efficient and timely execution, a 

competent estimate is the key foundation of a good budget. For a government agency, accurate estimates 

help in assessing the reasonableness of a contractor’s proposals and program budgets. Credible cost 

estimates also help program offices justify budgets to the Congress, OMB, department secretaries, and 

others. Moreover, cost estimates are often used to help determine how budget cuts may hinder a program’s 

progress or effectiveness.

Outside the government, contractors need accurate estimates of the costs required to complete a task in 

order to ensure maximum productivity and profitability. Estimates that are too low can reduce profits 
GAO-09-3SP Chapter 4 39

if the contract is firm fixed price, and estimates that are too high will diminish a contractor’s ability to 

compete in the marketplace. 

While contractors occasionally propose unrealistically low cost estimates for strategic purposes—for 

example, “buying-in”—such outcomes can be attributed to poor cost estimating. This sometimes happens 

when contractors are highly optimistic in estimating potential risks. As a program whose budget is 

based on such estimates is developed, it becomes apparent sooner or later that either the developer or the 

customer must pay for a cost overrun, as case study 14 indicates. 

Case Study 14: Realistic Estimates, from Defense Acquisitions, 

GAO-05-183

In negotiating the contract for the first four Virginia class ships, program officials stated 

that they were constrained in negotiating the target price to the amount funded for the 

program, risking cost growth at the outset. The shipbuilders said that they accepted a 

challenge to design and construct the ships for $748 million less than their estimated 

costs, because the contract protected their financial risk. Despite the significant risk of cost 

growth, the Navy did not identify any funding for probable cost growth, given available 

guidance at the time. The fiscal year 2005 President’s Budget showed that budgets for the 

two Virginia class case study ships had increased by $734 million. However, on the basis 

of July 2004 data, GAO projected that additional cost growth on contracts for the two 

ships would be likely to reach $840 million, perhaps higher. In the fiscal year 2006 budget, 

the Navy requested funds to cover cost increases expected to reach to approximately $1 

billion.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

Cost Estimates and Affordability

Affordability is the degree to which an acquisition program’s funding requirements fit within the agency’s 

overall portfolio plan. Whether a program is affordable depends a great deal on the quality of its cost 

estimate. Therefore, agencies can follow the 12-step estimating process we outlined in chapter 1 to 

ensure that they are creating and making decisions based on credible cost estimates. The 12-step process 

addresses best practices, including defining the program’s purpose, developing the estimating plan, 

defining the program’s characteristics, determining the estimating approach, identifying ground rules and 

assumptions, obtaining data, developing the point estimate, conducting sensitivity analysis, performing 

a risk or uncertainty analysis, documenting the estimate, presenting it to management for approval, and 

updating it to reflect actual costs and changes. Following these steps ensures that realistic cost estimates 

are developed and presented to management, enabling them to make informed decisions about whether 

the program is affordable within the portfolio plan. 

Decision makers should consider affordability at each decision point in a program’s life cycle. It is 

important to know the program’s cost at particular intervals, in order to ensure that adequate funding 

is available to execute the program according to plan. Affordability analysis validates that the program’s 

acquisition strategy has an adequate budget for its planned resources (see figure 5).
40 Chapter 4 GAO-09-3SP

Figure 5: An Affordability Assessment

0

2

4

6

8

10

12

14

16

18

20

FY1

FY2

FY3

FY4

FY5

FY6

FY7

FY8

FY9

FY10

FY11

FY12

FY13

FY14

FY15

FY16

FY17

FY18

FY19

FY20

Program A

Program B

Program C

Program D

Program E

Program F

Program G

Dollars (in billions)

Projected budget

Source: DOD.

In figure 5, seven programs A–G are plotted against time, with the resources they will need to support 

their goals. The benefit of plotting the programs together gives decision makers a high-level analysis of 

their portfolio and the resources they will need in the future. In this example, it appears that funding 

needs are relatively stable in fiscal years 1–12, but from fiscal year 12 to fiscal year 16, an increasing need 

for additional funding is readily apparent. This is commonly referred to as a bow-wave, meaning there is 

an impending spike in the requirement for additional funds. Whether these funds will be available will 

determine which programs remain within the portfolio. Because the programs must compete against one 

another for limited funds, it is considered a best practice to perform the affordability assessment at the 

agency level, not program by program. 

While approaches may vary, an affordability assessment should address requirements at least through 

the programming period and, preferably, several years beyond. Thus, LCCEs give decision makers 

important information in that not all programs require the same type of funding profile. In fact, different 

commodities require various outlays of funding and are affected by different cost drivers. Figure 6

illustrates this point with typical funding curves by program phase. It shows that while some programs 

may cost less to develop—for example, research and development in construction programs differ from 

fixed-wing aircraft—they may require more or less funding for investment, operations, and support in the 

out-years. 
GAO-09-3SP Chapter 4 41

Figure 6: Typical Capital Asset Acquisition Funding Profiles by Phase 

0

20

40

60

80

100

R&D Investment

Construction

Percent of project cost

O&S/

disposal

0

20

40

60

80

100

R&D Investment

Ships

Percent of project cost

O&S/

disposal

0

20

40

60

80

100

R&D Investment

Space

Percent of project cost

O&S/

disposal

0

20

40

60

80

100

R&D Investment

Fixed-wing aircraft

Percent of project cost

O&S/

disposal

0

20

40

60

80

100

R&D Investment

Surface vehicles

Percent of project cost

O&S/

disposal

Source: GAO and DOD.

Line graphs or sand charts like those in figure 5, therefore, are often used to show how a program fits 

within the organizational plan, both overall and by individual program components. Such charts allow 

decision makers to determine how and if the program fits within the overall budget. It is very important 

for LCCEs to be both realistic and timely, available to decision makers as early as possible. Case studies 15 

and 16 show how this often does not happen.

Case Study 15: Importance of Realistic LCCEs, from Combating Nuclear 

Smuggling, GAO-07-133R

The Department of Homeland Security’s (DHS) Domestic Nuclear Detection Office (DNDO)

had underestimated life-cycle costs for plastic scintillators and advanced spectroscopic 

portal monitors. Although DNDO’s analysis assumed a 5-year life cycle for both, DNDO 

officials told GAO that a 10-year life cycle was more reasonable. DNDO’s analysis had 

assumed annual maintenance costs at 10 percent of their procurement costs: maintenance 

costs for the scintillators would be about $5,500 per year per unit, based on a $55,000 

purchase price, and maintenance costs for the monitors would be about $38,000 per year 

per unit, based on a $377,000 purchase price. DNDO’s analysis had not accounted for about 

$181 million in potential maintenance costs for the monitors alone. With the much higher 

maintenance costs, and doubling the life cycle, the long-term implications would be 

magnified.

GAO, Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support 

the Purchase of New Radiation Detection Portal Monitors Was Not Based on 

Available Performance Data and Did Not Fully Evaluate All the Monitors’ Costs 

and Benefits, GAO-07-133R (Washington, D.C.: Oct. 17, 2006).
42 Chapter 4 GAO-09-3SP

Case Study 16: Importance of Realistic LCCEs, from Space Acquisitions, 

GAO-07-96

GAO has in the past identified a number of causes behind cost growth and related 

problems in DOD’s major space acquisition programs, but several consistently stand 

out. On a broad scale, DOD starts more weapons programs than it can afford, creating 

competition for funding that encourages low-cost estimating and optimistic scheduling, 

overpromising, suppressing bad news, and for space programs, forsaking the opportunity 

to identify and assess potentially better alternatives. Programs focus on advocacy at the 

expense of realism and sound management. 

With too many programs in its portfolio, DOD is invariably forced to shift funds to and from 

programs—particularly as programs experience problems that require more time and 

money. Such shifts, in turn, have had costly, reverberating effects. In previous testimony 

and reports, GAO has stressed that DOD could avoid costly funding shifts.

It could do this by developing an overall investment strategy to prioritize systems in its 

space portfolio with an eye toward balancing investments between legacy systems and 

new programs, as well as between science and technology programs and acquisition 

investments. Such prioritizing would also reduce incentives to produce low estimates.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Evolutionary Acquisition and Cost Estimation

GAO has reported that evolutionary acquisition is in line with commercial best practices.24 In evolutionary 

acquisition, a program evolves to its ultimate capabilities on the basis of mature technologies and available 

resources. This approach allows commercial companies to develop and produce more sophisticated 

products faster and less expensively than their predecessors. 

Commercial companies have found that trying to capture the knowledge required to stabilize a product 

design that entails significant new technical content is an unmanageable task, especially if the goal is to 

reduce development cycle times and get the product to the marketplace as quickly as possible. Therefore, 

product features and capabilities that cannot be achieved in the initial development are planned for 

development in the product’s future generations, when the technology has proven mature and other 

resources are available. 

Figure 7 compares evolutionary to single-step acquisition, commonly called the big bang approach. An 

evolutionary environment for developing and delivering new products reduces risk and makes cost more 

predictable. While a customer may not initially receive an ultimate capability, the product is available 

sooner, with higher quality and reliability and at a lower and more predictable cost. With this approach, 

improvements can be planned for the product’s future generations. (See case study 17.)

24GAO, Best Practices: Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from F/A-22 Program, GAO-03-645T

(Washington, D.C.: Apr. 11, 2003), pp. 2–3.
GAO-09-3SP Chapter 4 43

Figure 7: Evolutionary and Big Bang Acquisition Compared

Evolutionary acquisition approach

Single-step acquisition approach

Beginning 5 years 10 years 15 years

Beginning 5 years 10 years 15 years

Needed

technologies

are mature

Needed

technologies

are mature

1st generation

• Basic stealth platform

2nd generation

• Basic stealth platform

• Advanced avionics

3rd generation

• Basic stealth platform

• Advanced avionics

• Advanced intelligence

 and communications

1st generation

• Basic stealth platform

• Advanced avionics

• Advanced intelligence

 and communications

Source: GAO.

Case Study 17: Evolutionary Acquisition and Cost Estimates, from Best 

Practices, GAO-03-645T

The U.S. Air Force F/A-22 tactical fighter acquisition strategy was, at the outset, to achieve 

full capability in a big bang approach. By not using an evolutionary approach, the F/A22 took on significant risk and onerous technological challenges. While the big bang 

approach might have allowed the Air Force to compete more successfully for early funding, 

it hamstrung the program with many new, undemonstrated technologies, preventing the 

program from knowing cost and schedule ramifications throughout development. Cost, 

schedule, and performance problems resulted.

GAO, Best Practices: Better Acquisition Outcomes Are Possible If DOD Can 

Apply Lessons from F/A-22 Program, GAO-03-645T (Washington, D.C.: Apr. 

11, 2003).

Two development processes support evolutionary acquisition: incremental development and spiral 

development. Both processes are based on maturing technology over time instead of trying to do it all 

at once, as in the big bang approach. Both processes allow for developing hardware and software in 

manageable pieces by inserting new technology and capability over time. This usually results in fielding 

an initial hardware or software increment (or block) of capability with steady improvements over less time 

than is possible with a full development effort.
44 Chapter 4 GAO-09-3SP

Incremental Development

In incremental development, a desired capability is known at the beginning of the program and is met 

over time by developing several increments, each dependent on available mature technology. A core set of 

functions is identified and released in the first increment. Each new increment adds more functionality, 

and this process continues until all requirements are met. This assumes that the requirements are known 

up front and that lessons learned can be incorporated as the program matures. (See fig. 8.)

Figure 8: Incremental Development

0 0

Capability

Single step

Capability

Technology base

Requirements Requirements

Capability

IOC

FOC Capability

Technology base

Time Time

No capability Initial operationally useful capability

Incremental

Source: GAO.

Note: IOC = initial operational capability; FOC = final operational capability.

The advantages of incremental development are that a working product is available after the first 

increment and that each cycle results in greater capability. In addition, the program can be stopped 

when an increment is completed and still provide a usable product. Project management and testing can 

be easier, because the program is broken into smaller pieces. Its disadvantages are that the majority of 

the requirements must be known early, which is sometimes not feasible. In addition, cost and schedule 

overruns may result in an incomplete system if the program is terminated, because each increment only 

delivers a small part of the system at a time. Finally, operations and support for the program are often less 

efficient because of the need for additional learning for each increment release. (See case study 18.)

Case Study 18: Incremental Development, from Customs Service 

Modernization, GAO/AIMD-99-41

The U.S. Customs Service was developing and acquiring the Automated Commercial 

Environment (ACE) program in 21 increments. At the time of GAO’s review, Customs defined 

the functionality of only the first 2 increments, intending to define more later. Customs 

had nonetheless estimated costs and benefits for and had committed to investing in all 21 

increments. It had not estimated costs and benefits for each increment and did not know 

whether each increment would produce a reasonable return on investment. Furthermore, 

once it had deployed an increment at a pilot site for evaluation, Customs was not validating 

that estimated benefits had actually been achieved. It did not even know whether the 

program’s first increment, being piloted at three sites, was producing expected benefits 

or was cost-effective. Customs could determine only whether the first increment was 

performing at a level “equal to or better than” the legacy system.

GAO, Customs Service Modernization: Serious Management and Technical 

Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 1999).
GAO-09-3SP Chapter 4 45

Spiral Development

In spiral development, a desired capability is identified but the end-state requirements are not yet known. 

These requirements are refined through demonstration and risk management, based on continuous user 

feedback. This approach allows each increment to provide the best possible capability. Spiral development is 

often used in the commercial market, because it significantly reduces technical risk while incorporating new 

technology. The approach can, however, lead to increased cost and schedule risks. Spiral development can 

also present contract challenges due to repeating phases, trading requirements, and redefining deliverables. 

The advantage of spiral development is that it provides better risk management, because user needs and 

requirements are better defined. Its disadvantage is that the process is a lot harder to manage and usually 

results in increased cost and longer schedule.

While both incremental and spiral development have advantages and disadvantages, their major difference 

is the knowledge of the final product available to the program from the outset. With incremental 

development, the program office is aware of the final product to be delivered but develops it in stages. 

With spiral development, the final version of the product remains undetermined until the final stage has 

been completed—that is, the final product design is not known while the system is being built. 

Even though it is a best practice to follow evolutionary development rather than the big bang approach, 

it often makes cost estimating more difficult, because it requires that cost estimates be developed more 

frequently. In some cases, cost estimates made for programs are valid only for the initial increment or spiral, 

because future increments and spirals are not the product they were at the outset. Nevertheless, this approach 

is considered a best practice because it helps avoid unrealistic cost estimates, resulting in more realistic longrange investment funding and more effective resource allocation. Moreover, realistic cost estimates help 

management decide between competing options and increase the probability that the programs will succeed.

1. Best Practices Checklist: The Estimate

 The cost estimate type is clearly defined and is appropriate for its purpose.

 The cost estimate contains all elements suitable to its type—ICA, ICE, 

IGCE, LCCE, rough order of magnitude, total ownership cost: development, 

procurement, operating and support, disposal costs, and all sunk costs.

3 AOA, CEA, EA, cost-benefit analysis: consistently evaluate all alternatives. 

3 EA, cost-benefit analysis: portray estimates as present values.

 All program costs have been estimated, including all life-cycle costs.

 The cost estimate is independent of funding source and appropriations.

 An affordability analysis has been performed at the agency level to see how 

the program fits within the overall portfolio. 

3 The agency has a process for developing cost estimates that includes 

the 12-step best practice process outlined in chapter 1.

3 An overall agency portfolio sand chart displays all costs for every 

program. 

 The estimate is updated as actual costs become available from the EVM 

system or requirements change. 

 Post mortems and lessons learned are continually documented.
46 Chapter 4 GAO-09-3SP
GAO-09-3SP Chapter 5 47

The Cost Estimate’s Purpose, Scope, and 

Schedule

Chapter 5

A cost estimate is much more than just a single number. It is a compilation of many lower-level cost 

element estimates that span several years, based on the program schedule. Credible cost estimates are 

produced by following the rigorous 12 steps outlined in chapter 1 and are accompanied by detailed 

documentation. The documentation addresses the purpose of the estimate, the program background 

and system description, its schedule, the scope of the estimate (in terms of time and what is and is not 

included), the ground rules and assumptions, all data sources, estimating methodology and rationale, the 

results of the risk analysis, and a conclusion about whether the cost estimate is reasonable. Therefore, a 

good cost estimate—while taking the form of a single number—is supported by detailed documentation 

that describes how it was derived and how the expected funding will be spent in order to achieve a given 

objective. 

Purpose

The purpose of a cost estimate is determined by its intended use, and its intended use determines its scope 

and detail. Cost estimates have two general purposes: (1) to help managers evaluate affordability and 

performance against plans, as well as the selection of alternative systems and solutions, and (2) to support 

the budget process by providing estimates of the funding required to efficiently execute a program.

More specific applications include providing data for trade studies, independent reviews, and baseline 

changes. Regardless of why the cost estimate is being developed, it is important that the program’s 

purpose link to the agency’s missions, goals, and strategic objectives. The purpose of the program should 

also address the benefits it intends to deliver, along with the appropriate performance measures for 

benchmarking progress. 

Scope

To determine an estimate’s scope, cost analysts must identify the customer’s needs. That is, the cost 

estimator must determine if the estimate is required by law or policy or is requested. For example, 10 

U.S.C. § 2434 requires an independent cost estimate before a major defense acquisition program can 

advance into system development and demonstration or production and deployment. The statute specifies 

that the full life-cycle cost—all costs of development, procurement, military construction, and operations 

and support, without regard to funding source or management control—must be provided to the decision 

maker for consideration. 

In other cases, a program manager might want initially to address development and procurement, with 

estimates of operations and support to follow. However, if an estimate is to support the comparative 

analysis of alternatives, all cost elements of each alternative should be estimated to make each alternative’s 

cost transparent in relation to the others. 
48 Chapter 5 GAO-09-3SP

Where appropriate, the program manager and the cost estimating team should work together to 

determine the scope of the cost estimate. The scope will be determined by such issues as the time involved, 

what elements of work need to be estimated, who will develop the cost estimates, and how much cost 

estimating detail will be included. Where the program is in its life cycle will influence the quantity of 

detail for the cost estimate as well as the amount of data to be collected. For example, early in the life cycle 

the project may have a concept with no solid definition of the work involved. A cost estimate at this point 

in the life cycle will probably not require extensive detail. As the program becomes better defined, more 

detailed estimates should be prepared. 

Once the cost analysts know the context of the estimate or the customer’s needs, they can determine 

the estimate’s scope by its intended use and the availability of data. For example, if an independent cost 

analyst is typically given the time and other resources needed to conduct a thorough analysis, the analysis 

is expected to be more detailed than a what-if exercise. For either, however, more data are likely to be 

available for a system in production than for one that is in the early stages of development.

More detail, though, does not necessarily mean greater accuracy. Pursuing too much detail too early may 

be detrimental to an estimate’s quality. If a detailed technical description of the system being analyzed is 

lacking, along with detailed cost data, analysts will find it difficult to identify and estimate all the cost 

elements. It may be better to develop the estimate at a relatively high system level to ensure capturing 

all the lower-level elements. This is the value of so-called parametric estimating tools, which operate at a 

higher level of detail and are used when a system lacks detailed technical definition and cost data. These 

techniques also allow the analyst to link cost and schedule to measures of system size, functionality, or 

complexity in advance of detailed design definition.

Analysts should develop, and tailor, an estimate plan whose scope coincides with data availability and 

the estimate’s ultimate use. For a program in development, which is estimated primarily with parametric 

techniques and factors, the scope might be at a higher level of the WBS. (WBS is discussed in chapter 8.) 

As the program enters production, a lower level of detail would be expected.

As the analysts develop and revise the estimating plan, they should keep management informed of the 

initial approach and any changes in direction or method.25 Since the plan serves as an agreement between 

the customer and cost estimating team, it must clearly reflect the approved approach and should be 

distributed formally to all participants and organizations involved. 

Schedule

Regardless of an estimate’s ultimate use and its data availability, time can become an overriding constraint 

on its detail. When defining the elements to be estimated and when developing the plan, the cost 

estimating team must consider its time constraints relative to team staffing. Without adequate time to 

develop a competent estimate, the team may be unable to deliver a product of sufficiently high quality. For 

example, a rough-order-of-magnitude estimate could be developed in days, but a first-time budget-quality 

estimate would likely require many months. If, however, that budget estimate were simply an update to a 

25An estimate that supports an independent estimate for a DOD program presumably entails no requirement that the independent 

cost estimating team keep program management informed. Instead, the program office and independent cost estimators would be 

expected to maintain communication and brief one another on their results, so as to understand any differences between the two 

estimates.
GAO-09-3SP Chapter 5 49

previous estimate, it could be done faster. The more detail required, the more time and staff the estimate 

will require. It is important, therefore, that auditors understand the context of the cost estimate—why and 

how it was developed and whether it was an initial or follow-on estimate. (See case study 19.)

Case Study 19: The Estimate’s Context, from DOD Systems 

Modernization, GAO-06-215

Program officials told GAO that they had not developed the 2004 cost estimate in 

accordance with all SEI’s cost estimating criteria, because they had only a month to 

complete the economic analysis. By not following practices associated with reliable 

estimates—by not making a reliable estimate of system life-cycle costs—the Navy had 

decided on a course of action not based on sound and prudent decision making. This 

meant that the Navy’s investment decision was not adequately justified and that to the 

extent that program budgets were based on cost estimates, the likelihood of funding 

shortfalls and inadequate funding reserves was increased.

GAO, DOD Systems Modernization: Planned Investment in the Naval 

Tactical Command Support System Needs to Be Reassessed, GAO-06-215

(Washington, D.C.: Dec. 5, 2005).

After the customer has defined the task, the cost estimating team should create a detailed schedule that 

includes realistic key decision points or milestones and that provides margins for unforeseen, but not 

unexpected, delays. The team must ensure that the schedule is not overly optimistic. If the team wants 

or needs to compress the schedule to meet a due date, compression is acceptable as long as additional 

resources are available to complete the effort that fewer analysts would have accomplished in the longer 

period of time. If additional resources are not available, the estimate’s scope must be reduced. 

The essential point is that the team must attempt to ensure that the schedule is reasonable. When this is 

not possible, the schedule must be highlighted as having curtailed the team’s depth of analysis and the 

estimate’s resulting confidence level.

2. Best Practices Checklist: Purpose, Scope, and Schedule

 The estimate’s purpose is clearly defined.

 Its scope is clearly defined.

 The level of detail the estimate is to be conducted at is consistent with the 

level of detail available for the program. For example, an engineering buildup estimate should be conducted only on a well-defined program.

 The team has been allotted adequate time and resources to develop the 

estimate. 
50 Chapter 5 GAO-09-3SP
GAO-09-3SP Chapter 6 51

The Cost Assessment Team

Chapter 6

Cost estimates are developed with an inexact knowledge of what the final technical solution will be. 

Therefore, the cost assessment team must manage a great deal of risk—especially for programs that are 

highly complex or on technology’s cutting edge. Since cost estimates seek to define what a given solution 

will ultimately cost, the estimate must be bound by a multitude of assumptions and an interpretation of 

what the historical data represent. This tends to be a subjective effort, and these important decisions are 

often left to a cost analyst’s judgment. A cost analyst must possess a variety of skills to develop a highquality cost estimate that satisfies the 12 steps identified in chapter 1, as figure 9 illustrates. 

Figure 9: Disciplines and Concepts in Cost Analysis

Cost

Analysis

Engineering

Design

Materials

Performance parameters

Production engineering

Production process

Program development test

Scheduling

System integration

Computer science/mathematics

Analysis of commercial models

Analysis of proposals

Development of cost estimating

 relationship

Model development

Programming

Statistics

Forecasting

Learning curve applications

Regression analysis

Risk/uncertainty analysis

Sensitivity analysis

Accounting

Cost data analysis

Financial analysis

Overhead analysis

Proposal analysis

Economics

Break-even analysis

Foreign exchange rates

Industrial base analysis

Inflation

Labor agreements

Present value analysis

Public and government affairs

Appropriations process

Auditors

Legislative issues

Outside factors

Budgeting

Budget appropriations

Internal company (industry)

Program specific

Interpersonal skills

Approach

Estimate

Knowledge

Source: GAO.

Each discipline in figure 9 applies to cost estimating in its own unique way. For example, having an 

understanding of economics and accounting will help the cost estimator better understand the importance 

of inflation effects and how different accounting systems capture costs. Budgeting knowledge is important 

for knowing how to properly allocate resources over time so that funds are available when needed. Because 

cost estimates are often needed to justify enhancing older systems, having an awareness of engineering, 

computer science, mathematics, and statistics will help identify cost drivers and the type of data needed 

to develop the estimate. It also helps for the cost estimator to have adequate technical knowledge when 

meeting with functional experts so that credibility and a common understanding of the technical aspects 

of the program can be quickly established. Finally, cost estimators who are able to “sell” and present their 
52 Chapter 6 GAO-09-3SP

estimate by defending it with solid facts and reliable data stand a better chance of its being used as a basis 

for program funding. In addition, cost estimators need to have solid interpersonal skills, because working 

and communicating with subject matter experts is vital for understanding program requirements.

Team Composition and Organization

Program office cost estimates are normally prepared by a multidisciplinary team whose members have 

functional skills in financial management, engineering, acquisition and logistics, scheduling, and 

mathematics, in addition to communications.26 The team should also include participants or reviewers 

from the program’s operating command, product support center, maintenance depot, and other units 

affected in a major way by the estimate.27 Team members might also be drawn from other organizations. 

In the best case, the estimating team is composed of persons who have experience in estimating all 

cost elements of the program. Since this is seldom possible, the team leader should be familiar with the 

team members’ capabilities and assign tasks accordingly. If some are experienced in several areas, while 

others are relatively inexperienced in all areas, the team leader should assign the experienced analysts 

responsibility for major sections of the estimate while the less experienced analysts work under their 

supervision. 

An analytic approach to cost estimates typically entails a written study plan detailing a master schedule 

of specific tasks, responsible parties, and due dates. For complex efforts, the estimating team might be 

organized as a formal, integrated product team. For independent estimates, the team might be smaller 

and less formal. In either case, the analysis should be coordinated with all stakeholders, and the study plan 

should reflect each team member’s responsibilities.

What is required of a cost estimating team depends on the type and purpose of the estimate and the 

quantity and quality of the data. More detailed estimates generally require larger teams, more time and 

effort, and more rigorous techniques. For example, a rough-order-of-magnitude estimate—a quick, highlevel cost estimate—generally requires less time and effort than a budget-quality estimate. In addition, the 

estimating team must be given adequate time to develop the estimate. Following the 12 steps takes time 

and cannot be rushed—rushing would significantly risk the quality of the results. 

One of the most time consuming steps in the cost estimating process is step 6: obtaining the data. Enough 

time should be scheduled to collect the data, including visiting contractor sites to further understand the 

strengths and limitations of the data that have been collected. If there is not enough time to develop the 

estimate, then the schedule constraint should be clearly identified in the ground rules and assumptions, so 

that management understands the effect on the estimate’s quality and confidence. 

26 Since schedules are the foundation of the performance plan, having a scheduling staff member integrated on the team is critical 

for validating the plan’s reasonableness. A scheduler can determine the feasibility of the network schedule by analyzing its durations. 

27An independent cost estimate for a major defense acquisition program under 10 U.S.C. § 2434 must be prepared by an office or 

other entity (such as the Office of the Secretary of Defense Cost Analysis Improvement Group) that is not under the supervision, 

direction, or control of the military department, defense agency, or other component directly responsible for carrying out the 

program’s development or acquisition. If the decision authority has been delegated to an official of the military department, defense 

agency, or other DOD component, then the estimate must be prepared by an office or other entity not directly responsible for 

carrying out the development or acquisition. 
GAO-09-3SP Chapter 6 53

Cost estimating requires good organizational skills, in order to pull together disparate data for each 

cost element and to package it in a meaningful way. It also requires engineering and mathematical 

skills, to fully understand the quality of the data available. Excellent communication skills are also 

important for clarifying the technical aspects of a program with technical specialists. If the program 

has no technical baseline description, or if the cost estimating team must develop one, it is essential that 

the team have access to the subject matter experts—program managers, system and software engineers, 

test and evaluation analysts—who are familiar with the program or a program like it. Moreover, team 

members need good communication skills to interact with these experts in ways that are meaningful and 

productive.

Cost Estimating Team Best Practices

Centralizing the cost estimating team and process—cost analysts working in one group but supporting 

many programs—represents a best practice, according to the experts we interviewed. Centralization 

facilitates the use of standardized processes, the identification of resident experts, a better sharing of 

resources, commonality and consistency of tools and training, more independence, and a career path with 

more opportunities for advancement. Centralizing cost estimators and other technical and business experts 

also allows for more effective deployment of technical and business skills while ensuring some measure of 

independence. 

A good example is in the Cost Analysis Improvement Group (CAIG) in the Office of the Secretary of 

Defense. Its cost estimates are produced by a centralized group of civilian government personnel to ensure 

long-term institutional knowledge and no bias toward results. Some in the cost estimating community 

consider a centralized cost department that provides cost support to multiple program offices, with a 

strong organizational structure and support from its leadership, to be a model. 

In contrast, decentralization often results in ad hoc processes, limited government resources (requiring 

contractor support to fill the gaps), and decreased independence, since program offices typically fund 

an effort and since program management personnel typically rate the analysts’ performance. The major 

advantage of a decentralized process is that analysts have better access to technical experts. Under a 

centralized process, analysts should thus make every effort to establish contacts with appropriate technical 

experts.

Finally, organizations that develop their own centralized cost estimating function but outside the 

acquiring program represent the best practice over organizations that develop their cost estimates in a 

decentralized or ad hoc manner under the direct control of a program office. One of the many benefits 

of centralized structure is the ability to resist pressure to lower the cost estimate when it is higher than the 

allotted budget. Furthermore, reliance on support contractors raises questions from the cost estimating 

community about whether numbers and qualifications of government personnel are sufficient to provide 

oversight of and insight into contractor cost estimates. Other experts in cost estimating suggested that 

reliance on support contractors can be a problem if the government cannot evaluate how good a cost 

estimate is or if the ability to track it is lacking. Studies have also raised the concern that relying on 

support contractors makes it more difficult to retain institutional knowledge and instill accountability. 

Therefore, to mitigate any bias in the cost estimate, government customers of contractor-produced cost 

estimates must have a high enough level of experience to determine whether the cost estimate conforms to 

the best practices outlined in this Guide. 
54 Chapter 6 GAO-09-3SP

Certification and Training for Cost Estimating and EVM Analysis

Since the experience and skills of the members of a cost estimating team are important, various 

organizations have established training programs and certification procedures. For example, SCEA’s 

certification program provides a professional credential to both members and nonmembers for education, 

training, and work experience and a written examination on basic concepts and methods for cost 

estimating. Another example is the earned value professional certification offered by the Association for 

the Advancement of Cost Engineering International that PMI’s College of Performance Management 

endorses; it requires candidates to have the requisite experience and the ability to pass a rigorous written 

exam.

Under the Defense Acquisition Workforce Improvement Act, DOD established a variety of certification 

programs through the Defense Acquisition University (DAU).28 DAU provides a full range of basic, 

intermediate, and advanced certification training; assignment-specific training; performance support; 

job-relevant applied research; and continuous learning opportunities. Although DAU’s primary mission is 

to train DOD employees, all federal employees are eligible to attend as space is available. One career field 

is in business, cost estimating, and financial management. Certification levels are based on education, 

experience, and training. Since this certification is available to all federal employees, it is considered a 

minimum training requirement for cost estimators. 

In addition to the mandatory courses in table 5, DAU encourages analysts to be trained in courses 

identified in its Core Plus Development Guide. These courses cover a wide range of cost estimating 

and earned value topics, such as acquisition reporting concepts and policy requirements, analysis of 

alternatives, baseline maintenance, basic software acquisition management, business case analysis, 

business management modernization, contract source selection, cost as an independent variable, economic 

analysis, EVM system validation and surveillance, integrated acquisition for decision makers, operating 

and support cost analysis, principles of schedule management, program management tools, and risk 

management. The standards for the business, cost estimating, and financial management levels of 

certification are shown in table 5.

Table 5: Certification Standards in Business, Cost Estimating, and Financial Management in the 

Defense Acquisition Education, Training, and Career Development Program

Level Education Experience Training

I Desired Baccalaureate 

Mandatory 1 year of acquisition in 

business, cost estimating, or 

financial management

ACQ 101: Fundamentals of Systems 

Acquisition Management 

and 2 of the following:

BCF 101: Fundamentals of Cost 

Analysis

BCF 102: Fundamentals of Earned 

Value

BCF 103: Fundamentals of Business 

Financial Management

28Defense Acquisition Workforce Improvement Act, codified at 10 U.S.C. ch. 87. 
GAO-09-3SP Chapter 6 55

Level Education Experience Training

II Desired Baccalaureate 2 additional years in business, 

cost estimating, or financial 

management

Mandatory 2 years of acquisition in 

business, cost estimating, or 

financial management

ACQ 201: (Parts A & B) 

Intermediate Systems 

Acquisition and

BCF 205: Contractor Business 

Strategies and, if not 

taken at Level I,

BCF 101: Fundamentals of Cost 

Analysis or

BCF 102: Fundamentals of Earned 

Value Management or

BCF 103: Fundamentals of Business 

Financial Management

and one of the following:

BCF 203: Intermediate Earned 

Value Management or

BCF 204: Intermediate Cost 

Analysis or 

BCF 211: Acquisition Business 

Management

III Desired Baccalaureate or 

24 semester hours 

among 10 coursesa

or Master’s

4 additional years of 

acquisition in business, 

cost estimating, or financial 

management

Mandatory BCF 301: Business, Cost 

Estimating, and Financial 

Management Workshop

Source: DAU.

a

The 10 courses are accounting, business finance, contracts, economics, industrial management, law, marketing, organization and 

management, purchasing, and quantitative methods.

When reviewing an agency’s cost estimate, an auditor should question the cost estimators about whether 

they have both the requisite formal training and substantial on-the-job training to develop cost estimates 

and keep those estimates updated with EVM analysis. Continuous learning by participating in cost 

estimating and EVM conferences is important for keeping abreast of the latest techniques and maximizing 

lessons learned. Agency cost estimators and EVM analysts, as well as GAO’s auditors, should attend such 

conferences to keep their skills current. Maintaining skills is essential if subject matter experts are to be 

relied on to apply best practices in their roles.

While formal training is important, so is on-the-job training and first-hand knowledge from participating 

in plant and site visits. On-site visits to see what is being developed and how engineering and 

manufacturing are executed are invaluable to cost estimators and auditors. To understand the complexity 

of the tasks necessary to deliver a product, site visits should always be included in the audit plan.

SEI’s Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software 

Organizations lists six requisites for reliable estimating and gives examples of evidence needed to satisfy 
56 Chapter 6 GAO-09-3SP

them. It also contains a checklist for estimating whether an organization provides its commitment and 

support to the estimators. SEI’s criteria are helpful for determining whether cost estimators have the skills 

and training to effectively develop credible cost estimates. (See appendix VIII for a link to SEI’s material.)

While much of this Cost Guide’s focus is on cost estimating, in chapter 18 we focus on EVM and how 

it follows the cost estimate through its various phases and determines where there are cost and schedule 

variances and why. This information is vitally important to keeping the estimate updated and for keeping 

abreast of program risks. Because of performance measurement requirements (including the use of 

EVM), OMB issued policy guidance in August 2005 to agency chief information officers on improving 

information technology projects. OMB stated that the Federal Acquisition Institute (co-located with 

DAU) was expanding EVM system training to the program management and contracting communities 

and instructed agencies to refer to DAU’s Web site for a community of practice that includes the following 

resources:29

■ 6 hours of narrated EVM tutorials (Training Center), 

■ descriptions and links to EVM tools (Tools), 

■ additional EVM-related references and guides (Community Connection), 

■ DOD policy and contracting guidance (Contract Documents and DOD Policy and Guidance), 

■ a discussion forum (Note Board), and 

■ an on-line reference library (Research Library).

Such resources are important for agencies and auditors in understanding what an EVM system can offer 

for improving program management.

3. Best Practices Checklist: Cost Assessment Team

 The estimating team’s composition is commensurate with the assignment 

(see SEI’s checklists for more details).

3 The team has the proper number and mix of resources.

3 Team members are from a centralized cost estimating organization.

3 The team includes experienced and trained cost analysts.

3 The team includes, or has direct access to, analysts experienced in the 

program’s major areas. 

3 Team members’ responsibilities are clearly defined.

3 Team members’ experience, qualifications, certifications, and training 

are identified. 

3 The team participated in on-the-job training, including plant and site 

visits. 

 A master schedule with a written study plan has been developed.

 The team has access to the necessary subject matter experts.

29DAU’s Web site is at https://acc.dau.mil/evm.
GAO-09-3SP Chapter 7 57

Technical Baseline Description Definition 

and Purpose

Chapter 7

Key to developing a credible estimate is having an adequate understanding of the acquisition program—

the acquisition strategy, technical definition, characteristics, system design features, and technologies 

to be included in its design. The cost estimator can use this information to identify the technical and 

program parameters that will bind the cost estimate. The amount of information gathered directly affects 

the overall quality and flexibility of the estimate. Less information means more assumptions must be 

made, increasing the risk associated with the estimate. Therefore, the importance of this step must be 

emphasized, because the final accuracy of the cost estimate depends on how well the program is defined. 

The objective of the technical baseline is to provide in a single document a common definition of the 

program—including a detailed technical, program, and schedule description of the system—from which 

all LCCEs will be derived—that is, program and independent cost estimates. At times, the information 

in the technical baseline will drive or facilitate the use of a particular estimating approach. However, the 

technical baseline should be flexible enough to accommodate a variety of estimating methodologies. It is 

also critical that the technical baseline contain no cost data, so that it can be used as the common baseline 

for independently developed estimates.30

In addition to providing a comprehensive program description, the technical baseline is used to 

benchmark life-cycle costs and identify specific technical and program risks. In this way, it helps the 

estimator focus on areas or issues that could have a major cost effect.

Process

In general, program offices are responsible for developing and maintaining the technical baseline 

throughout the life cycle, since they know the most about their program. A best practice is to assign an 

integrated team of various experts—system engineers, design experts, schedulers, test and evaluation 

experts, financial managers, and cost estimators—to develop the technical baseline at the beginning of 

the project. The program manager and the senior executive oversight committee approve the technical 

baseline to ensure that it contains all information necessary to define the program’s systems and develop 

the cost estimate. 

Furthermore, the technical baseline should be updated in preparation for program reviews, milestone 

decisions, and major program changes. The credibility of the cost estimate will suffer if the technical 

baseline is not maintained. Without explicit documentation of the basis of a program’s estimates, 

it is difficult to update the cost estimate and provide a verifiable trace to a new cost baseline as key 

assumptions change during the course of the program’s life.

30As used in this Cost Guide, the technical baseline is similar to DOD’s Cost Analysis Requirements Description (CARD) and 

NASA’s Cost Analysis Data Requirement (CADRE). 
58 Chapter 7 GAO-09-3SP

It is normal and expected that early program technical baselines will be imprecise or incomplete and that 

they will evolve as more information becomes known. However, it is essential that the technical baseline 

provide the best available information at any point in time. To try to create an inclusive view of the 

program, assumptions should be made about the unknowns and should be agreed on by management. 

These assumptions and their corresponding justifications should be documented in the technical baseline, 

so their risks are known from the beginning.

Schedule

The technical baseline must be available in time for all cost estimating activities to proceed on schedule. 

This often means that it is submitted as a draft before being made final. The necessary lead time will 

vary by organization. One example is the CAIG in the Office of the Secretary of Defense, which requires 

that the Cost Analysis Requirements Description be submitted in draft 180 days before the Defense 

Acquisition Board milestone and in final form 45 days before the milestone review.

Contents

Since the technical baseline is intended to serve as the baseline for developing LCCEs, it must provide 

information on development, testing, procurement, installation and replacement, operations and 

support, planned upgrades, and disposal. In general, a separate technical baseline should be prepared for 

each alternative; as the program matures, the number of alternatives and, therefore, technical baselines 

decreases. Although technical baseline content varies by program (and possibly even by alternative), it 

always entails a number of sections, each focusing on a particular aspect of the program being assessed. 

Table 6 describes typical technical baseline elements.

Table 6: Typical Technical Baseline Elements 

Element Description

System purpose Describes the system’s mission and how it fits into the program; should give the 

estimator a concept of its complexity and cost

Detailed technical 

system and performance 

characteristics

Includes key functional requirements and performance characteristics; the 

replaced system (if applicable); who will develop, operate, and maintain 

the system; descriptions of hardware and software components (including 

interactions, technical maturity of critical components, and standards); system 

architecture and equipment configurations (including how the program will 

interface with other systems); key performance parameters; information assurance; 

operational concept; reliability analysis; security and safety requirements; test and 

evaluation concepts and plans

Work breakdown 

structure 

Identifies the cost and technical data needed to develop the estimate 

Description of legacy or 

similar systems

A legacy (or heritage or predecessor) system has characteristics similar to the 

system being estimated; often the new program is replacing it. The technical 

baseline includes a detailed description of the legacy hardware and software 

components; technical protocols or standards; key performance parameters; 

operational and maintenance logistics plan; training plan; phase-out plan; and the 

justification for replacing the system
GAO-09-3SP Chapter 7 59

Element Description

Acquisition plan or 

strategy

Includes the competition strategy, whether multiyear procurement will be 

used, and whether the program will lease or buy certain items; it should identify 

the type of contract awarded or to be awarded and, if known, the contractor 

responsible for developing and implementing the system

Development, test, and 

production quantities and 

program schedule

Includes quantities required for development, test (e.g., test assets), and 

production; lays out an overall development and production schedule that 

identifies the years of its phases—the schedule should include a standard Gantt 

chart with major events such as milestone reviews, design reviews, and major 

tests—and that addresses, at a high level, major program activities, their duration 

and sequence, and the critical path 

System test and 

evaluation plan

Includes the number of tests and test assets, criteria for entering into testing, exit 

criteria for passing the test, and where the test will be conducted

Deployment details Includes standard platform and site configurations for all scenarios (peacetime, 

contingency, war) and a transition plan between legacy and new systems

Safety plan Includes any special or unique system safety considerations that may relate to 

specific safety goals established through standards, laws, regulations, and lessons 

learned from similar systems

Training plan Includes training for users and maintenance personnel, any special certifications 

required, who will provide the training, where it will be held, and how often it will 

be offered or required

Disposal and 

environmental effect

Includes identification of environment impact, mitigation plan, and disposal 

concept

Operational concept Includes program management details, such as how, where, and when the system 

will be operated; the platforms on which it will be installed; and the installation 

schedule

Personnel requirements Includes comparisons to the legacy system (if possible) in salary levels, skill-level 

quantity requirements, and where staff will be housed

Logistics support details Includes maintenance and sparing plans, as well as planned upgrades

Changes from the 

previous technical 

baseline

Includes a tracking of changes, with a summary of what changed and why

Source: DOD, DOE, and SCEA.

Programs following an incremental development approach should have a technical baseline that clearly 

states system characteristics for the entire program. In addition, the technical baseline should define the 

characteristics to be included in each increment, so that a rigorous LCCE can be developed. For programs 

with a spiral development approach, the technical baseline tends to evolve as requirements become 

better defined. In earlier versions of a spiral development program, the technical baseline should clearly 

state the requirements that are included and those that have been excluded. This is important, since a 

lack of defined requirements can lead to cost increases and delays in delivering services, as case study 20

illustrates. 
60 Chapter 7 GAO-09-3SP

Case Study 20: Defining Requirement, from United States Coast Guard, 

GAO-06-623

The U.S. Coast Guard contracted in September 2002 to replace its search and rescue 

communications system, installed in the 1970s, with a new system known as Rescue 21. 

The acquisition and initial implementation of Rescue 21, however, resulted in significant 

cost overruns and schedule delays. By 2005, its estimated total acquisition cost had 

increased to $710.5 million from 1999’s $250 million, and the schedule for achieving full 

operating capability had been delayed from 2006 to 2011. GAO reported in May 2006 on 

key factors contributing to the cost overruns and schedule delays, including requirements 

management. Specifically, GAO found that the Coast Guard did not have a rigorous 

requirements management process. 

Although the Coast Guard had developed high-level requirements, it relied solely on the 

contractor to manage them. According to Coast Guard acquisition officials, they had taken 

this approach because of the performance-based contract vehicle. GAO’s experience 

in reviewing major systems acquisitions has shown that it is important for government 

organizations to exercise strong leadership in managing requirements, regardless of the 

contracting vehicle. 

Besides not effectively managing requirements, Rescue 21 testing revealed numerous 

problems linked to incomplete and poorly defined user requirements. For example, a 

Coast Guard usability and operability assessment of Rescue 21 stated that most of the 

operational advancements envisioned for the system had not been achieved, concluding 

that these problems could have been avoided if the contract had contained user 

requirements.

A key requirement was to “provide a consolidated regional geographic display.” The 

contractor provided a capability based on this requirement but, during testing, the 

Coast Guard operators believed that the maps did not display sufficient detail. Such 

discrepancies led to an additional statement of work that defined required enhancements 

to the system interface, such as screen displays.

GAO reported that if deploying Rescue 21 were to be further delayed, Coast Guard sites 

and services would be affected in several ways. Key functionality, such as improved 

direction finding and improved coverage of coastal areas, would not be available as 

planned. Coast Guard personnel at those sites would continue to use outdated legacy 

communications systems for search and rescue operations, and coverage of coastal 

regions would remain limited. In addition, delays could result in costly upgrades to 

the legacy system in order to address communications coverage gaps, as well as other 

operational concerns.

GAO, United States Coast Guard: Improvements Needed in Management and 

Oversight of Rescue System Acquisition, GAO-06-623 (Washington, D.C.: May 

31, 2006).

Fully understanding requirements up front helps increase the accuracy of the cost estimate. While each 

program should have a technical baseline that addresses each element in table 6, each program’s aspects 

are unique. In the next section, we give examples of system characteristics and performance parameters 

typically found in government cost estimates, including military weapon systems and civilian construction 

and information systems.
GAO-09-3SP Chapter 7 61

Key System Characteristics and Performance Parameters

Since systems differ, each one has unique physical and performance characteristics. Analysts need specific 

knowledge about them before they can develop a cost estimate for a weapon system, an information 

system, or a construction program. 

While the specific physical and performance characteristics for a system being estimated will be dictated 

by the system and the methodology used to perform the estimate, several general characteristics have been 

identified in the various guides we reviewed. Table 7 lists general characteristics shared within several 

system types.

Table 7: General System Characteristics

System Characteristic Type

Aircraft Breakdown of airframe 

unit weight 

by material type

Combat ceiling and speed

Internal fuel capacity

Length

Load factor

Maximum altitude 

Maximum speed (knots at 

sea level)

Mission and profile

Weight Airframe unit weight, combat, empty, maximum gross, 

payload, structure

Wetted area

Wing Wingspan, wing area, wing loading

Automated 

information 

systems 

Architecture

Commercial off-the-shelf 

software used 

Customization of 

commercial off-the-shelf 

software

Expansion factors

Memory size

Processor type 

Proficiency of 

programmers

Programming language 

used

Software sizing metric
62 Chapter 7 GAO-09-3SP

System Characteristic Type

Construction Changeover

Environmental impact

Geography

Geology

Liability

Location Land value, proximity to major roads, relocation expenses 

Material type Composite, masonry, metal, tile, wood shake 

Number of stories

Permits

Public acceptance

Square feet

Systemization

Missiles Height

Length 

Payload

Propulsion type

Range

Sensors

Weight

Width 

Ships Acoustic signature

Full displacement 

Full load weight 

Length overall 

Lift capacity

Light ship weight

Margin

Maximum beam 

Number of screws 

Payload

Propulsion type

Shaft horsepower 

Space Attitude

Design life and reliability

Launch vehicle

Mission and duration

Orbit type

Pointing accuracy

Satellite type

Thrust

Weight and volume
GAO-09-3SP Chapter 7 63

System Characteristic Type

Tanks and trucks Engine

Height

Horsepower

Length

Weight

Width

Payload

Source: DOD and GAO.

Once a system’s unique requirements have been defined, they must be managed and tracked continually 

throughout the program’s development. If requirements change, both the technical baseline and cost 

estimate should be updated so that users and management can understand the effects of the change. 

When requirements are not well managed, users tend to become disillusioned, and costs and schedules 

can spin out of control, as case study 21 demonstrates. 

Case Study 21: Managing Requirements, from DOD Systems 

Modernization, GAO-06-215

The Naval Tactical Command Support System (NTCSS) was started in 1995 to help U.S. 

Navy personnel manage ship, submarine, and aircraft support activities. At the time of 

GAO’s review, about $1 billion had been spent to partially deploy NTCSS to about half 

its intended sites. In December 2005, GAO reported that the Navy had not adequately 

conducted requirements management and testing activities for the system. For example, 

requirements had not been prioritized or traced to related documentation to ensure that 

the system’s capabilities would meet users’ needs. As a result, failures in developmental 

testing had prevented NTCSS’s latest component from passing operational testing twice 

over the preceding 4 years. From the Navy’s data, the recent trend in key indicators 

of system maturity, such as the number and nature of reported system problems and 

change proposals, showed that problems with NTCSS had persisted and that they could 

involve costly rework. In addition, the Navy did not know the extent to which NTCSS’s 

optimized applications were meeting expectations—even though the applications had 

been deployed to 229 user sites since 1998—because metrics to demonstrate that the 

expectations had been met had not been defined and collected.

GAO, DOD Systems Modernization: Planned Investment in the Naval 

Tactical Command Support System Needs to Be Reassessed, GAO-06-215

(Washington, D.C.: Dec. 5, 2005).

Case study 21 shows that an inability to manage requirements leads to additional costs and inefficient 

management of resources. To manage requirements, they must first be identified. The bottom line is that the 

technical baseline should document the underlying technical and program assumptions necessary to develop 

a cost estimate and update changes as they occur. Moreover, the technical baseline should also identify 

the level of risk associated with the assumptions so that the estimate’s credibility can be determined. As we 

stated previously, the technical baseline should mature in the same manner as the program evolves. Because 

it is evolutionary, earlier versions of the technical baseline will necessarily include more assumptions and, 

therefore, more risk, but these should decline as risks become either realized or retired.
64 Chapter 7 GAO-09-3SP

4. Best Practices Checklist: Technical Baseline Description

 There is a technical baseline:

3 The technical baseline has been developed by qualified personnel such 

as system engineers.

3 It has been updated with technical, program, and schedule changes, 

and it contains sufficient detail of the best available information at any 

given time.

3 The information in the technical baseline generally drives the cost 

estimate and the cost estimating methodology.

3 The cost estimate is based on information in the technical baseline and 

has been approved by management.

 The technical baseline answers the following:

3 What the program is supposed to do—requirements;

3 How the program will fulfill its mission—purpose;

3 What it will look like—technical characteristics;

3 Where and how the program will be built—development plan;

3 How the program will be acquired—acquisition strategy;

3 How the program will operate—operational plan;

3 Which characteristics affect cost the most—risk.
GAO-09-3SP Chapter 8 65

Work Breakdown Structure

A work breakdown structure is the cornerstone of every program because it defines in detail the work 

necessary to accomplish a program’s objectives. For example, a typical WBS reflects the requirements, 

what must be accomplished to develop a program, and provides a basis for identifying resources and tasks 

for developing a program cost estimate. A WBS is also a valuable communication tool between systems 

engineering, program management, and other functional organizations because it provides a clear picture 

of what needs to be accomplished and how the work will be done. Accordingly, it is an essential element 

for identifying activities in a program’s integrated master schedule. In addition, it provides a consistent 

framework for planning and assigning responsibility for the work. Initially set up when the program is 

established, the WBS becomes successively detailed over time as more information becomes known about 

the program.

A WBS is a necessary program management tool because it provides a basic framework for a variety of 

related activities like estimating costs, developing schedules, identifying resources, determining where 

risks may occur, and providing the means for measuring program status using EVM. Furthermore, a well 

structured WBS helps promote accountability by identifying work products that are independent of one 

another. It also provides the framework to develop a schedule and cost plan that can easily track technical 

accomplishments—in terms of resources spent in relation to the plan as well as completion of activities 

and tasks—enabling quick identification of cost and schedule variances. 

Best Practice: Product-Oriented WBS

A WBS deconstructs a program’s end product into successive levels with smaller specific elements until 

the work is subdivided to a level suitable for management control. By breaking work down into smaller 

elements, management can more easily plan and schedule the program’s activities and assign responsibility 

for the work. It also facilitates establishing a schedule, cost, and EVM baseline. Establishing a productoriented WBS is a best practice because it allows a program to track cost and schedule by defined 

deliverables, such as a hardware or software component. This allows a program manager to more precisely 

identify which components are causing cost or schedule overruns and to more effectively mitigate the root 

cause of the overruns.

A WBS breaks down product-oriented elements into a hierarchical structure that shows how elements 

relate to one another as well as to the overall end product. A 100 percent rule is followed that states that 

“the next level of decomposition of a WBS element (child level) must represent 100 percent of the work 

applicable to the next higher (parent) element.”31 This is considered a best practice by many experts in cost 

31Gregory T. Haugan, Work Breakdown Structures for Projects, Programs, and Enterprises (Vienna, Va.: Management Concepts, 

2008), p. 38.

Chapter 8
66 Chapter 8 GAO-09-3SP

estimating, because a product-oriented WBS following the 100 percent rule ensures that all costs for all 

deliverables are identified. Failing to include all work for all deliverables can lead to schedule delays and 

subsequent cost increases. It can also result in confusion among team members. To avoid these problems, 

standardizing the WBS is a best practice in organizations where there is a set of program types that are 

standard and typical. This enables an organization to simplify the development of the top-level program 

work breakdown structures by publishing the standard. It also facilitates an organization’s ability to collect 

and share data from common WBS elements among many programs. The more data that are available for 

creating the cost estimate, the higher the confidence level will be. 

Its hierarchical nature allows the WBS to logically sum the lower-level elements that support the 

measuring of cost, schedule, and technical analysis in an EVM system. A good WBS clearly defines the 

logical relationship of all program elements and provides a systematic and standardized way for collecting 

data across all programs. Therefore, a WBS is an essential part of developing a program’s cost estimate and 

enhancing an agency’s ability to collect data necessary to support future cost estimates. Moreover, when 

appropriately integrated with systems engineering, cost estimating, EVM, and risk management, a WBS 

provides the basis to allow program managers to have a better view into a program’s status, facilitating 

continual improvement. 

A WBS is developed and maintained by a systems engineering process that produces a product-oriented 

family tree of hardware, software, services, data, and facilities. It can be thought of as an illustration of 

what work will be accomplished to satisfy a program’s requirements. The WBS diagrams the effort in 

small discrete pieces, or elements, to show how each one relates to the others and to the program as a 

whole. These elements such as hardware, software, and data are further broken down into specific lowerlevel elements. The lowest level of the WBS is defined as the work package level. 

The number of levels for a WBS varies from program to program and depends on a program’s complexity 

and risk. Work breakdown structures need to be expanded to a level of detail that is sufficient for planning 

and successfully managing the full scope of work. However, each WBS should, at the very least, include 

three levels. The first level represents the program as a whole and therefore contains only one element—

the program’s name. The second level contains the major program segments, and level three contains the 

lower-level components or subsystems for each segment. These relationships are illustrated in figure 10, 

which depicts a very simple automobile system WBS.

Figure 10: A Product-Oriented Work Breakdown Structure

Interior

Automobile

system

Chassis Shell

Subcomponent Subcomponent Subcomponent

Level 1

Level 2

Level 3

Exterior Powertrain

Source: © 2005 MCR, LLC, “Developing a Work Breakdown Structure.”
GAO-09-3SP Chapter 8 67

In figure 10, all level 2 elements would also have level 3 subcomponents; chassis is the example in the 

figure. For some level 2 elements, level 3 would be the lowest level of breakdown; for others, still lower 

levels would be required. The elements at each lower level of breakdown are called “children” of the next 

higher level, which are the “parents.” The parent–child relationship allows for logical connections and 

relationships to emerge and a better understanding of the technical effort involved. It also helps improve 

the ability to trace relationships within the cost estimate and EVM system. 

In the example in figure 10, the chassis would be a child of the automobile system but the parent of 

subcomponents 1–3. In constructing a WBS, the 100 percent rule always applies. That is, the sum of a 

parent’s children must always equal the parent. Thus, in figure 10, the sum of chassis, shell, interior, and so 

on must equal the automobile system. In this way, the WBS makes sure that each element is defined and 

related to only one work effort, so that all activities are included and accounted for. It also helps identify the 

specialists who are needed to complete the work and who will be responsible so that effort is not duplicated. 

It is important to note that a product-oriented WBS reflects cost, schedule, and technical performance on 

specific portions of a program, while a functional WBS does not provide that level of detail. For example, 

an overrun on a specific item in figure 10 (for example, powertrain) might cause program management 

to change a specification, shift funds, or modify the design. If the WBS were functionally based (for 

example, in manufacturing, engineering, or quality control), then management would not have the right 

information to get to the root cause of the problem. Therefore, since only a product-oriented WBS relates 

costs to specific hardware elements—the basis of most cost estimates—it represents a cost estimating best 

practice. Case study 22 highlights problems that can occur by not following this best practice.

Case Study 22: Product-Oriented Work Breakdown Structure, from Air 

Traffic Control, GAO-08-756

Federal Aviation Administration (FAA) required the use of EVM on its major information 

technology investments. GAO found key components not fully consistent with best practices. 

We reported that leading organizations establish EVM policies that require programs to use 

a product-oriented structure for defining work products. FAA’s policy and guidance are not 

consistent with best practices because it requires its programs to establish a standard WBS 

using a function-oriented structure. FAA work is thus delineated by functional activities, such 

as design engineering, requirements analysis, and quality control. A product-oriented WBS 

would reflect cost, schedule, and technical performance on specific deliverables. 

Without a product-oriented approach, program managers may not have the detailed 

information needed to make decisions on specific program components. For example, 

cost overruns associated with a specific radar component could be quickly identified and 

addressed using a product-oriented structure. If a function-oriented structure were used, 

these costs could be spread out over design, engineering, etc. 

FAA program managers using a product-oriented WBS need to transfer their data to FAA’s 

required function-oriented WBS when reporting to management. EVM experts agree that 

such mapping efforts are time-consuming, subject to error, and not always consistent. Until 

FAA establishes a standard product-oriented WBS, program officials may not be obtaining 

the information they need. 

GAO, Air Traffic Control: FAA Uses Earned Value Techniques to Help Manage 

Information Technology Acquisitions, but Needs to Clarify Policy and 

Strengthen Oversight, GAO-08-756 (Washington, D.C.: July 18, 2008).
68 Chapter 8 GAO-09-3SP

Since best practice is for the WBS prime mission elements to be product-oriented, the WBS should not be 

structured or organized at a second or third level according to any element not a product or not being in or 

itself a deliverable:

■ design engineering, requirements analysis, logistics, risk, quality assurance, and test engineering 

(all functional engineering efforts), aluminum stock (a material resource), and direct costs (an 

accounting classification);32

■ program acquisition phases (for example, development and procurement) and types of funds used 

in those phases (for example, research, development, test, and evaluation);

■ rework, retesting, and refurbishing, which should be treated as activities of the WBS element;

■ nonrecurring and recurring classifications, for which reporting requirements should be structured 

to ensure that they are segregated;

■ cost saving efforts—such as total quality management initiatives and acquisition reform 

initiatives—included in the elements they affect, not captured separately;

■ the organizational structure of the program office or contractor; 

■ the program schedule—instead the WBS will drive the necessary schedule activities;

■ meetings, travel, and computer support, which should be included in the WBS elements they are 

associated with;

■ generic terms (terms for WBS elements should be as specific as possible); and

■ tooling, which should be included with the equipment being produced.

While functional activities are necessary for supporting a product’s development, the WBS should not be 

organized around them. Only products should drive the WBS, not common support activities. Moreover, 

the WBS dictionary should state where the functional elements fall within the products and how the 

statement of work elements come together to make specific products. 

Common WBS Elements

In addition to including product-oriented elements, every WBS includes program management as a level 2 

element and other common elements like integration and assembly, government furnished equipment, and 

government testing. Table 8 lists and describes common elements that support the program. For instance, 

systems engineering, program management, integration, and testing are necessary support functions for 

developing, testing, producing, and fielding hardware or software elements. 

Table 8: Common Elements in Work Breakdown Structures

Common element Description

Integration, assembly, test, 

and checkout

All effort of technical and functional activities associated with the design, 

development, and production of mating surfaces, structures, equipment, parts, 

materials, and software required to assemble level 3 equipment (hardware and 

software) elements into level 2 mission equipment (hardware and software)

32When following the product-oriented best practice, there should not be WBS elements for various functional activities like design 

engineering, logistics, risk, or quality, because these efforts should be embedded in each activity.
GAO-09-3SP Chapter 8 69

Common element Description

System engineering The technical and management efforts of directing and controlling a totally 

integrated engineering effort of a system or program

Program management The business and administrative planning, organizing, directing, coordinating, 

controlling, and approval actions designated to accomplish overall program 

objectives not associated with specific hardware elements and not included in 

systems engineering

Training Deliverable training services, devices, accessories, aids, equipment, and parts 

used to facilitate instruction in which personnel will learn to operate and 

maintain the system with maximum efficiency

Data The deliverable data that must be on a contract data requirements list, 

including technical publications, engineering data, support data, and 

management data needed to configure management, cost, schedule, 

contractual data management, and program management

System test and evaluation The use of prototype, production, or specifically fabricated hardware and 

software to obtain or validate engineering data on the performance of the 

system in developing program (in DOD, normally funded from research, 

development, test, and evaluation appropriations); also includes all effort 

associated with design and production of models, specimens, fixtures, and 

instrumentation in support of the system-level test program

Peculiar support equipment Equipment uniquely needed to support the program: vehicles, equipment, 

tools, and the like to fuel, service, transport, hoist, repair, overhaul, assemble 

and disassemble, test, inspect, or otherwise maintain mission equipment, as 

well as equipment or software required to maintain or modify the software 

portions of the system 

Common support 

equipment

Equipment not unique to the program and available in inventory for use by 

many programs

Operational and site 

activation

Installation of mission and support equipment in the operations or support 

facilities and complete system checkout or shakedown to ensure operational 

status; may include real estate, construction, conversion, utilities, and 

equipment to provide all facilities needed to house, service, and launch prime 

mission equipment

Facilities Includes construction, conversion, or expansion of existing industrial facilities 

for production, inventory, and contractor depot maintenance required as a 

result of the specific system

Initial spares and repair parts Includes the deliverable spare components, assemblies, and subassemblies 

used for initial replacement purposes in the materiel system equipment end 

item

Source: DOD.

Therefore, in addition to having a product-oriented WBS for the prime mission equipment that breaks 

down the physical pieces of, for example, an aircraft, information technology system, or satellite, the WBS 

should include these common elements to ensure that all effort is identified at the outset. This, in turn, 

will facilitate planning and managing the overall effort, since the WBS should be the starting point for 

developing the detailed schedule. Figure 11 shows a program WBS, including common elements, for an 

aircraft system.
70 Chapter 8 GAO-09-3SP

Figure 11: A Work Breakdown Structure with Common Elements

Level 1

Level 2

Level 3

Source: © 2005 MCR, LLC, “Developing a Work Breakdown Structure.”

System test

and

evaluation

Aircraft

system

System engineering/

Program management

Air

vehicle

Airframe Propulsion Fire control

Data Training

While the top-level WBS encompasses the whole program, the contractor must also develop a contract 

WBS that extends the lower-level components to reflect its responsibilities. See figure 12.

Figure 12: A Contract Work Breakdown Structure

 

);$,5&5$)7

$,59(+,&/(

$,5)5$0(

35238/6,216.3:'

&20081,&$7,216,'(17,),&$7,21

1$9,*$7,21*8,'$1&(

),5(&21752/

5$'$5

5(&(,9(5

75$160,77(5

$17(11$

5$'$5$33/,&$7,2166:72&6&,/(9(/

5$'$5,17(*$66(0%/<7(67$1'&+.287

12,7$8/$9('1$76(70(76<6 

7$8/$9('1$76(771(032/(9(' 12, /25712&(5,)

$5(32 7, 12,7$8/$9('1$76(7/$12 5$'$5

&20 638. 5(9,(&(5

752338612,7$8/$9('1$76(7 5(77,061$57

6(,7,/,&$)76(7 $11(71$

71(0(*$1$00$5*253*1,5((1,*1(60(76<6 27:6612,7$&,/33$5$'$5 /(9(/,&6&

((1,*1(60(76<6 *1,5 '/,8%

71(0(*$1$00$5*253 Q«,&6&

76,*2/12,7,6,84&$ 6&, &27,&6& 782.+&'1$*(71,,&6

71(03,84(75233865$,/8&(3 Q«'/,8%

(586$(0'1$76(7 71(03,84(71(0 Q«,&6&

6 71(03,84(*1,/'1$+'1$752338 782.+&'1$*(71,,&6&27,&6&

71(03,84(752338612002& 2$&,/33$5$'$5 %066$*(71,:6617,   7  &  76( 782.+

75$,1,1* 5$'$56<67(06:

65(1,$57(&1$1(71,$0 ,8% '/

(&,9('*1,1,$57:(5&5,$ Q«,&6&

6/$,5(7$0(6582&*1,1,$57 &27,&6& 782.+&'1$*(71,,&6

'$7$ %8,/'«Q

612,7$&,/%83/$&,1+&(7 Q«,&6&

(1*,1((5,1*'$7$ &6&,72&6&,,17(*$1'&+.287

$7$'71(0(*$1$0 </%0(66$*(71,:60(76<65$'$5   6(7 7 &  782.+

$7$'7523386 6$*(71,5$'$5 782.+&'1$76(7</%0(6

<527,623('$7$' 12,7$5*(71,052)7$/3

12,7$9,7&$(7,6612,7$5(32 $5*253*1,5((1,*1(0(76<6 71(0(*$1$00

7523386/$&,1+&(7527&$5712& 12,7$8/$9('1$76(70(76<6

675$35,$3(5'1$6(5$36/$,7,1, *1,1,$57

'$7$

3(&8/,$56833257(48,30(17

&200216833257(48,30(17

,1,7,$/63$5(6$1'5(3$,53$576

352*5$0:%6

&2175$&7:%6

Source: DOD.
GAO-09-3SP Chapter 8 71

Figure 12 shows how a prime contractor may require its subcontractor to use the WBS to report work 

progress. In this example, the fire control effort (a level 3 element in the prime contractor’s WBS) is 

the first level for the subcontractor. Thus, all fire control expenditures at level 1 of the subcontractor’s 

contract WBS would map to the fire control element at level 3 in the program WBS. This shows how 

a subcontractor would break a level 3 item down to lower levels to accomplish the work, which when 

rolled up to the prime WBS, would show effort at levels 4–7. Always keep in mind that the structure 

provided by the prime contractor WBS will identify the work packages that are the responsibility of the 

subcontractor. The subcontractor will also need to decompose the work further in its own WBS as well.

WBS Development

A WBS should be developed early to provide for a conceptual idea of program size and scope. As the 

program matures, so should the WBS. Like the technical baseline, the WBS should be considered a living 

document. Therefore, as the technical baseline becomes further defined with time, the WBS will also 

reflect more detail. For example, as specification requirements become better known and the statement 

of work is updated, the WBS will include more elements. As more elements are added to the WBS, 

the schedule is capable of greater definition, giving more insight into the program’s cost, schedule, and 

technical relationships.

It is important that each WBS be accompanied by a dictionary of the various WBS elements and their 

hierarchical relationships. A WBS dictionary is simply a document that describes in brief narrative format 

what work is to be performed in each WBS element. Each element is presented in an outline to show 

how it relates to the next higher element and what is included to ensure clear relationships. With minor 

changes and additions the WBS dictionary can be converted into a statement of work. Although not the 

normal approach, the dictionary may also be expanded by the program manager to describe the resources 

and processes necessary for producing each element in cost, technical, and schedule terms. Also, since the 

WBS is product related, it is closely related to, and structured somewhat the same as, an indented bill of 

materials for the primary product. Like the WBS, its dictionary should be updated when changes occur. 

After the program is baselined, updating the WBS should be part of a formal process, as in configuration 

management.

Standardized WBS

Standardizing the WBS is considered a best practice because it enables an organization to collect and 

share data among programs. Standardizing work breakdown structures results in more consistent cost 

estimates, allows data to be shared across organizations, and leads to more efficient program execution. 

WBS standardization also facilitates cost estimating relationship development and allows for common 

cost measures across multiple contractors and programs. Not standardizing WBSs causes extreme 

difficulty in comparing costs from one contractor or program to another, resulting in substantial expense 

to government estimating agencies when collecting and reconciling contractor cost and technical data in 

consistent format. 

The standardized WBS logic should support the engineering perspective on how the program is being 

built. The WBS should be a communication tool that can be used across all functions within the 

program. To foster flexibility, WBS standardization should occur at a high level—such as WBS level 

3—so that lower levels can be customized to reflect how the specific program’s work will be managed. For 
72 Chapter 8 GAO-09-3SP

high-risk or costly elements, however, management can make decisions to standardize the WBS to whatever 

level is necessary to properly gain insight. Thus, the WBS should be standard at a high level, with flexibility 

in the lower levels to allow detailed planning once the schedule is laid out. Furthermore, the same standard 

WBS should be used for developing the cost estimate and the program schedule and setting up the EVM 

performance measurement baseline. Relying on a standard WBS can enable program managers to better 

plan and manage their work and helps in updating the cost estimate with actual costs—the final critical step 

in our 12 steps to a high-quality cost estimate. 

A standardized product-oriented WBS can help define high-level milestones and cost driver relationships 

that can be repeated in future applications. In addition to helping the cost community, standard WBSs can 

result in better portfolio management. Programs reporting to a standard WBS enable leadership to make 

better decisions about where to apply contingency reserve and where systemic problems are occurring, like 

integration and test. Using this information, management can take action by adjusting investment and 

obtaining lessons learned. As a result, it is easier to manage programs if they are reporting in the same format.

Besides the common elements shown in table 8, DOD has identified, for each defense system, a standard 

combination of hardware and software that defines the end product for that system. In its 2005 updated 

WBS handbook, DOD defined and described the WBS, provided instructions on how to develop one, 

and defined specific defense items.33 The primary purpose of the handbook is to develop the top levels 

of the WBS with uniform definitions and a consistent approach. Developed through the cooperation of 

the military services, with assistance from industry associations, its benefit is improved communication 

throughout the acquisition process.

In addition to defining a standard WBS for its weapon systems, DOD has developed a common cost 

element structure that, while not a product-oriented WBS, standardizes the vocabulary for cost elements 

for automated information systems undergoing DOD review.34 The cost element structure is also designed 

to standardize the systems, facilitating the validation process. Furthermore, DOD requires that all the cost 

elements be included in LCCEs for automated information systems submitted for review. Table 9 gives an 

example of the cost element structure for an automated information system. 

Table 9: Cost Element Structure for a Standard DOD Automated Information System

Element 1 and subelements Element 2 and subelements Element 3 and subelements

1.0 Investment 2.0 System operations & 

support 3.0 Legacy system phase-out

1.1 Program management 2.1 System management 3.1 System management 

1.1.1 Personnel 2.1.1 Personnel 3.1.1 Personnel

1.1.2 Travel 2.1.2 Travel 3.1.2 Travel

1.1.3 Other government 

support

2.1.3 Other government 

support

3.1.3 Other government 

support

1.1.4 Other 2.1.4 Other

33DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Material Items, MIL-HDBK-881A (Washington, 

D.C.: July 30, 2005).

34Ronald C. Wilson, Department of Defense Automated Information Systems Economic Analysis Guide (Washington, D.C.: 

Department of Defense, May 1, 1995), att. B, pp. 39–75, Cost Element Structure Definitions.
GAO-09-3SP Chapter 8 73

Element 1 and subelements Element 2 and subelements Element 3 and subelements

1.0 Investment 2.0 System operations & 

support 3.0 Legacy system phase-out

1.2 Concept exploration 2.2 Annual operations 

investment

3.2 Phase-out investment

1.2.1 Engineering analysis 3.2.1 Hardware

& specification 2.2.1 Maintenance 

investment

3.2.2 Software

1.2.2 Concept exploration 

hardware

3.2.3 Hazardous material 

2.2.2 Replenishment spares handling

1.2.3 Concept exploration 

software

2.2.3 Replenishment 

supplies

3.3 Phase-out operations & 

support

1.2.4 Concept exploration 

data

2.3 Hardware maintenance 3.3.1 Hardware 

maintenance

1.2.5 Exploration 

documentation

2.3.1 Hardware 

maintenance

3.3.2 Software 

maintenance

1.2.6 Concept exploration 

testing

2.3.2 Maintenance support 3.3.3 Unit & subunit 

operations 2.3.3 Other hardware 

maintenance 3.3.4 Megacenter 

operations

1.2.7 Facilities

2.4 Software maintenance 1.2.8 Other

1.3 System development 2.4.1 Commercial off-theshelf software3.3.5 Phase-out contracts

1.3.1 System design & 

specification

2.4.2 Application & mission 

software

1.3.2 Prototype & test site 

investment

2.4.3 Communication 

software

1.4 System procurement 2.5 Megacenter maintenance

1.4.1 Deployment 

hardware

2.6 Data maintenance

1.4.2 System deployment 

software

2.6.1 Mission application 

data

1.4.3 Initial 

documentation

2.6.2 Standard 

administrative data

1.4.4 Logistics support 

equipment

2.7 Site operations

1.4.5 Initial spares 2.7.1 System operational 

personnel

1.4.6 Warranties 2.7.2 Utility requirement 

1.5 Outsource investment 2.7.3 Fuel

1.5.1 Capital investment 2.7.4 Facilities lease & 

maintenance

1.5.2 Software 

development

2.7.5 Communications 

1.5.3 System user 

investment

2.7.6 Base operating & 

support

1.6 System implementation 

& fielding

2.7.7 Recurring training
74 Chapter 8 GAO-09-3SP

Element 1 and subelements Element 2 and subelements Element 3 and subelements

1.0 Investment 2.0 System operations & 

support 3.0 Legacy system phase-out

1.6.1 Training 2.7.8 Miscellaneous support

1.6.2 Integration, test, 

acceptance

2.8 Environmental & 

hazardous

1.6.3 Common support 

equipment

2.9 Contract leasing

1.6.4 Site activation & 

facilities

1.6.5 Initial supplies

1.6.6 Engineering 

changes

1.6.7 Initial logistics 

support

1.6.8 Office furniture & 

furnishings

1.6.9 Data upload & 

transition

1.6.10 Communications

1.6.11 Other

1.7 Upgrades

1.7.1 Upgrade 

development

1.7.2 Life cycle upgrades 

1.7.3 Central mega center 

upgrades

1.8 Disposal & reuse

1.8.1 Capital recoupment 

1.8.2 Retirement

1.8.3 Environmental & 

hazardous 

Source: DOD.

This standard WBS should be tailored to fit each program. In some cases, the cost element structure 

contains built-in redundancies that provide flexibility in accounting for costs. For example, logistics 

support costs could occur in either investment or operations and support. However, it is important that 

the cost element structure of the automated information system not double count costs that could be 

included in more than one cost element. While the structure is flexible, the same rules as those of a 

WBS apply, in that children are assigned to only one parent. (Appendix IX contains numerous examples 

of standard work breakdown structures for, among others, surface, sea, and air transportation systems; 

military systems; communications systems; and systems for construction and utilities.) 
GAO-09-3SP Chapter 8 75

WBS and Scheduling

The WBS should be used as the outline for the integrated master schedule, using the levels of indenture 

down to the work package level. Since the WBS defines the work in lower levels of detail, its framework 

provides the starting point for defining all activities and tasks that will be used to develop the program 

schedule. 

The lowest level of the WBS is the work package. Within the work packages, the activities are defined 

and scheduled. When developing the program schedule, the WBS—in outline form—should be simply 

cut and pasted into the software. From there, the lower-level work packages and subsequent activities and 

tasks are defined. 

Accordingly, the WBS provides a logical and orderly way to begin preparing the detailed schedule, 

determining the relationships between activities, and identifying resources required to accomplish the 

tasks. Therefore, high-level summary tasks and all the detailed tasks in the schedule should map directly 

to the WBS to ensure that the schedule encompasses the entire work effort.

WBS and EVM

By breaking the work into smaller, more manageable work elements, a WBS can be used to integrate the 

scheduled activities and costs for accomplishing each work package at the lowest level of the WBS. This is 

essential for developing the resource-loaded schedule that forms the foundation for the EVM performance 

measurement baseline. Thus, a WBS is an essential part of EVM cost, schedule, and technical 

monitoring, because it provides a consistent framework from which to measure progress. This framework 

can be used to monitor and control costs based on the original baseline and to track where and why there 

were differences. In this way, the WBS serves as the common framework for analyzing the original cost 

estimate and the final cost outcome. 

When analysts use cost, schedule, and technical information organized by the WBS hierarchical structure, 

they can summarize data to provide management valuable information at any phase of the program. 

Furthermore, because a WBS addresses the entire program, managers at any level can assess their progress 

against the cost estimate plan. This helps keep program status current and visible so that risks can be 

managed or mitigated quickly. Without a WBS, it would be much more difficult to analyze the root cause 

of cost, schedule, and technical problems and to choose the optimum solution to fix them.

The WBS also provides a common thread between EVM and the integrated master schedule (IMS)—the 

time-phased schedule DOD and other agencies use for assessing technical performance. This link to the 

WBS can allow for further understanding of program cost and schedule variances. When the work is 

broken down into small pieces, progress can be linked to the IMS for better assessments of cost, technical, 

schedule, and performance issues. The WBS also enhances project control by tying the contractual work 

scope to the IMS, which DOD commonly uses to develop a program’s technical goals and plans.

WBS and Risk Management

The WBS is also valuable for identifying and monitoring risks. During the cost estimating phase, the 

WBS is used to flag elements likely to encounter risks, allowing for better contingency planning. During 
76 Chapter 8 GAO-09-3SP

program execution, the WBS is used to monitor risks using the EVM system, which details plans to a 

level that is needed to accomplish all tasks. 

In scheduling the work, the WBS can help identify activities in the schedule that are at risk because 

resources are lacking or because too many activities are planned in parallel to one another. In addition, risk 

items can be mapped to activities in the schedule and the results can be examined through a schedule risk 

analysis (more detail is in appendix X). 

WBS Benefits

Elements of a WBS may vary by phase, since different activities are required for development, production, 

operations, and support. Establishing a master WBS as soon as possible for the program’s life cycle that 

details the WBS for each phase provides many program benefits:

■ segregating work elements into their component parts; 

■ clarifying relationships between the parts, the end product, and the tasks to be completed; 

■ facilitating effective planning and assignment of management and technical responsibilities; 

■ helping track the status of technical efforts, risks, resource allocations, expenditures, and the cost 

and schedule of technical performance within the appropriate phases, since the work in phases 

frequently overlaps;

■ helping ensure that contractors are not unnecessarily constrained in meeting item requirements; 

and

■ providing a common basis and framework for the EVM system and the IMS, facilitating 

consistency in understanding program cost and schedule performance and assigning to the 

appropriate phase. Since the link between the requirements, WBS, the statement of work, IMS, 

and the integrated master plan provides specific insights into the relationship between cost, 

schedule, and performance, all items can be tracked to the same WBS elements. 

As the program or system matures, engineering efforts should focus on system-level performance 

requirements—validating critical technologies and processes and developing top-level specifications. As 

the specifications are further defined, the WBS will better define the system in terms of its specifications. 

After the system concept has been determined, major subsystems can be identified and lower-level 

functions determined, so that lower-level system elements can be defined, eventually completing the total 

system definition. The same WBS can be used throughout, updating and revising it as the program or 

system development proceeds and as the work in each phase progresses. One of the outputs of each phase 

is an updated WBS covering the succeeding phases.

In summary, a well-developed WBS is essential to the success of all acquisition programs. A 

comprehensive WBS provides a consistent and visible framework that improves communication; helps 

in the planning and assignment of management and technical responsibilities; and facilitates tracking 

engineering efforts, resource allocations, cost estimates, expenditures, and cost and technical performance. 

Without one, a program is most likely to encounter problems, as case studies 23 and 24 illustrate. 
GAO-09-3SP Chapter 8 77

Case Study 23: Developing Work Breakdown Structure, from NASA, 

GAO-04-642

For more than a decade, GAO had identified NASA’s contract management as a high-risk 

area. NASA had been unable to collect, maintain, and report the full cost of its programs 

and projects. Because of persistent cost growth in a number of NASA programs, GAO was 

asked to assess 27 programs—10 in detail. GAO found that only 3 of the 10 had provided 

a complete breakdown of the work to be performed, despite agency guidance calling 

for projects to break down the work into smaller units to facilitate cost estimating and 

program management and to help ensure that relevant costs were not omitted. 

Underestimating full life-cycle costs creates the risk that a program may be underfunded 

and subject to major cost overruns. It may be reduced in scope, or additional funding 

may have to be appropriated to meet objectives. Overestimating life-cycle costs creates 

the risk that a program will be thought unaffordable and it could go unfunded. Without 

a complete WBS, NASA’s programs cannot ensure that its LCCEs capture all relevant 

costs, which can mean cost overruns. Inconsistent WBS estimates across programs can 

cause double counting or, worse, costs can be underestimated when historical program 

costs are used for projecting future costs for similar programs. Among its multiple 

recommendations, GAO recommended that NASA base its cost estimates for each program 

on a WBS that encompassed both in-house and contractor efforts and develop procedures 

that would prohibit proposed projects from proceeding through review and approval if 

they did not address the elements of recommended cost estimating practices.

GAO, NASA: Lack of Disciplined Cost-Estimating Processes Hinders Effective 

Program Management, GAO-04-642 (Washington, D.C.: May 28, 2004).

Case Study 24: Developing Work Breakdown Structure, from 

Homeland Security, GAO-06-296

The Department of Homeland Security (DHS) established U.S. Visitor and Immigrant Status 

Indicator Technology (US–VISIT) to collect, maintain, and share information, including 

biometric identifiers, on selected foreign nationals entering and exiting the United States. 

Having reported that the program had not followed effective cost estimating practices, GAO 

recommended that DHS follow effective practices for estimating future increments. 

GAO then reported on the cost estimates for the latest increment in February 2006, 

finding US–VISIT’s cost estimates still insufficient. For example, they did not include a 

detailed WBS and they omitted important cost elements such as system testing. The 

uncertainties associated with the latest system increment cost estimate were not identified. 

Uncertainty analysis provides the basis for adjusting estimates to reflect unknown facts and 

circumstances that could affect costs, and it identifies risk associated with the cost estimate. 

Program officials stated that they recognized the importance of developing reliable 

cost estimates and initiated actions to more reliably estimate the costs of future system 

increments. For example, US–VISIT chartered a cost analysis process action team to develop, 

document, and implement a cost analysis policy, process, and plan for the program. Program 

officials had also hired additional contracting staff with cost estimating experience.

GAO, Homeland Security: Recommendations to Improve Management of Key 

Border Security Program Need to Be Implemented, GAO-06-296 (Washington, 

D.C.: Feb. 14, 2006).
78 Chapter 8 GAO-09-3SP

5. Best Practices Checklist: Work Breakdown Structure

 A product-oriented WBS represents best practice.

 It reflects the program work that needs to be done. It 

3 clearly outlines the end product and major work for the program;

3 contains at least 3 levels of indenture;

3 is flexible and tailored to the program. 

 The 100 percent rule applies: the sum of the children equals the parent.

3 The WBS defines all work packages, which in turn include all cost 

elements and deliverables.

3 In addition to hardware and software elements, the WBS contains 

program management and other common elements to make sure all 

the work is covered.

 Each system has one program WBS but may have several contract WBSs 

that are extended from the program WBS, depending on the number of 

subcontractors. 

 The WBS is standardized so that cost data can be collected and used for 

estimating future programs. It

3 facilitates portfolio management, including lessons learned;

3 matches schedule, cost estimate, and EVM at a high level;

3 is updated as changes occur and the program becomes better defined;

3 includes functional activities within each element that are needed to 

support each product deliverable;

3 is the starting point for developing the program’s detailed schedule;

3 provides a framework for identifying and monitoring risks and the 

effectiveness of contingency plans;

3 provides for a common language between the government program 

management office, technical specialists, prime contractors, and 

subcontractors.

 The WBS has a dictionary that

3 defines each element and how it relates to others in the hierarchy; 

3 clearly describes what is included in each element;

3 describes resources and functional activities needed to produce the 

element product;

3 links each element to other relevant technical documents.
GAO-09-3SP Chapter 9 79

Ground Rules and Assumptions

Chapter 9

Cost estimates are typically based on limited information and therefore need to be bound by the 

constraints that make estimating possible. These constraints usually take the form of assumptions that 

bind the estimate’s scope, establishing baseline conditions the estimate will be built from. Because of the 

many unknowns, cost analysts must create a series of statements that define the conditions the estimate is 

to be based on. These statements are usually made in the form of ground rules and assumptions (GR&A). 

By reviewing the technical baseline and discussing the GR&As with customers early in the cost estimating 

process, analysts can flush out any potential misunderstandings. GR&As

■ satisfy requirements for key program decision points,

■ answer detailed and probing questions from oversight groups,

■ help make the estimate complete and professional,

■ present a convincing picture to people who might be skeptical,

■ provide useful estimating data and techniques to other cost estimators, 

■ provide for reconstruction of the estimate when the original estimators are no longer available, and

■ provide a basis for the cost estimate that documents areas of potential risk to be resolved.

Ground Rules

Ground rules and assumptions, often grouped together, are distinct. Ground rules represent a common 

set of agreed on estimating standards that provide guidance and minimize conflicts in definitions. When 

conditions are directed, they become the ground rules by which the team will conduct the estimate. The 

technical baseline requirements discussed in chapter 7 represent cost estimate ground rules. Therefore, a 

comprehensive technical baseline provides the analyst with all the necessary ground rules for conducting 

the estimate.

Assumptions

Without firm ground rules, the analyst is responsible for making assumptions that allow the estimate 

to proceed. In other words, assumptions are required only where no ground rules have been provided. 

Assumptions represent a set of judgments about past, present, or future conditions postulated as true 

in the absence of positive proof. The analyst must ensure that assumptions are not arbitrary, that 

they are founded on expert judgments rendered by experienced program and technical personnel. 

Many assumptions profoundly influence cost; the subsequent rejection of even a single assumption by 

management could invalidate many aspects of the estimate. Therefore, it is imperative that cost estimators 

brief management and document all assumptions well, so that management fully understands the 

conditions the estimate was structured on. Failing to do so can lead to overly optimistic assumptions that 
80 Chapter 9 GAO-09-3SP

heavily influence the overall cost estimate, to cost overruns, and to inaccurate estimates and budgets. (See 

case study 25.) 

Case Study 25: The Importance of Assumptions, from Space 

Acquisitions, GAO-07-96

Estimated costs for DOD’s major space acquisition programs increased about $12.2 billion, 

nearly 44 percent, above initial estimates for fiscal years 2006 through 2011. Such growth 

has had a dramatic effect on DOD’s overall space portfolio. To cover the added costs of 

poorly performing programs, DOD shifted scarce resources from other programs, creating 

a cascade of cost and schedule inefficiencies.

GAO’s case study analyses found that program office cost estimates—specifically, 

assumptions they were based on—were unrealistic in eight areas, many interrelated. In 

some cases, such as assumptions regarding weight growth and the ability to gain leverage 

from legacy systems, past experiences or contrary data were ignored. In others, such as 

when contractors were given more program management responsibility or when growth 

in the commercial market was predicted, estimators assumed that promises of reduced 

cost and schedule would be borne out but did not have the benefit of experience to factor 

them into their work.

GAO also identified flawed assumptions that reflected deeper flaws in acquisition 

strategies or development approaches. For example, five of six programs GAO reviewed 

assumed that technologies would be sufficiently mature when needed, even though they 

began without a complete understanding of how long it would take or how much it would 

cost to ensure that they could work as intended. In four programs, estimators assumed few 

delays, even though the programs adopted highly aggressive schedules while attempting 

to make ambitious leaps in capability. In four programs, estimators assumed funding 

would stay constant, even though space and weapons programs frequently experienced 

funding shifts and the Air Force was in the midst of starting a number of costly new space 

programs to replenish older ones.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Global and Element-Specific Ground Rules and Assumptions

GR&As are either global or element specific. Global GR&As apply to the entire estimate; element-specific 

GR&As are driven by each WBS element’s detailed requirements. GR&As are more pronounced for 

estimates in the development phase, where there are more unknowns; they become less prominent as the 

program moves through development into production.

While each program has a unique set of GR&As, some are general enough that each estimate should 

address them. For example, each estimate should at a minimum define the following global GR&As: 

program schedule, cost limitations (for example, unstable funding stream or staff constraints), high-level 

time phasing, base year, labor rates, inflation indexes, participating agency support, and government 

furnished equipment.35

35Government furnished equipment can also be an assumption and is not always a ground rule.
GAO-09-3SP Chapter 9 81

One of the most important GR&As is to define a realistic schedule. It may be difficult to perform an indepth schedule assessment early to uncover the frequent optimism in initial program schedules. Ideally, 

members from manufacturing and the technical community should be involved in developing the 

program schedule, but often information is insufficient and assumptions must be made. In this case, it is 

important that this GR&A outline the confidence the team has in the ability to achieve the schedule so 

that it can be documented and presented to management. 

One major challenge in setting realistic schedules is that the completion date is often set by external factors 

outside the control of the program office before any analysis has been performed to determine whether 

it is feasible. Another predominant problem is that schedule risk is often ignored or not analyzed—or 

when it is analyzed, the analysis is biased. This can occur on the government (customer) or contractor side 

or both. Risk analysis conducted by a group independent of the project manager has a better chance of 

being unbiased than one conducted by the program manager. However, it should also be noted that many 

organizations are not mature enough to acknowledge or to apply program schedule or cost risk realism 

because of the possible repercussions. For example, a contractor may be less likely to identify schedule or 

cost risk if it fears negative reaction from the customer. Likewise, the customer may be unwilling to report 

cost or schedule risk from fear that the program could be canceled. 

Sometimes, management imposes cost limitations because of budget constraints. The GR&A should 

then clearly explain the limitation and how it affects the estimate. Usually, cost limitations are handled by 

delaying program content or by a funding shortfall if program content cannot be delayed. In many cases, 

such actions will both delay the program and increase its final delivered cost. Either way, management 

needs to be fully apprised of how this GR&A affects the estimate.

Estimates are time phased because program costs usually span many years. Time phasing spreads a 

program’s expected costs over the years in which they are anticipated to aid in developing a proper budget. 

Depending on the activities in the schedule for each year, some years may have more costs than others. 

Great peaks or valleys in annual funding should be investigated and explained, however, since staffing 

is difficult to manage with such variations from one year to another. Anomalies are easily discovered 

when the estimate is time phased. Cost limitations can also affect an estimate’s time phasing, if there are 

budget constraints for a given fiscal year. Additionally, changes in program priority will affect funding and 

timing—often a program starts with high priority but that priority erodes as it proceeds, causing original 

plans to be modified and resulting in later delivery and higher cost to the government. These conditions 

should be addressed by the estimate and their effects adequately explained. 

The base year is used as a constant dollar reference point to track program cost growth. Expressing an 

estimate in base year dollars removes the effects of economic inflation and allows for comparing separate 

estimates “apples to apples.” Thus, a global ground rule is to define the base year dollars that the estimate 

will be presented in and the inflation index that will be used to convert the base year costs into then-year 

dollars that include inflation. At a minimum, the inflation index, source, and approval authority should 

be clearly explained in the estimate documentation. Escalation rates should be standardized across similar 

programs, since they are all conducted in the same economic environment, and priority choices between 

them should not hinge on different assumptions about what is essentially an economic scenario common 

to all programs.
82 Chapter 9 GAO-09-3SP

Some programs result from two or more agencies joining together to achieve common program goals. 

When this happens, agreements should lay out each agency’s area of responsibility. An agency’s failing 

to meet its responsibility could affect the program’s cost and schedule. In the GR&A section, these 

conditions should be highlighted to ensure that management is firmly aware that the success of the 

estimate depends on the participation of other agencies.

Equipment that the government agrees to provide to a contractor can range from common supply items 

to complex electronic components to newly developed engines for aircraft. Because the estimator cannot 

predict whether deliveries of such equipment will be timely, assumptions are usually made that it will be 

available when needed. It is important that the estimate reflect the items that it assumes government will 

furnish, so that the risk to the estimate if items are delayed can be modeled and presented to management. 

In general, schedules represent delivery of material from external sources, including the government, with 

date-constrained milestones. A better approach is to include the supplier’s work to produce the product 

by a summary activity in the schedule, examine the possibility of delayed delivery, include that risk in a 

schedule risk analysis, and monitor the work of the supplier as the date approaches.

In addition to global GR&As, estimate-specific GR&As should be tailored for each program, including

■ life-cycle phases and operations concept;

■ maintenance concepts;

■ acquisition strategy, including competition, single or dual sourcing, and contract or incentive type;

■ industrial base viability;

■ quantities for development, production, and spare and repair parts;

■ use of existing facilities, including any modifications or new construction; 

■ savings for new ways of doing business; 

■ commonality or design inheritance assumptions;

■ technology assumptions and new technology to be developed;

■ technology refresh cycles;

■ security considerations that may affect cost; and 

■ items specifically excluded from the estimate.

The cost estimator should work with members from the technical community to tailor these specific 

GR&As to the program. Information from the technical baseline and WBS dictionary help determine 

some of these GR&As, like quantities and technology assumptions. The element-specific GR&As carry 

the most risk and therefore should be checked for realism and should be well documented in order for the 

estimate to be considered credible.

Assumptions, Sensitivity, and Risk Analysis

Every estimate is uncertain because of the assumptions that must be made about future projections. 

Sensitivity analysis that examines how changes to key assumptions and inputs affect the estimate helps 

mitigate uncertainty. Best practice cost models incorporate the ability to perform sensitivity analyses 

without altering the model so that the effect of varying inputs can be quickly determined (more 
GAO-09-3SP Chapter 9 83

information is in chapters 13 and 14). For example, suppose a decision maker challenges the assumption 

that 5 percent of the installed equipment will be needed for spares, asking that the factor be raised to 10 

percent. A sensitivity analysis would show the cost impact of this change. Because of the implications 

that GR&As can have when assumptions change, the cost estimator should always perform a sensitivity 

analysis that portrays the effects on the cost and schedule of an invalid assumption. Such analysis often 

provides management with an invaluable perspective on its decision making. 

In addition to sensitivity analysis, factors that will affect the program’s cost, schedule, or technical status 

should be clearly identified, including political, organizational, or business issues. Because assumptions 

themselves can vary, they should always be inputs to program risk analyses of cost and schedule. A typical 

approach to risk analysis emphasizes the breadth of factors that may be uncertain. In a risk identification 

exercise, the goal is to identify all potential risks stemming from a broad range of sources. A good 

starting point would be to examine the program’s risk management database to determine which WBS 

elements these risks could affect. Another option would be to examine risks identified during a program’s 

integrated baseline review—a risk based assessment of the program plan to see whether the requirements 

can be met within cost and schedule assumptions. Regardless of what method is used to identify risk, it is 

important that more than just cost, schedule, and technical risks are examined. For example, budget and 

funding risks, as well as risks associated with start-up activities, staffing, and organizational issues, should 

also be considered. Therefore, risks from all sources such as external, organizational, and even project 

management practices, in addition to the technical challenges, need to be addressed.

Well-supported assumptions should include documentation of an assumption’s source and should discuss 

any weaknesses or risks. Solid assumptions are measurable and specific. For example, an assumption that 

states “transaction volume will average 500,000 per month and is expected to grow at an annual rate of 

5 percent” is measurable and specific, while “transaction volumes will grow greatly over the next 5 years” 

is not as helpful. By providing more detail, cost estimators can perform risk and sensitivity analysis to 

quantify the effects of changes in assumptions. 

Assumptions should be realistic and valid. This means that historical data should back them up to 

minimize uncertainty and risk. Understanding the level of certainty around an estimate is imperative 

to knowing whether to keep or discard an assumption. Assumptions tend to be less certain earlier in a 

program, and become more reliable as more information is known about them. A best practice is to place 

all assumptions in a single spreadsheet tab so that risk and sensitivity analysis can be performed efficiently 

and quickly. Explicit assumptions should be available, but assumptions are also sometimes implicit—

implicit assumptions should be documented as well.

Certain ground rules should always be tested for risk. For example, the effects of the program schedule’s 

slipping on both cost and schedule should always be modeled and the results presented to management. 

This is especially important if the schedule was known to be aggressive or was not assessed for realism. 

Too often, we have found that when schedules are compressed, for instance to satisfy a potential 

requirements gap, the optimism in the schedule does not hold and the result is greater costs and schedule 

delays. Case study 26 gives examples of what happens in such situations.
84 Chapter 9 GAO-09-3SP

Case Study 26: Testing Ground Rules for Risk, from Space Acquisitions, 

GAO-07-96

GAO’s analyses of six ongoing space programs found that original cost estimates were 

unrealistic in a number of areas. The six programs included the following four.

Advanced Extremely High Frequency Satellite Program. The first AEHF launch was originally 

scheduled for June 2006. In response to a potential gap in satellite coverage because 

of the launch failure of the third Milstar satellite, DOD accelerated the schedule by 18 

months, aiming for December 2004. An unsolicited contractor proposal stated that it 

could meet this date, even though not all AEHF’s requirements had been fully determined. 

The program office thus knew that the proposed schedule was overly optimistic, but the 

decision was made at high levels in DOD to award the contract. DOD did not, however, 

commit the funding to support the activities and manpower needed to design and 

build the satellites more quickly. Funding issues further hampered development efforts, 

increased schedule delays, and contributed to cost increases.

National Polar-orbiting Operational Environmental Satellite System. When the NPOESS 

estimate was developed, the system was expected to be heavier, require more power, 

and have more than twice as many sensors as heritage satellites. Yet the program office 

estimated that the new satellites would be developed, integrated, and tested in less 

time than heritage satellites. Independent cost estimators highlighted to the NPOESS 

program office that the proposed integration schedule was unrealistic, compared to 

historical satellite programs. Later, the CAIG cautioned the program office that the system 

integration assembly and test schedule were unrealistic and the assumptions used to 

develop the estimate were not credible.

Space Based Infrared System High Program. The SBIRS schedule proposed in 1996 did 

not allow enough time for geosynchronous Earth orbit system integration. And it did 

not anticipate the program design and workmanship flaws that eventually cost the 

program considerable delays. The schedule was also optimistic with regard to ground 

software productivity and time needed to calibrate and assess satellite health. Delivery 

of highly elliptical orbit sensors was delayed by almost 3 years, the launch of the first 

geosynchronous Earth orbit satellite by 6 years.

Wideband Gapfiller Satellites. The request for proposals specified that the available WGS 

budget was $750 million for three satellites and that the ground control system was to 

be delivered within 36 months. Competing contractors were asked to offer maximum 

capacity, coverage, and connectivity in a contract that would use existing commercial 

practices and technologies. However, greater design complexity and supplier quality issues 

caused the WGS schedule to stretch to 78 months for the first expected launch. DOD’s 

history had been 55–79 months to develop satellites similar to WGS, so that while DOD’s 

experience was within the expected range, the original 36-month schedule was unrealistic.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Above and beyond the program schedule, some programs can be affected by the viability of the industrial 

base. Case study 27 illustrates. 
GAO-09-3SP Chapter 9 85

Case Study 27: The Industrial Base, from Defense Acquisition, 

GAO-05-183

For the eight case study ships GAO examined, cost analysts relied on the actual cost of 

previously constructed ships, without adequately accounting for changes in the industrial 

base, ship design, or construction methods. Cost data available to Navy cost analysts were 

based on higher ship construction rates from the 1980s. These data were based on lower 

costs because of economies of scale, which did not reflect the lower procurement rates 

after 1989. 

According to the shipbuilder, material cost increases on the CVN 76 and CVN 77 in the 

Nimitz class of aircraft carriers could be attributed to a declining supplier base and 

commodity price increases. Both carriers’ material costs had been affected by more than a 

15 percent increase in metals costs that in turn increased costs for associated components. 

Moreover, many of the materials used in the construction of aircraft carriers are highly 

specialized and unique—often produced by only one manufacturer. With fewer 

manufacturers competing in the market, the materials were highly susceptible to cost 

increases.

After the Seawolf submarine program was cancelled and, over a period of 6 years, 

submarine production had decreased from three to four submarines per year to one, 

many vendors left the nuclear submarine business to focus on more lucrative commercial 

product development. Prices for highly specialized material increased, since competition 

and business had diminished. 

For example, many vendors were reluctant to support the Virginia class submarine 

contract because costs associated with producing small quantities of highly specialized 

materials were not considered worth the investment—especially for equipment with no 

other military or commercial applications.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

Another area in which assumptions tend to be optimistic is technology maturity. Having reviewed the 

experiences of DOD and commercial technology development, GAO has found that programs that relied 

on technologies that demonstrated a high level of maturity were in a better position to succeed than those 

that did not. Simply put, the more mature technology is at the start of a program, the more likely it is that 

the program will meet its objectives. 

Technologies that are not fully developed represent a significant challenge and add a high degree of risk 

to a program’s schedule and cost. Programs typically assume that the technology required will arrive on 

schedule and be available to support the effort. While this assumption allows the program to continue, 

the risk that it will prove inaccurate can greatly affect cost and schedule. Case studies 28 and 29 provide 

examples.
86 Chapter 9 GAO-09-3SP

Case Study 28: Technology Maturity, from Defense Acquisitions, 

GAO-05-183

The lack of design and technology maturity led to rework, increasing the number of labor 

hours for most of the case study ships. For example, the design of the LPD 17, in the San 

Antonio class of transports, continued to evolve even as construction proceeded. When 

construction began on the DDG 91 and DDG 92, in the Arleigh Burke class of destroyers—

the first ships to incorporate the remote mine hunting system—the technology was still 

being developed. As a result, workers were required to rebuild completed ship areas to 

accommodate design changes.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

Case Study 29: Technology Maturity, from Space Acquisitions, 

GAO-07-96

The Advanced Extremely High Frequency (AEHF) program of communications satellites 

faced several problems of technology maturity. They included developing a digital 

processing system that would support 10 times the capacity of Milstar’s medium data rate, 

the predecessor satellite, without self-interference and using phased array antennas at 

extremely high frequencies, which had never been done before. In addition, the change 

from a physical process to an electronic process for crypto rekeys had not been expected 

at the start of AEHF. Milstar had required approximately 2,400 crypto rekeys per month 

and had been done physically. AEHF’s proposed capability was approximately 100,000—

too large for physical processing. Changing the rekeys to electronic processing was 

revolutionary and led to unexpected cost and schedule growth.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Cost estimators and auditors should not get trapped by overly optimistic technology forecasts. It is well 

known that program advocates tend to underestimate the technical challenge facing the development of 

a new system. Estimators and auditors alike should always seek to uncover the real risk by performing an 

uncertainty analysis. In doing so, it is imperative that cost estimators and auditors meet with engineers 

familiar with the program and its new technology to discuss the level of risk associated with the technical 

assumptions. Only then can they realistically model risk distributions using an uncertainty analysis and 

analyze how the results affect the overall cost estimate. 

Once the risk uncertainty and sensitivity analyses are complete, the cost estimator should formally 

convey the results of changing assumptions to management as early and as far up the line as possible. 

The estimator should also document all assumptions to help management understand the conditions the 

estimate was based on. When possible, the analyst should request an updated technical baseline in which 

the new assumptions have been incorporated as ground rules. Case study 30 illustrates an instance of 

management’s not knowing the effects of changing assumptions. 
GAO-09-3SP Chapter 9 87

Case Study 30: Informing Management of Changed Assumptions, 

from Customs Service Modernization, GAO/AIMD-99-41

The Automated Commercial Environment (ACE) was a major U.S. Customs Service 

information technology system modernization effort. In November 1997, it was estimated 

that ACE would cost $1.05 billion to develop, operate, and maintain between 1994 and 

2008. GAO found that the agency lacked a reliable estimate of what ACE would cost to 

build, deploy, and maintain. 

The cost estimates were understated, benefit estimates were overstated, and both were 

unreliable. Customs’ August 1997 cost-benefit analysis estimated that ACE would produce 

cumulative savings of $1.9 billion over a 10-year period. The analysis identified $644 

million in savings—33 percent of the total estimated savings—resulting from increased 

productivity. Because this estimate was driven by Customs’ assumption that every minute 

“saved” by processing transactions or analyzing data faster using ACE rather than its 

predecessor system would be productively used by all workers, it was viewed as a best 

case upper limit on estimated productivity improvements. 

Given the magnitude of the potential savings, even a small change in the assumption 

translated into a large reduction in benefits. For example, conservatively assuming that 

three-fourths of each minute saved would be used productively by three-fourths of all 

workers, the expected benefits would be reduced by about $282 million. Additionally, 

the analysis excluded costs for hardware and systems software upgrades at each port 

office. Using Customs’ estimate for acquiring the initial suite of port office hardware and 

systems software, and assuming a technology refreshment cycle of every 3 to 5 years, GAO 

estimated this cost at $72.9 million to $171.8 million.

Because Customs did not have reliable information on ACE costs and benefits and had not 

analyzed viable alternatives, it did not have adequate assurance that ACE was the optimal 

approach. In fact, it had no assurance at all that ACE would be cost-effective. Furthermore, 

it had not justified the return on its investment in each ACE increment and therefore 

would not be able to demonstrate whether ACE would be cost-effective until it had spent 

hundreds of millions of dollars to acquire the entire system. 

GAO recommended that Customs rigorously analyze alternative approaches to building 

ACE and, for each increment, use disciplined processes to prepare a robust LCCE, prepare 

realistic and supportable benefit expectations, and validate actual costs and benefits once 

an increment had been piloted.

GAO, Customs Service Modernization: Serious Management and Technical 

Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 

1999).
88 Chapter 9 GAO-09-3SP

6. Best Practices Checklist: Ground Rules and Assumptions

 All ground rules and assumptions have been

3 Developed by estimators with input from the technical community.

3 Based on information in the technical baseline and WBS dictionary.

3 Vetted and approved by upper management.

3 Documented to include the rationale behind the assumptions and 

historical data to back up any claims.

3 Accompanied by a level of risk of each assumption’s failing and its effect 

on the estimate.

 To mitigate risk, 

3 All GR&As have been placed in a single spreadsheet tab so that risk and 

sensitivity analysis can be performed quickly and efficiently.

3 All potential risks including cost, schedule, technical, and programmatic 

(e.g., risks associated with budget and funding, start up activities, 

staffing, and organizational issues) have been identified and traced to 

specific WBS elements.

ù A schedule risk analysis has been performed to determine the 

program schedule’s realism.

ù A cost risk analysis, incorporating the results of the schedule risk 

analysis, has been performed to determine the program’s cost 

estimate realism.

 Budget constraints, as well as the effect of delaying program content, have 

been defined.

3 Peaks and valleys in time-phased budgets have been explained.

3 Inflation index, source, and approval authority have been identified.

3 Dependence on participating agencies, the availability of government 

furnished equipment, and the effects if these assumptions do not hold 

have been identified.

3 Items excluded from the estimate have been documented and 

explained.

3 Technology was mature before it was included; if its maturity was 

assumed, the estimate addresses the effect of the assumption’s failure 

on cost and schedule.

 Cost estimators and auditors met with technical staff to determine risk 

distributions for all assumptions; the distributions were used in sensitivity 

and uncertainty analyses of the effects of invalid assumptions. Management 

has been briefed, and the results have been documented.
GAO-09-3SP Chapter 10 89

Data

Chapter 10

Data are the foundation of every cost estimate. How good the data are affects the estimate’s overall 

credibility. Depending on the data quality, an estimate can range anywhere from a mere guess to a highly 

defensible cost position. Credible cost estimates are rooted in historical data. Rather than starting from 

scratch, estimators usually develop estimates for new programs by relying on data from programs that 

already exist and adjusting for any differences. Thus, collecting valid and useful historical data is a key 

step in developing a sound cost estimate. The challenge in doing this is obtaining the most applicable 

historical data to ensure that the new estimate is as accurate as possible. One way of ensuring that the data 

are applicable is to perform checks of reasonableness to see if the results are similar. Different data sets 

converging toward one value provides a high degree of confidence in the data. 

Performing quality checks takes time and requires access to large quantities of data. This is often the most 

difficult, time-consuming, and costly activity in cost estimating. It can be exacerbated by a poorly defined 

technical baseline or WBS. However, by gathering sufficient data, cost estimators can analyze cost trends 

on a variety of related programs, which gives insight into cost estimating relationships that can be used to 

develop parametric models. 

Before collecting data, the estimator must fully understand what needs to be estimated. This 

understanding comes from the purpose and scope of the estimate, the technical baseline description, the 

WBS, and the ground rules and assumptions. Once the boundaries of the estimate are known, the next 

step is to establish an idea of what estimating methodology will be used. Only after these tasks have been 

performed should the estimator begin to develop an initial data collection plan.

Data Collection

Data collection is a lengthy process and continues throughout the development of a cost estimate and 

through the program execution itself. Many types of data need to be collected—technical, schedule, 

program, and cost data. Once collected, the data need to be normalized. Data can be collected in a variety 

of ways, such as from databases of past projects, engineering build-up estimating analysis, interviews, 

surveys, data collection instruments, and focus groups. After the estimate is complete, the data need to be 

well documented, protected, and stored for future use in retrievable databases. Cost estimating requires a 

continual influx of current and relevant cost data to remain credible. The cost data should be managed by 

estimating professionals who understand what the historical data are based on, can determine whether the 

data have value in future projections, and can make the data part of the corporate history. 

Cost data should be continually supplemented with written vendor quotes, contract data, and actual 

cost data for each new program. Moreover, cost estimators should know the program acquisition plans, 
90 Chapter 10 GAO-09-3SP

contracting processes, and marketplace conditions, all of which can affect the data. This knowledge 

provides the basis for credibly using, modifying, or rejecting the data in future cost estimates.

Knowing the factors that influence a program’s cost is essential for capturing the right data. Examples are 

equivalent source lines of code, number of interfaces for software development, number of square feet for 

construction, and the quantity of aircraft to be produced. To properly identify cost drivers, it is imperative 

that cost estimators meet with the engineers and other technical experts. In addition, by studying 

historical data, cost estimators can determine through statistical analysis the factors that tend to influence 

overall cost. Furthermore, seeking input from schedule analysts can provide valuable knowledge about 

how aggressive a program’s schedule may be.

Cost estimates must be based on realistic schedule information. Some costs such as labor, quality, 

supervision, rented space and equipment, and other time-related overheads depend on the duration of the 

activities they support. Often the cost estimators are in synch with the baseline schedule with the early 

estimates, but they also have to keep in touch with changes in the schedule, since schedule changes can 

lead to cost changes.

In addition to data for the estimate, backup data should be collected for performing cross-checks. This 

takes time and usually requires travel to meet with technical experts. It is important to plan ahead and 

schedule the time for these activities. Scheduling insufficient time can affect the estimator’s ability to 

collect and understand the data, which can then result in a less confident cost estimate.

Common issues in data collection include inconsistent data definitions in historical programs compared to 

the new program. Understanding what the historical data include is vital to data reliability. For example, 

are the data skewed because they are for a program that followed an aggressive schedule and therefore 

instituted second and third shifts to complete the work faster? Or was a new manufacturing process 

implemented that was supposed to generate savings but resulted in more costs because of initial learning 

curve problems? Knowing the history behind the data will allow for its proper allocation for future 

estimates.

Another issue is whether the data are even available. Some agencies may not have any cost databases. 

Data may be accessible at higher levels but information may not be sufficient to break them down to the 

lower levels needed to estimate various WBS elements. Data may be incomplete. For instance, they may 

be available for the cost to build a component, but the cost to integrate the component may be missing. 

Similarly, if data are in the wrong format, they may be difficult to use. For example, if the data are only in 

dollars and not hours, they may not be as useful if the labor and overhead rates are not available. 

Sometimes data are available, but the cost estimator cannot gain access to them. This can happen when 

the data are highly classified or considered competition sensitive. When this is the case, the cost estimator 

may have to change the estimating approach to fit the data that are available. Case study 31 gives an 

example.
GAO-09-3SP Chapter 10 91

Case Study 31: Fitting the Estimating Approach to the Data, from 

Space Acquisitions, GAO-07-96

The lack of reliable technical source data hampers cost estimating. Officials GAO spoke 

with believed that cost estimation data and databases on which to base cost estimates 

were incomplete, insufficient, and outdated. They cited the lack of reliable historical and 

current cost, technical, and program data and expressed concern that available cost, 

schedule, technical, and risk data were not similar to the systems they were developing 

cost estimates for. In addition, some expressed concern that relevant classified and 

proprietary commercial data might exist but were not usually available to the cost 

estimating community working on unclassified programs. Some believed that Air Force 

cost estimators needed to be able to use all relevant data, including those contained in 

National Reconnaissance Office cost databases, since the agency builds highly complex, 

classified satellites in comparable time and at comparable costs per pound.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Types of Data

In general, the three main types of data are cost data, schedule or program data, and technical data. 

Cost data generally include labor dollars (with supporting labor hours and direct costs and overhead 

rates), material and its overhead dollars, facilities capital cost of money, and profit associated with various 

activities. Program cost estimators often do not know about specific dollars, so they tend to focus mostly 

on hours of resources needed by skill level. These estimates of hours are often inputs to specialized 

databases to convert them to cost estimates in dollars.

Schedule or program data provide parameters that directly affect the overall cost. For example, lead-time 

schedules, start and duration of effort, delivery dates, outfitting, testing, initial operational capability dates, 

operating profiles, contract type, multiyear procurement, and sole source or competitive awards must all 

be considered in developing a cost estimate. 

Technical data define the requirements for the equipment being estimated, based on physical and 

performance attributes, such as length, width, weight, horsepower, and size. When technical data 

are collected, care must be taken to relate the types of technologies and development or production 

methodologies to be used. These change over time and require adjustments when estimating relationships 

are being developed.

Cost data must often be derived from program and technical data. Moreover, program and technical data 

provide context for cost data, which by themselves may be meaningless. Consider the difference between 

these two examples:

■ Operations and maintenance utilities cost $36,500.

■ The Navy consumes 50,000 barrels of fuel per day per ship.

In the operations and maintenance example, the technical and program descriptors are missing, requiring 

follow-up questions like: What specific utilities cost $36,500? Gas or electricity or telephone? What time 
92 Chapter 10 GAO-09-3SP

does this cost represent? A month or a year? and When were these costs accrued? In the current year or 5 

years ago? In the Navy example, a cost estimator would need to investigate what type of ship consumes 

50,000 barrels per day—aircraft carrier? destroyer?—and what type of fuel is consumed.36

It is essential that cost estimators plan for and gain access, where feasible, to cost and technical and 

program data in order to develop a complete understanding of what the data represent. Without this 

understanding, a cost estimator may not be able to correctly interpret the data, leading to greater risk that 

the data can be misapplied. 

Sources of Data

Since all cost estimating methods are data-driven, analysts must know the best data sources. Table 10

lists some basic sources. Analysts should use primary data sources whenever possible. Primary data are 

obtained from the original source, can usually be traced to an audited document, are considered the best 

in quality, and are ultimately the most useful. Secondary data are derived rather than obtained directly 

from a primary source. Since they were derived, and thus changed, from the original data, their overall 

quality is lower and less useful. In many cases, secondary data are actual data that have been “sanitized” to 

obscure their proprietary nature. Without knowing the details, analysts will find such data of little use. 

Table 10: Basic Primary and Secondary Data Sources

Data type Primary Secondary

Basic accounting records x

Data collection input forms x

Cost reports x x

Historical databases x x

Interviews x x

Program briefs x x

Subject matter experts x x

Technical databases x x

Other organizations x x

Contracts or contractor estimates x

Cost proposals x

Cost studies x

Focus groups x

Research papers x

Surveys x

Source: DOD and NASA. 

Cost estimators must understand whether and how data were changed before deciding whether they will 

be useful. Furthermore, it is always better to use actual costs rather than estimates as data sources, since 

actual costs represent the most accurate data available.

36The examples and paragraph are © 2003, Society of Cost Estimating and Analysis, “Data Collection and Normalization: How to 

Get the Data and Ready It for Analysis.”
GAO-09-3SP Chapter 10 93

While secondary data should not be the first choice, they may be all that is available. Therefore, the cost 

estimator must seek to understand how the data were normalized, what the data represent, how old they 

are, and whether they are complete. If these questions can be answered, the secondary data may be useful 

for estimating and would certainly be helpful for cross-checking the estimate for reasonableness.

Sources of historical data include business plans, catalog prices, contract performance reports, contract 

funds status reports, cost and software data reports, forward pricing rate agreements, historical cost 

databases, market research, program budget and accounting data from prior programs, supplier cost 

information, historical or current vendor quotes, and weight reports. In the operating and support area, 

common data sources include DOD’s Visibility and Management of Operating and Support Costs 

management information system. Cost estimators should collect actual cost data from a list of similar and 

legacy programs. Since most new programs are improvements over existing ones, data should be available 

that share common characteristics with the new program.

Historical data provide the cost estimator insight into actual costs on similar programs, including any 

cost growth since the original estimate. As a result, historical data can be used to challenge optimistic 

assumptions. For example, a review of the average labor rates for similar tasks on other programs could 

be a powerful reality check against assumptions of skill mixes and overall effort. In addition, historical 

data from a variety of contractors can be used to establish generic program costs or they can be used to 

establish cost trends of a specific contractor across a variety of programs. 

Historical data also provide contractor cost trends relative to proposal values, allowing the cost estimator 

to establish adjustment factors if relying on proposal data for estimating purposes. Additionally, insights 

can be obtained on cost accounting structures to allow an understanding of how a certain contractor 

charges things like other direct costs and overhead. 

However, historical cost data also contain information from past technologies, so it is essential that 

appropriate adjustments are made to account for differences between the new system and the existing 

system with respect to such things as design characteristics, manufacturing processes (automation versus 

hands-on labor), and types of material used. This is where statistical methods, like regression, that analyze 

cost against time and performance characteristics can reveal the appropriate technology-based adjustment. 

CPRs and cost and software data reports are excellent sources of historical cost data for DOD programs. 

The CPR is the primary report of cost and schedule progress on contracts containing EVM compliance 

requirements. It contains the time-phased budget, the actual cost, and earned value, which is the budgeted 

value of completed work. 

By reviewing CPR data, the cost analyst can gain valuable insights into performance issues that may 

be relevant to future procurements. For instance, CPR data can provide information about changes to 

the estimate to complete (or the total expected cost of the program) and the performance measurement 

baseline, and it explains the reason for any variances. Before beginning any analysis of such reports, the 

analyst should perform a cursory assessment to ensure that the contractor has prepared them properly. 
94 Chapter 10 GAO-09-3SP

The several ways of analyzing cost data reports all use three basic elements in various combinations:

■ budgeted cost for work scheduled (BCWS), or the amount of budget allocated to complete a 

specific amount of work at a particular time;

■ budgeted cost for work performed (BCWP), also known as earned value, which represents 

budgeted value of work accomplished; and 

■ actual cost of work performed (ACWP), or actual costs incurred for work accomplished.37

Cost data reports are often used in estimating analogous programs, from the assumption that it is 

reasonable to expect similar programs at similar contractors’ plants to incur similar costs. This analogy 

may not hold for the costs of hardware or software but may hold in the peripheral WBS areas of data, 

program management, or systems engineering. If the analyst can then establish costs for the major 

deliverables, such as hardware or software, a factor may be applied for each peripheral area of the WBS, 

based on historical data available from cost reports. Sometimes, the data listed in the WBS include 

elements that the analyst may not be using in the present estimate—spares, training, support equipment. 

In such cases, these elements should be removed before the data are analyzed.

Rate and factor agreements contain rates and factors agreed to by the contractor and the appropriate 

government negotiator. Because the contractor’s business base may be fluid, with direct effect on these 

rates and factors, such agreements do not always exist. Information in them represents negotiated direct 

labor, overhead, general and administrative data, and facilities capital cost of money. These agreements 

could cover myriad factors, depending on each contractor’s accounting and cost estimating structure. 

Typical factors are material scrap, material handling, quality control, sustaining tooling, and miscellaneous 

engineering support factors.

The scope of the estimate often dictates the need to consult with other organizations for raw data. Once 

government test facilities have been identified, for example, those organizations can be contacted for 

current cost data, support cost data, and the like. Other government agencies could also be involved with 

the development of similar programs and can be potential sources of data. Additionally, a number of 

government agencies and industry trade associations publish cost data that are useful in cost estimating. 

The Defense Contract Management Agency (DCMA) and the Defense Contract Audit Agency (DCAA) 

help DOD cost analysts obtain validated data. Both agencies have on-site representatives at most major 

defense contractor facilities. Navy contractor resident supervisors of shipbuilding, for example, help 

obtain validated data. Before a contract is awarded, DCMA provides advice and services to help construct 

effective solicitations, identify potential risks, select the most capable contractors, and write contracts that 

meet customers’ needs. In evaluating contract proposals, DCMA assists in the review of the proposal 

assumptions to identify how tightly scope was constrained to reduce risk premiums in the proposed 

cost. After a contract is awarded, DCMA monitors contractors’ performance and management systems 

to ensure that cost, product performance, and delivery schedules comply with the contract’s terms and 

schedule. It is common for DCMA auditors to be members of teams assembled to review elements of 

proposals, especially in areas of labor and overhead rates, cost, and supervision of man-hour percentages. 

37We discuss these terms in chapters 18 and 19.
GAO-09-3SP Chapter 10 95

DCMA analysts often provide independent estimates at completion for programs; they are another 

potential source of information for cost analysts. 

DCAA performs necessary contract audits for DOD. It provides accounting and advisory services 

for contracts and subcontracts to all DOD components responsible for procurement and contract 

administration. Cost analysts should establish and nurture contacts with these activities, so that a 

continual flow of current cost-related information can be maintained. Although civil agencies have no 

comparable organizations, DCMA and DCAA occasionally provide support to them. 

Another area of potential cost data are contractor proposals. Analysts should remember that a contractor 

proposal as a source of data is a proposal—a document that represents the contractor’s best estimate of 

cost. Proposals also tend to be influenced by the amount the customer has to spend. When this is the 

case, the proposal data should be viewed as suspect, and care should be taken to determine if the proposal 

data are supportable. Because of this, an estimate contained in a contractor’s proposal should be viewed 

with some caution. During source selection in a competitive environment, for instance, lower proposed 

costs may increase the chances of receiving a contract award. This being so, it is very important to analyze 

the cost data for realism. A proposal can nonetheless provide much useful information and should be 

reviewed, when available, for the following: 

■ structure and content of the contractor’s WBS;

■ contractor’s actual cost history on the same or other programs;

■ negotiated bills of material;

■ subcontracted items;

■ government-furnished equipment compared to contractor-furnished equipment lists;

■ contractor rate and factor data, based on geography and makeup of workforce;

■ a self-check to ensure that all pertinent cost elements are included; 

■ top-level test of reasonableness;

■ technological state-of-the-art assumptions; and

■ estimates of management reserve and level of risk. 

Because of the potential for bias in proposal data, the estimator must test the data to see whether they 

deviate from other similar data before deciding whether they are useful for estimating. This can be done 

through a plant visit, where the cost estimator visits the contractor to discuss the basis for the proposal 

data. As with any potential source of data, it is critical to ensure that the data apply to the estimating 

task and are valid for use. In the next two sections, we address how a cost estimator should perform these 

important activities.

Data Applicability

Because cost estimates are usually developed with data from past programs, it is important to examine 

whether the historical data apply to the program being estimated. Over time, modifications may have 

changed the historical program so that it is no longer similar to the new program. For example, it does 

not make sense to use data from an information system that relied on old mainframe technology when 
96 Chapter 10 GAO-09-3SP

the new program will rely on server technology that can process data at much higher speeds. Having good 

descriptive requirements of the data is imperative in determining whether the data available apply to what 

is being estimated.

To determine the applicability of data to a given estimating task, the analyst must scrutinize them in light 

of the following issues: 

■ Do the data require normalization to account for differences in base years, inflation rates 

(contractor compared to government), or calendar year rather than fiscal year accounting systems?

■ Is the work content of the current cost element consistent with the historical cost element?

■ Have the data been analyzed for performance variation over time (such as technological advances)? 

Are there unambiguous trends between cost and performance over time?

■ Do the data reflect actual costs, proposal values, or negotiated prices and has the type of contract 

been considered? 

Proposal values are usually extremely optimistic and can lead to overly optimistic cost estimates and 

budgets. Furthermore, negotiated prices do not necessarily equate to less optimistic cost estimates.

■ Are sufficient cost data available at the appropriate level of detail to use in statistical measurements?

■ Are cost segregations clear, so that recurring data are separable from nonrecurring data and 

functional elements (manufacturing, engineering) are visible?

■ Have risk and uncertainty for each data element been taken into account? High-risk elements 

usually cause optimistic cost estimates.

■ Have legal or regulatory changes affected cost for the same requirement?

■ When several historical values are available for the same concept, are they in close agreement or are 

they dispersed? 

If they are in close agreement, as long as the definitions agree they should provide valuable insight. If they 

are different, perhaps the issues are not settled, the approaches are still at variance, and historical data may 

not be as useful for estimating current programs’ costs.

Once these questions have been answered, the next step is to assess the validity of the data before they can 

be used to confidently predict future costs.

Validating and Analyzing the Data

The cost analyst must consider the limitations of cost data before using them in an estimate. Historical 

cost data have two predominant limitations:

■ the data represent contractor marketplace circumstances that must be known if they are to have 

future value, and 

■ current cost data eventually become dated.

The first limitation is routinely handled by recording these circumstances as part of the data collection 

task. To accommodate the second limitation, an experienced cost estimator can either adjust the 
GAO-09-3SP Chapter 10 97

data (if applicable) or decide to collect new data. In addition, the contract type to be used in a future 

procurement—for example, firm fixed-price, fixed-price incentive, or cost plus award fee—may differ 

from that of the historical cost data. Although this does not preclude using the data, the analyst must be 

aware of such conditions, so that an informed data selection decision can be made. A cost analyst must 

attempt to address data limitations by

■ ensuring that the most recent data are collected,

■ evaluating cost and performance data together to identify correlation,

■ ensuring a thorough knowledge of the data’s background, and

■ holding discussions with the data provider.

Thus, it is best practice to continuously collect new data so they can be used for making comparisons and 

determining and quantifying trends. This cannot be done without background knowledge of the data. 

This knowledge allows the estimator to confidently use the data directly, modify them to be more useful, 

or simply reject them.

Once the data have been collected, the next step is to create a scatter plot to see what they look like. Scatter 

plotting provides a wealth of visual information about the data, allowing the analyst to quickly determine 

outliers, relationships, and trends. In scatter charts, cost is typically treated as the dependent variable and 

is plotted on the y axis, while various independent variables are plotted on the x axis. These independent 

variables depend on the data collected but are typically technical—weight, lines of code, speed—or 

operational parameters—crew size, flying hours. These statistics provide information about the amount of 

dispersion in the data set, which is important for determining risk.

The cost estimator should first decide which independent variables are most likely to be cost drivers and 

then graph them separately. The extent to which the points are scattered will determine how likely it is 

that each independent variable is a cost driver. The less scattered the points are, the more likely it is that 

the variable is a cost driver. Eventually, the analyst will use statistical techniques to distinguish cost drivers, 

but using scatter charts is an excellent way to reduce their number.

The cost estimator should also examine each scatter chart in unit space to determine if a linear relationship 

exists. Many relationships are not linear; in such cases, the estimator can often perform a transformation 

to make the data linear. If the data appear to be exponential when plotted in unit space, the analyst should 

try plotting the natural log of the independent variable on the y axis. If the data appear to represent a 

power function, the analyst should try plotting the natural log of both the cost and the independent 

variable. In both cases, the goal is to transform the data appropriately to reveal a linear relationship, 

because most cost estimating relationships are based on linear regression.

After analyzing the data through a scatter plot, the estimator should calculate descriptive statistics to 

characterize and describe the data groups. Important statistics include sample size, mean, standard 

deviation, and coefficient of variation. Calculating the mean provides the estimator with the best estimate, 

because it is the average of the historical data. To determine the dispersion within the data set, the 

estimator must calculate the standard deviation. Finally, the estimator should calculate the coefficient of 

variation so that variances between data sets can be compared. 
98 Chapter 10 GAO-09-3SP

The coefficient of variation is calculated by dividing the standard deviation by the mean.38 This provides 

a percentage that can be used to examine which data set has the least variation. Once the statistics have 

been derived, creating visual displays of them helps discern differences among groups. Bar charts, for 

example, are often useful for comparing averages. Histograms can be used to examine the distribution of 

different data sets in relation to their frequency. They can also be used for determining potential outliers. 

(Chapter 11 has more information on statistical approaches.)

Many times, estimates are not based on actual data but are derived by subjective engineering judgment. 

All engineering judgments should be validated before being used in a cost estimate. Validation involves 

cross-checking the results, in addition to analyzing the data and examining the documentation for the 

judgment. Graphs and scatter charts can often help validate an engineering judgment, because they can 

quickly point out any outliers.

It is never a good idea to discard an outlier without first understanding why a data point is outside the 

normal range. An outlier is a data point that is typically defined as falling outside the expected range of 

three standard deviations. Statistically speaking, outliers are rare, occurring only 0.3 percent of the time. 

If a data point is truly an outlier, it should be removed from the data set, because it can skew the results. 

However, an outlier should not be removed simply because it appears too high or too low compared to the 

rest of the data set. Doing so is naïve. Instead, a cost estimator should provide adequate documentation 

as to why an outlier was removed and this documentation should include comparisons to historical data 

that show the outlier is in fact an anomaly. If possible, the documentation should describe why the outlier 

exists; for example, there might have been a strike, a program restructuring, or a natural disaster that 

skewed the data. If the historical data show the outlier is just an extreme case, the cost estimator should 

retain the data point; otherwise, it will appear that the estimator was trying to manipulate the data. This 

should never be done, since all available historical data are necessary for capturing the natural variation 

within programs.

EVM Data Reliability

In chapter 3, we discussed top-level EVM data reliability tasks such as

■ requesting a copy of the EVM system compliance letter showing the contractor’s ability to satisfy 

the 32 guidelines; 

■ requesting a copy of the IBR documentation and final briefing to see what risks were identified and 

what weaknesses, if any, were found; 

■ determining whether EVM surveillance is being done by qualified and independent staff; and 

■ determining the financial accounting status of the contractor’s EVM system to see whether any 

adverse opinions would call into question the reliability of the accounting data.

In addition to these tasks, auditors should perform a sanity check to see if the data even make sense. For 

example, the auditor should review all WBS elements in the CPR to determine whether there are any data 

anomalies such as

38 The coefficient of variation is a useful descriptive statistic for comparing the degree of variation from various data sets, even if the 

means are very different.
GAO-09-3SP Chapter 10 99

■ negative values for BCWS, BCWP, ACWP, estimate at completion (EAC), or budget at 

completion (BAC);

■ large month-to-month performance swings (BCWP) not attributable to technical or schedule 

problems (may indicate cost collection issues);

■ BCWS and BCWP data with no corresponding ACWP;

■ BCWP with no BCWS or ACWP;

■ ACWP with no BCWS or BCWP;

■ large and continuing unexplained variances between ACWP and BCWP;

■ inconsistencies between EAC and BAC (for example, EAC with no BAC or BAC with no EAC);

■ ACWP greater than EAC;

■ BCWP or BCWS exceed the BAC.

Despite the fact that these anomalies should be rare and fully explained in the variance analysis portion 

of the report, unfortunately we have found programs that submit CPRs with these types of errors. Case 

study 32 highlights this issue.

Case Study 32: Data Anomalies, from Cooperative Threat Reduction, 

GAO-06-692

The EVM system the contractor was using to record, predict, and monitor progress 

contained flawed and unreliable data. GAO found serious discrepancies in the data, such as 

improper calculations and accounting errors. For example, from September 2005 through 

January 2006 the contractor’s EVM reports had not captured almost $29 million in actual 

costs for the chemical weapons destruction facility project. EVM current period data were 

not accurate because of historical data corruption, numerous mistakes in accounting 

accruals, and manual budget adjustments. The mistakes underestimated the true cost of 

the project by ignoring cost variances that had already occurred. 

For example, the Moscow project management task had been budgeted at a cost of 

$100,000. According to the January 2006 EVM report, the work was complete, but the 

actual cost was $2.6 million—an overrun of approximately $2.5 million that the EVM 

report failed to capture. Such data were misleading and skewed the project’s overall 

performance. Unreliable EVM data limited DOD’s efforts to accurately measure progress on 

the Shchuch’ye project and estimate its final completion date and cost. 

GAO recommended that the Secretary of Defense direct the Defense Threat Reduction 

Agency, in conjunction with the U.S. Army Corps of Engineers, to ensure that the 

contractor’s EVM system contain valid, reliable data and that the system reflect actual cost 

and schedule conditions; withhold a portion of the contractor’s award fee until the EVM 

system produced reliable data; and require the contractor to perform an IBR after awarding 

the contract for completing Building 101.

GAO, Cooperative Threat Reduction: DOD Needs More Reliable Data to Better 

Estimate the Cost and Schedule of the Shchuch’ye Facility, GAO-06-692

(Washington, D.C.: May 31, 2006).
100 Chapter 10 GAO-09-3SP

Data Normalization

The purpose of data normalization (or cleansing) is to make a given data set consistent with and 

comparable to other data used in the estimate. Since data can be gathered from a variety of sources, they 

are often in many different forms and need to be adjusted before being used for comparison analysis or 

as a basis for projecting future costs. Cost data are adjusted in a process called normalization, stripping 

out the effect of certain external influences. The objective of data normalization is to improve data 

consistency, so that comparisons and projections are more valid and other data can be used to increase the 

number of data points. Data are normalized in several ways.

Cost Units

Cost units primarily adjust for inflation. Because the cost of an item has a time value, it is important to 

know the year in which funds were spent. For example, an item that cost $100 in 1990 is more expensive 

than an item that cost $100 in 2005 because of the effects of inflation over the 15 years that would make 

the 1990 item more expensive when converted to a 2005 equivalent cost. Costs may also be adjusted for 

currency conversions. 

In addition to inflation, the cost estimator needs to understand what the cost represents. For example, 

does it represent only direct labor or does it include overhead and the contractor’s profit? Finally, cost 

data have to be converted to equivalent units before being used in a data set. That is, costs expressed 

in thousands, millions, or billions of dollars must be converted to one format—for example, all costs 

expressed in millions of dollars.

Sizing Units

Sizing units normalize data to common units—for example, cost per foot, cost per pound, dollars per 

software line of code. When normalizing data for unit size, it is very important to define exactly what the 

unit represents: What constitutes a software line of code? Does it include carriage returns or comments? 

The main point is to clearly define what the sizing metric is so that the data can be converted to a 

common standard before being used in the estimate.

Key Groupings

Key groupings normalize data by similar missions, characteristics, or operating environments by cost type 

or work content. Products with similar mission applications have similar characteristics and traits, as do 

products with similar operating environments. For example, space systems exhibit characteristics different 

from those of submarines, but the space shuttle has characteristics distinct from those of a satellite even 

though they may share common features. Costs should also be grouped by type. For example, costs should 

be broken out between recurring and nonrecurring or fixed and variable costs.

Technology Maturity

Technology maturity normalizes data for where a program is in its life cycle; it also considers learning 

and rate effects. The first unit of something would be expected to cost more than the 1,000th unit, just 

as a system procured at one unit per year would be expected to cost more per unit than the same system 

procured at 1,000 units per year. Technology normalization is the process of adjusting cost data for 

productivity improvements resulting from technological advancements that occur over time. 
GAO-09-3SP Chapter 10 101

In effect, technology normalization is the recognition that technology continually improves, so a cost 

estimator must make a subjective attempt to measure the effect of this improvement on historical program 

costs. For instance, an item developed 10 years ago may have been considered state of the art and the costs 

would be higher than normal. Today, that item may be available off the shelf and therefore the costs would 

be considerably less. 

Therefore, technology normalization is the ability to forecast technology by predicting the timing and 

degree of change of technological parameters associated with the design, production, and use of devices. 

Being able to adjust the cost data to reflect where the item is in its life cycle, however, is very subjective, 

because it requires identifying the relative state of technology at different points in time.

Homogeneous Groups

Using homogeneous groups normalizes for differences between historical and new program WBS 

elements in order to achieve content consistency. To do this type of normalization, a cost estimator 

needs to gather cost data that can be formatted to match the desired WBS element definition. This may 

require adding and deleting certain items to get an apples-to-apples comparison. A properly defined WBS 

dictionary is necessary to avoid inconsistencies. 

Recurring and Nonrecurring Costs

Embedded within cost data are recurring and nonrecurring costs. These are usually estimated separately 

to keep one-time nonrecurring costs from skewing the costs for recurring production units. For this 

reason, it is important to segregate cost data into nonrecurring and recurring categories.

Nonrecurring Costs

SCEA defines nonrecurring costs as the elements of development and investment costs that generally 

occur only once in a system’s life cycle. They include all the effort required to develop and qualify an 

item, such as defining its requirements and its allocation, design, analysis, development, qualification, and 

verification. Costs for the following are generally nonrecurring: 

■ manufacturing and testing development units, both breadboard and engineering, for hardware, as 

well as qualification and life-test units; 

■ retrofitting and refurbishing development hardware for requalification; 

■ developing and testing virtually all software before beginning routine system operation; 

nonrecurring integration and test efforts usually end when qualification tests are complete;

■ providing services and some hardware, such as engineering, before and during critical design 

review; 

■ developing, acquiring, producing, and checking all tooling, ground handling, software, and 

support equipment and test equipment.

Recurring Costs

As defined by SCEA, recurring costs are incurred for each item produced or each service performed. For 

example, the costs associated with producing hardware—that is, manufacturing and testing, providing 

engineering support for production, and supporting that hardware with spare units or parts—are 
102 Chapter 10 GAO-09-3SP

recurring costs. Recurring integration and testing, including the integration and acceptance testing of 

production units at all WBS levels, also represent recurring costs. In addition, refurbishing hardware for 

operational or spare units is a recurring cost, as is maintaining test equipment and production support 

software. In contrast, maintaining system operational software, although recurring in nature, is often 

considered part of operating and support costs, which might also have nonrecurring components. 

Similar to nonrecurring and recurring costs are fixed and variable costs. Fixed costs are static, regardless of 

the number of quantities to be produced. An example of a fixed cost is the cost to rent a facility. A variable 

cost is directly affected by the number of units produced and includes such things as the cost of electricity 

or overtime pay. Knowing what the data represent is important for understanding anomalies that can 

occur as the result of production unit cuts. 

The most important reason for differentiating recurring from nonrecurring costs is in their application to 

learning curves. Simply put, learning curve theory applies only to recurring costs. Cost improvement or 

learning is generally associated with repetitive actions or processes, such as those directly tied to producing 

an item again and again. Categorizing as recurring or variable costs that are affected by the quantity 

of units being produced adds more clarity to the data. An analyst who knows only the total cost of 

something does not know how much of that cost is affected by learning.

Inflation Adjustments

In the development of an estimate, cost data must be expressed in like terms. This is usually accomplished 

by inflating or deflating cost data to express them in a base year that will serve as a point of reference for 

a fixed price level. Applying inflation is an important step in cost estimating. If a mistake is made or the 

inflation amount is not correct, cost overruns can result, as case study 33 illustrates.

Case Study 33: Inflation, from Defense Acquisitions, GAO-05-183

Inflation rates can significantly affect ship budgets. Office of the Secretary of Defense 

(OSD) and OMB inflation indexes are based on a forecast of the implicit price deflator for 

the gross domestic product. Until recently, the Navy had used OSD and OMB inflation rates; 

shipbuilding industry rates were historical higher. As a result, contracts were signed and 

executed using industry-specific inflation rates while budgets were based on the lower 

inflation rates, creating a risk of cost growth from the outset. For the ships reviewed, this 

difference in inflation rates explained 30 percent of the $2.1 billion cost growth. The Navy 

had changed its inflation policy in February 2004, directing program offices to budget with 

what the Navy believed were more realistic inflation indexes, anticipating that this would 

help curtail requests for prior-year completion funds.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

Applying inflation correctly is necessary if the cost estimate is to be credible. In simple terms, inflation 

reflects the fact that the cost of an item usually continues to rise over time. Inflation rates are used to 

convert a cost from its current year into a constant base year so that the effects of inflation are removed. 

When cost estimates are stated in base-year dollars, the implicit assumption is that the purchasing power 

of the dollar has remained unchanged over the period of the program being estimated. Cost estimates are 
GAO-09-3SP Chapter 10 103

normally prepared in constant dollars to eliminate the distortion that would otherwise be caused by pricelevel changes. This requires the transformation of historical or actual cost data into constant dollars. 

For budgeting purposes, however, the estimate must be expressed in then-year dollars to reflect the 

program’s projected annual costs by appropriation. This requires applying inflation to convert from baseyear to then-year dollars. Cost estimators must make assumptions about what inflation indexes to use, 

since any future inflation index is uncertain. In cases in which inflation decreases over time, applying the 

wrong inflation rate will result in a higher cost estimate. Worse is the situation in which the inflation is 

higher than projected, resulting in costs that are not sufficient to keep pace with inflation, as illustrated 

in case study 33. Thus, it is imperative that inflation assumptions be well documented and that the 

cost estimator always perform uncertainty and sensitivity analysis to study the effects of changes on the 

assumed rates. 

Selecting the Proper Indexes

The cost estimator will not have to construct an index to apply inflation but will select one to apply 

to cost data. Often, the index is directed by higher authority, such as OMB. In this way, all programs 

can be compared and aggregated with the same escalation rate, since they are all being executed in the 

same economic circumstances. This doesn’t mean that the forward escalation rates are correct—in fact, 

escalation rates are difficult to forecast—but that program comparisons will at least not be confused by 

different assumptions about escalation. When the index is not directed, a few general guidelines can 

help the cost estimator select the correct index. Because all inflation indexes measure the average rate 

of inflation for a particular market basket of goods, the objective in making a choice is to select the one 

whose market basket most closely matches the program to be estimated. The key is to use common sense 

and objective judgment. For example, the consumer price index would be a poor indicator of inflation 

for a new fighter aircraft, because the market baskets obviously do not match. Labor escalation would be 

affected by different factors than, say, fuel or steel costs. Although the selected index will never exactly 

match the market basket of costs, the closer the match, the better the estimate. 

Weighted indexes are used to convert constant, base-year, dollars to then-year dollars and vice versa. Raw 

indexes are used to change the economic base of constant dollars from one base year to another. Contract 

prices are stated in then-year dollars, and weighted indexes are appropriate for converting them to baseyear dollars. Published historical cost data are frequently, but not always, normalized to a common base 

year, and raw indexes are appropriate for changing the base year to match that of the program being 

estimated. It is important that the cost estimator determine what year dollars cost data are expressed in, so 

that normalization for inflation can be done properly.

Schedule risk can affect the magnitude of escalation in a cost estimate. The escalation dollars are often 

estimated by applying a monthly escalation rate (computed so that compounding monthly values equates 

to the forecasted annual rate) to dollars forecasted to be spent in each month. If the schedule is delayed, 

a dollar that would have been escalated by, say, 30 months might now be escalated for 36 months. Even 

if the cost estimate in today’s dollars is an accurate estimate, a schedule slip would affect the amount of 

escalation.

In addition, the question of escalating the contingency reserve arises. Some cost estimating systems 

calculate the contingency on base-year dollars but do not escalate the contingency, perhaps because 
104 Chapter 10 GAO-09-3SP

they do not have a way to determine when the dollars will be spent. In a cost risk analysis, in contrast, 

the contingency reserve is computed during the simulation using the risk in the line-item costs. If 

the simulated line-item costs are then subjected to escalation during the same simulation, the process 

effectively escalates the contingency. This is appropriate, since contingency money is just more money 

needed to be spent on the statement of work, and it should be affected by escalation as is any other money 

spent.

Data Documentation

After the data have been collected, analyzed, and normalized, they must be documented and stored 

for future use. One way to keep a large amount of historical data viable is to continually supplement 

them with every new system’s actual return costs and with every written vendor quote or new contract. 

Although data have many sources, the predominant sources are the manufacturers who make the item 

or similar items. It can take years for a cost estimator to develop an understanding of these sources and 

to earn the trust of manufacturers regarding the use of their proprietary and business-sensitive data. 

Once trust has been established and maintained for some time, the cost estimator can normally expect a 

continual flow of useful data.

All data collection activities must be documented as to source, work product content, time, units, and 

assessment of accuracy and reliability. Comprehensive documentation during data collection greatly 

improves quality and reduces subsequent effort in developing and documenting the estimate. The data 

collection format should serve two purposes. First, the format should provide for the full documentation 

and capture of information to support the analysis. Second, it should provide for standards that will aid in 

mapping other forms of cost data. 

Previously documented cost estimates may provide useful data for a current estimate. Relying on previous 

estimates can save the cost estimator valuable time by eliminating the need to research and conduct 

statistical analyses that have already been conducted. For example, a documented program estimate 

may provide the results of research on contractor data, identification of significant cost drivers, or actual 

costs, all of which are valuable to the cost estimator. Properly documented estimates describe the data 

used to estimate each WBS element, and this information can be used as a good starting point for the 

new estimate. Moreover, relying on other program estimates can be valuable in understanding various 

contractors and providing cross-checks for reasonableness. 

Because many cost documents are secondary sources of information, the cost estimator should be 

cautious. When using information from documented cost estimates, the analyst should fully understand 

the data. For example, if a factor was constructed from CPRs, the cost estimator should ask the following 

questions to see if the data are valid for the new program:

■ What was the base used in the ratio?

■ Are the WBS elements consistent with those of the system being estimated—for example, is data 

management included in the data or the systems engineering and program management element? 

■ Was the factor computed from the ACWP or the EAC? 

■ What percentage complete is the contract?
GAO-09-3SP Chapter 10 105

Thus, previous estimates can provide the cost estimator with valuable data and can also save time, since 

they provide a structure from which to develop the new cost estimate. They also help avoid reinventing the 

wheel, since the estimator can leverage off the work of others. However, the cost estimator will still have to 

perform follow-on work before fully relying on these data.

7. Best Practices Checklist: Data

 As the foundation of an estimate, data 

3 Have been gathered from historical actual cost, schedule and program, 

and technical sources;

3 Apply to the program being estimated;

3 Have been analyzed for cost drivers;

3 Have been collected from primary sources, if possible, and secondary 

sources as the next best option, especially for cross-checking results;

3 Have been adequately documented as to source, content, time, units, 

assessment of accuracy and reliability, and circumstances affecting the 

data;

3 Have been continually collected, protected, and stored for future use;

3 Were assembled as early as possible, so analysts can participate in site 

visits to understand the program and question data providers.

 Before being used in a cost estimate, the data were

3 Fully reviewed to understand their limitations and risks;

3 Segregated into nonrecurring and recurring costs;

3 Validated, using historical data as a benchmark for reasonableness;

3 Current and found applicable to the program being estimated;

3 Analyzed with a scatter plot to determine trends and outliers; 

3 Analyzed with descriptive statistics;

3 Normalized to account for cost and sizing units, mission or application, 

technology maturity, and content so they are consistent for 

comparisons;

3 Normalized to constant base-year dollars to remove the effects of 

inflation, and the inflation index was documented and explained. 
106 Chapter 10 GAO-09-3SP
GAO-09-3SP Chapter 11 107

Developing a Point Estimate

Chapter 11

In this chapter, we discuss step 7 in the high-quality estimating process. Step 7 pulls all the information 

together to develop the point estimate—the best guess at the cost estimate, given the underlying data. 

High-quality cost estimates usually fall within a range of possible costs, the point estimate being between 

the best and worst case extremes. (We explain in chapter 14 how to develop this range of costs using risk 

and uncertainty analysis.) The cost estimator must perform several activities to develop a point estimate: 

■ develop the cost model by estimating each WBS element, using the best methodology, from the 

data collected;

■ include all estimating assumptions in the cost model;

■ express costs in constant-year dollars;

■ time-phase the results by spreading costs in the years they are expected to occur, based on the 

program schedule; and

■ add the WBS elements to develop the overall point estimate.

Having developed the overall point estimate, the cost estimator must then

■ validate the estimate by looking for errors like double counting and omitted costs and ensuring 

that estimates are comprehensive, accurate, well-documented, and credible (more information on 

validation is in chapter 15);

■ compare the estimate against the independent cost estimate and examine where and why there are 

differences;

■ perform cross-checks on cost drivers to see if results are similar; and

■ update the model as more data become available or as changes occur and compare the results 

against previous estimates.

We have already discussed how to develop a WBS and GR&As, collect and normalize the data into 

constant base-year dollars, and time-phase the results. Once all the data have been collected, analyzed, and 

validated, the cost estimator must select a method for developing the cost estimate. 

Cost Estimating Methods

The three commonly used methods for estimating costs are analogy, engineering build-up, and 

parametric. An analogy uses the cost of a similar program to estimate the new program and adjusts for 

differences. The engineering build-up method develops the cost estimate at the lowest level of the WBS, 

one piece at a time, and the sum of the pieces becomes the estimate. The parametric method relates cost to 

one or more technical, performance, cost, or program parameters, using a statistical relationship.
108 Chapter 11 GAO-09-3SP

Which method to select depends on where the program is in its life cycle. Early in the program, definition 

is limited and costs may not have accrued. Once a program is in production, cost and technical data from 

the development phase can be used to estimate the remainder of the program. Table 11 gives an overview 

of the strengths, weaknesses, and applications of the three methods. 

Table 11: Three Cost Estimating Methods Compared

Method Strength Weakness Application

Analogy ■ Requires few data 

■ Based on actual data

■ Reasonably quick

■ Good audit trail 

■ Subjective adjustments

■ Accuracy depends on 

similarity of items

■ Difficult to assess effect of 

design change 

■ Blind to cost drivers

■ When few data are 

available

■ Rough-order-ofmagnitude estimate

■ Cross-check

Engineering 

build-up 

■ Easily audited

■ Sensitive to labor rates

■ Tracks vendor quotes

■ Time honored

■ Requires detailed design 

■ Slow and laborious

■ Cumbersome 

■ Production estimating

■ Software development

■ Negotiations

Parametric ■ Reasonably quick

■ Encourages discipline

■ Good audit trail

■ Objective, little bias

■ Cost driver visibility

■ Incorporates real-world 

effects (funding, technical, 

risk)

■ Lacks detail

■ Model investment

■ Cultural barriers

■ Need to understand 

model’s behavior

■ Budgetary estimates

■ Design-to-cost trade 

studies

■ Cross-check

■ Baseline estimate

■ Cost goal allocations

Source: © 2003, MCR, LLC, “Cost Estimating: The Starting Point of EVM.”

Other cost estimating methods include

■ expert opinion, which relies on subject matter experts to give their opinion on what an element 

should cost;39

■ extrapolating, which uses actual costs and data from prototypes to predict the cost of future 

elements; and

■ learning curves, which is a common form of extrapolating from actual costs.

In the sections below, we describe these methods and their advantages and disadvantages. Finally, we 

discuss how to pull all the methods together to develop the point estimate.

Analogy Cost Estimating Method

An analogy takes into consideration that no new program, no matter how state of the art it may be 

technologically, represents a totally new system. Most new programs evolve from programs already fielded 

that have had new features added on or that simply represent a new combination of existing components. 

The analogy method uses this concept for estimating new components, subsystems, or total programs. 

That is, an analogy uses actual costs from a similar program with adjustments to account for differences 

39Expert opinion, also known as engineering judgment, is commonly applied to fill gaps in a relatively detailed WBS when one or 

more experts are the only qualified source of information, particularly in matters of specific scientific technology. 
GAO-09-3SP Chapter 11 109

between the requirements of the existing and new systems. A cost estimator typically uses this method 

early in a program’s life cycle, when insufficient actual cost data are available but the technical and 

program definition is good enough to make the necessary adjustments. 

Adjustments should be made as objectively as possible, by using factors (sometimes scaling parameters) 

that represent differences in size, performance, technology, or complexity. The cost estimator should 

identify the important cost drivers, determine how the old item relates to the new item, and decide how 

each cost driver affects the overall cost. All estimates based on the analogy method, however, must pass 

the “reasonable person” test—that is, the sources of the analogy and any adjustments must be logical, 

credible, and acceptable to a reasonable person. In addition, since analogies are one-to-one comparisons, 

the historical and new systems should have a strong parallel. 

Analogy relies a great deal on expert opinion to modify the existing system data to approximate the new 

system. If possible, the adjustments should be quantitative rather than qualitative, avoiding subjective 

judgments as much as possible. An analogy is often used as a cross-check for other methods. Even when 

an analyst is using a more detailed cost estimating technique, an analogy can provide a useful sanity 

check. Table 12 shows how an analogy works.

Table 12: An Example of the Analogy Cost Estimating Method

Parameter Existing system New system

Cost of new system

(assuming a linear relationship)

Engine F-100 F-200

Thrust 12,000 lbs 16,000 lbs

Cost $5.2 million X (16,000/12,000) x $5.2 million = $6.9 million

Source: © 2003, Society of Cost Estimating and Analysis (SCEA), “Costing Techniques.” 

The equation in table 12 implicitly assumes a linear relationship between engine cost and amount of 

thrust. However, there should be a compelling scientific or engineering reason why an engine’s cost is 

directly proportional to its thrust. Without more data (or an expert on engine costs), it is hard to know 

what parameters are the true drivers of cost. Therefore, when using the analogy method, it is important 

that the estimator research and discuss with program experts the reasonableness of technical program 

drivers to determine whether they are significant cost drivers.

The analogy method has several advantages: 

■ It can be used before detailed program requirements are known.

■ If the analogy is strong, the estimate will be defensible.

■ An analogy can be developed quickly and at minimum cost.

■ The tie to historical data is simple enough to be readily understood.

Analogies also have some disadvantages:

■ An analogy relies on a single data point.

■ It is often difficult to find the detailed cost, technical, and program data required for analogies.

■ There is a tendency to be too subjective about the technical parameter adjustment factors.
110 Chapter 11 GAO-09-3SP

The last disadvantage can be best explained with an example. If a cost estimator assumes that a 

new component will be 20 percent more complex but cannot explain why, this adjustment factor is 

unacceptable. The complexity must be related to the system’s parameters, such as that the new system will 

have 20 percent more data processing capacity or will weigh 20 percent more. Case study 34 highlights 

what can happen when technical parameter assumptions are too optimistic.

Case Study 34: Cost Estimating Methods, from Space Acquisitions, 

GAO-07-96

In 2004, Advanced Extremely High Frequency (AEHF) satellite program decision makers 

relied on the program office cost estimate rather than the independent estimate the CAIG 

developed to support the production decision. The program office estimated that the 

system would cost about $6 billion, on the assumption that AEHF would have 10 times 

more capacity than Milstar, the predecessor satellite, at half the cost and weight. However, 

the CAIG concluded that the program could not deliver more data capacity at half the 

weight, given the state of the technology. In fact, the CAIG believed that to get the desired 

increase in data rate, the weight would have to increase proportionally. As a result, the 

CAIG estimated that AEHF would cost $8.7 billion and predicted a $2.7 billion cost overrun. 

The CAIG relied on weight data from historical satellites to estimate the program’s 

cost, because it considered weight to be the best cost predictor for military satellite 

communications. The historical data from the AEHF contractor showed that the weight had 

more than doubled since the program began and that the majority of the weight growth 

was in the payload. The Air Force also used weight as a cost predictor but attributed the 

weight growth to structural components rather than the more costly payload portion of 

the satellite. The CAIG stated that major cost growth was inevitable from the program start 

because historical data showed that it was possible to achieve a weight reduction or an 

increase in data capacity but not both at the same time.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Engineering Build-Up Cost Estimating Method

The engineering build-up cost estimating method builds the overall cost estimate by summing or “rollingup” detailed estimates done at lower levels of the WBS. Because the lower-level estimating associated with 

the build-up method uses industrial engineering principles, it is often referred to as engineering build-up 

and is sometimes referred to as a grass-roots or bottom-up estimate. 

An engineering build-up estimate is done at the lowest level of detail and consists of labor and materials 

costs that have overhead and fee added to them. In addition to labor hours, a detailed parts list is required. 

Once in hand, the material parts are allocated to the lowest WBS level, based on how the work will be 

accomplished. In addition, quantity and schedule have to be considered in order to capture the effects 

of learning. Typically, cost estimators work with engineers to develop the detailed estimates. The cost 

estimator’s focus is to get detailed information from the engineer in a way that is reasonable, complete, and 

consistent with the program’s ground rules and assumptions. The cost estimator must find additional data 

to validate the engineer’s estimates. 
GAO-09-3SP Chapter 11 111

An engineering build-up method is normally used during the program’s production, because the 

program’s configuration has to be stabilized, and actual cost data are required to complete the estimate. 

The underlying assumption of this method is that historical costs are good predictors of future costs. 

The premise is that data from the development phase can be used to estimate the cost for production. 

As illustrated in table 13, the build-up method is used when an analyst has enough detailed information 

about building an item—such as number of hours and number of parts—and the manufacturing process 

to be used.

Table 13: An Example of the Engineering Build-Up Cost Estimating Method

Problem Similar aircraft Solution Result

Estimate sheet metal 

cost of the inlet nacelle 

for a new aircraft

F/A-18 inlet nacelle Apply historical F/A-18 

variance for touch labor 

effort and

apply support labor 

factor to adjust 

estimated touch labor 

hours

2,000 hours x 1.2 = 2,400 

touch labor hours and 

2,400 labor hours x 1.48 = 

3,522 labor hours (touch labor 

plus support labor) estimate 

for new aircraft

Standard hours to 

produce a new nacelle 

are estimated at 2,000 

for touch labor; adjust 

to reflect experience 

of similar aircraft and 

support labor effort

F/A-18 inlet nacelle 

experienced a 20% 

variance in touch 

labor effort above the 

industrial engineering 

standard. In addition, 

F/A-18 support labor 

was equal to 48% of the 

touch labor hours 

Average labor rates would 

then be used to convert these 

total labor hours into costs

Source: © 2003, Society of Cost Estimating and Analysis (SCEA), “Costing Techniques.” 

Because of the high level of detail, each step of the work flow should be identified, measured, and tracked, 

and the results for each outcome should be summed to make the point estimate.

The several advantages to the build-up technique include

■ the estimator’s ability to determine exactly what the estimate includes and whether anything was 

overlooked,

■ its unique application to the specific program and manufacturer,

■ that it gives good insight into major cost contributors, and

■ easy transfer of results to other programs.

Some disadvantages of the engineering build-up method are that

■ it can be expensive to implement and it is time consuming,

■ it is not flexible enough to answer what-if questions,

■ new estimates must be built for each alternative,

■ the product specification must be well known and stable, 

■ all product and process changes must be reflected in the estimate, 
112 Chapter 11 GAO-09-3SP

■ small errors can grow into larger errors during the summation, and

■ some elements can be omitted by accident.

Parametric Cost Estimating Method

In the parametric method, a statistical relationship is developed between historical costs and program, 

physical, and performance characteristics. The method is sometimes referred to as a top-down approach. 

Types of physical characteristics used for parametric estimating are weight, power, and lines of code. 

Other program and performance characteristics include site deployment plans for information technology 

installations, maintenance plans, test and evaluation schedules, technical performance measures, and crew 

size. These are just some examples of what could be a cost driver for a particular program. 

Sources for these cost drivers are often found in the technical baseline, cost analysis requirements 

document or cost analysis data requirement. The important thing is that the attributes used in a 

parametric estimate should be cost drivers of the program. The assumption driving the parametric 

approach is that the same factors that affected cost in the past will continue to affect future costs. This 

method is often used when little is known about a program except for a few key characteristics like weight 

or volume. 

Using a parametric method requires access to historical data, which may be difficult to obtain. If the data 

are available, they can be used to determine the cost drivers and to provide statistical results and can be 

adjusted to meet the requirements of the new program. Unlike an analogy, parametric estimating relies 

on data from many programs and covers a broader range. Confidence in a parametric estimate’s results 

depends on how valid the relationships are between cost and the physical attributes or performance 

characteristics. Using this method, the cost estimator must always present the related statistics, 

assumptions, and sources for the data.

The goal of parametric estimating is to create a statistically valid cost estimating relationship using 

historical data. The parametric CER can then be used to estimate the cost of the new program by entering 

its specific characteristics into the parametric model. CERs established early in a program’s life cycle 

should be continually revisited to make sure they are current and the input range still applies to the new 

program. In addition, parametric CERs should be well documented, because serious estimating errors 

could occur if the CER is improperly used. 

Parametric techniques can be used in a wide variety of situations, ranging from early planning estimates 

to detailed contract negotiations. It is always essential to have an adequate number of relevant data 

points, and care must be taken to normalize the dataset so that it is consistent and complete. In software, 

the development environment—that is, the extent to which the requirements are understood and the 

strength of the programmers’ skill and experience—is usually the major cost driver. Because parametric 

relationships are often used early in a program, when the design is not well defined, they can easily be 

reflected in the estimate as the design changes simply by adjusting the values of the input parameters. 

It is important to make sure that the program attributes being estimated fall within (or, at least, not far 

outside) the CER dataset. For example, if a new software program was expected to contain 1 million 

software lines of code and the data points for a software CER were based on programs with lines of code 

ranging from 10,000 to 250,000, it would be inappropriate to use the CER to estimate the new program.
GAO-09-3SP Chapter 11 113

To develop a parametric CER, cost estimators must determine the cost drivers that most influence cost. 

After studying the technical baseline and analyzing the data through scatter charts and other methods, 

the cost estimator should verify the selected cost drivers by discussing them with engineers. The CER 

can then be developed with a mathematical expression, which can range from a simple rule of thumb (for 

example, dollars per pound) to a complex regression equation. 

The more simplified CERs include rates, factors, and ratios. A rate uses a parameter to predict cost, using 

a multiplicative relationship. Since rate is defined to be cost as a function of a parameter, the units for 

rate are always dollars per something. The rate most commonly used in cost estimating is the labor rate, 

expressed in dollars per hour.

A factor uses the cost of another element to estimate a new cost using a multiplier. Since a factor is defined 

to be cost as a function of another cost, it is often expressed as a percentage. For example, travel costs may 

be estimated as 5 percent of program management costs.

A ratio is a function of another parameter and is often used to estimate effort. For example, the cost to 

build a component could be based on the industry standard of 20 hours per subcomponent. 

Rates, factors, and ratios are often the result of simple calculations (like averages) and many times do not 

include statistics. Table 14 contains a parametric cost estimating example.

Table 14: An Example of the Parametric Cost Estimating Method

Program attribute Calculation

A cost estimating relationship (CER) for site activation (SA) is a 

function of the number of workstations (NW)

SA = $82,800 + ($26,500 x NW)

Data range for the CER 7–47 workstations based on 11 data points

Cost to site activate a program with 40 workstations $82,800 + ($26,500 x 40) = $1,142,800

Source: © 2003, Society of Cost Estimating and Analysis (SCEA), “Costing Techniques.” 

In table 14, the number of workstations is the cost driver. The equation is linear but has both a fixed 

component (that is, $82,800) and a variable component (that is, $26,500 x NW). 

In addition, the range of the data is from 7 to 47 workstations, so it would be inappropriate to use this 

CER for estimating the activation cost of a site with as few as 2 or as many as 200 workstations. 

In fact, at one extreme, the CER estimates a cost of $82,800 for no workstation installations, which is 

not logical. Although we do not show any CER statistics for this example, the CERs should always be 

presented with their statistics. The reason for this is to enable the cost estimator to understand the level of 

variation within the data and model its effect with uncertainty analysis. 

CERs should be developed using regression techniques, so that statistical inferences may be drawn. 

To perform a regression analysis, the first step is to determine what relationship exists between cost 

(dependent variable) and its various drivers (independent variables). This relationship is determined by 

developing a scatter chart of the data. If the data are linear, they can be fit by a linear regression. If they are 

not linear and transformation of the data does not produce a linear fit, nonlinear regression can be used. 

The independent variables should have a high correlation with cost and should be logical. 
114 Chapter 11 GAO-09-3SP

For example, software complexity can be considered a valid driver of the cost of developing software. The 

ultimate goal is to create a fit with the least variation between the data and the regression line. This process 

helps minimize the statistical error or uncertainty brought on by the regression equation. 

The purpose of the regression is to predict with known accuracy the next real-world occurrence of the 

dependent variable (or the cost), based on knowledge of the independent variable (or some physical, 

operational, or program variable). Once the regression is developed, the statistics associated with the 

relationship must be examined to see if the CER is a strong enough predictor to be used in the estimate. 

Most statistics can be easily generated with the regression analysis function of spreadsheet software. 

Among important regression statistics are 

■ R-squared, 

■ statistical significance, 

■ the F statistic, and 

■ the t statistic.

R-squared 

The R-squared (R2) value measures the strength of the association between the independent and 

dependent (or cost) variables. The R2 value ranges between 0 and 1, where 0 indicates that there is no 

relationship between cost and its independent variable, and 1 means that there is a perfect relationship 

between them. Thus, the higher R2 is the better. An R2 of 91 percent in the example in table 14, for 

example, would mean that the number of workstations (NW) would explain 91 percent of the variation in 

site activation costs, indicating that it is a very good cost driver.

Statistical Significance

Statistical significance is the most important factor for deciding whether a statistical relationship is 

valid. An independent variable can be considered statistically significant if there is small probability 

that its corresponding coefficient is equal to zero, because a coefficient of zero would indicate that 

the independent variable has no relationship to cost. Thus, it is desirable that the probability that the 

coefficient is equal to zero be as small as possible. How small is denoted by a predetermined value called 

the significance level. For example, a significance level of .05 would mean there was a 5 percent probability 

that a variable was not statistically significant. Statistical significance is determined by both the regression 

as a whole and each regression variable. 

F Statistic

The F statistic is used to judge whether the CER as a whole is statistically significant by testing to see 

whether any of the variables’ coefficients are equal to zero. The F statistic is defined as the ratio of the 

equation’s mean squares of the regression to its mean squared error, also called the residual. The higher the 

F statistic is, the better the regression, but it is the level of significance that is important.

t Statistic

The t statistic is used to judge whether individual coefficients in the equation are statistically significant. It 

is defined as the ratio of the coefficient’s estimated value to its standard deviation. As with the F statistic, 

the higher the t statistic is, the better, but it is the level of significance that is important.
GAO-09-3SP Chapter 11 115

The Parametric Method: Further Considerations 

The four statistics described above are just some of the statistical analyses that can be used to validate a 

CER. (For more information on statistics or hardware cost estimating, a good reference is the Parametric 

Estimating Handbook.

40) Once the statistics have been evaluated, the cost estimator picks the best CER—

that is, the one with the least variation and the highest correlation to cost. 

The final step in developing the CER is to validate the results, using a data set different from the one used 

to generate the equation, to see if the results are similar. Again, it is important to use a CER developed 

from programs whose variables are within the same data range as those used to develop the CER. 

Deviating from the CER variable input range could invalidate the relationship and skew the results. We 

note several other pitfalls associated with CERs.

Always question the source of the data underlying the CER. Some CERs may be based on data that are 

biased by unusual events like a strike, hurricane, or major technical problems that required a lot of rework. 

To mitigate this risk, it is essential to understand the data the CER is based on and, if possible, to use 

other historical data to check the validity of the results.

All equations should be checked for common sense to see if the relationship described by the CER is 

reasonable. This helps avoid the mistake that the relationship adequately describes one system but does not 

apply to the one being estimated.

Normalizing the data to make them consistent is imperative to good results. All cost data should be 

converted to constant base years. In addition, labor and material costs should be broken out separately, 

since they may require different inflation factors to convert them to constant dollars. Moreover, 

independent variables should be converted into like units for various physical characteristics such as 

weight, speed, and length.

Historical cost data may have to be adjusted to reflect similar accounting categories, which might be 

expressed differently from one company to another. 

It is important to fully understand all CER modeling assumptions and to examine the reliability of the 

dataset, including its sources, to see if they are reasonable.

Among the several advantages to parametric cost estimating are its

■ Versatility: If the data are available, parametric relationships can be derived at any level, whether 

system or subsystem component. And as the design changes, CERs can be quickly modified and 

used to answer what-if questions about design alternatives.

■ Sensitivity: Simply varying input parameters and recording the resulting changes in cost can 

produce a sensitivity analysis. 

■ Statistical output: Parametric relationships derived from statistical analysis generally have both 

objective measures of validity (statistical significance of each estimated coefficient and of the model 

40 See International Society of Parametric Analysts, Parametric Estimating Handbook©, 4th ed. (Vienna, Va.: ISPA/SCEA Joint 

Office, 2008). www.ispa-cost.org/newbook.htm. The handbook and its appendixes detail, and give examples of, how to develop, 

test, and evaluate CERs.
116 Chapter 11 GAO-09-3SP

as a whole) and a calculated standard error that can be used in risk analysis. This information can 

be used to provide a confidence level for the estimate, based on the CER’s predictive capability. 

■ Objectivity: CERs rely on historical data that provide objective results. This increases the estimate’s 

defensibility.

Disadvantages to parametric estimating include

■ Database requirements: The underlying database must be consistent and reliable. It may be timeconsuming to normalize the data or to ensure that the data were normalized correctly, especially if 

someone outside the estimator’s team developed the CER. Without understanding how the data 

were normalized, the analyst has to accept the database on faith—sometimes called the black-box 

syndrome, in which the analyst simply plugs in numbers and unquestioningly accepts the results. 

Using a CER in this manner can increase the estimate’s risk.

■ Currency: CERs must represent the state of the art; that is, they must be updated to capture the 

most current cost, technical, and program data.

■ Relevance: Using data outside the CER range may cause errors, because the CER loses its 

predictive ability for data outside the development range.

■ Complexity: Complicated CERs (such as nonlinear CERs) may make it difficult for others to 

readily understand the relationship between cost and its independent variables.

Parametric Cost Models

Many cost estimating models are based on parametric methods. They may estimate hardware or 

software costs. Depending on the model, the database may contain cost, technical, and programmatic 

data at the system, component, and subcomponent level. Parametric models typically consist of several 

interrelated CERs and are often computerized. They may involve extensive use of cost-to-noncost CERs, 

multiple independent variables related to a single cost effect, or independent variables defined in terms of 

weapon system performance or design characteristics rather than more discrete material requirements or 

production processes. Information technology databases and computer modeling may be used in these 

types of parametric cost estimating systems.

When using parametric models, many times the underlying data are proprietary, so access to the raw data 

may not be available. When the inputs to the parametric models are qualitative, as often happens, they 

should be objectively assessed. In addition, many parameters should be selected to tailor the model to the 

specific hardware or software product that is being estimated. Therefore, it is also important to calibrate 

the parametric model to best reflect the particular situation or environment in which the product will be 

developed. Finally, the model should be validated using historical data to determine how well it predicts 

costs. 

Parametric models are always useful for cross-checking the reasonableness of a cost estimate that is derived 

by other means. As a primary estimating method, parametric models are most appropriate during the 

engineering concept phase when requirements are still somewhat unclear and no bill of materials exists. 

When this is the situation, it is imperative that the parametric model is based on historical cost data 

and that the model is calibrated to those data. To ensure that the model is a good predictor of costs, it 

should demonstrate that it actually reflects or replicates known data to a reasonable degree of accuracy. In 
GAO-09-3SP Chapter 11 117

addition, the model should demonstrate that the cost-to-noncost estimating relationships are logical and 

that the data used for the parametric model can be verified and traced back to source documentation. 

Using parametric cost models has several advantages: 

■ They can be adjusted to best fit the hardware or software being estimated. 

■ Cost estimates are based on a database of historical data.

■ They can be calibrated to match a specific development environment.

Their disadvantages are that

■ their results depend on the quality of the underlying database,

■ they require many inputs that may be subjective, and

■ accurate calibration is required for valid results.

Expert Opinion

Expert opinion is generally considered too subjective but can be useful in the absence of data. It is possible 

to alleviate this concern by probing further into the experts’ opinions to determine if real data back them 

up. If so, the analyst should attempt to obtain the data and document the source. 

The cost estimator’s interviewing skills are also important for capturing the experts’ knowledge so that the 

information can be used properly. However, cost estimators should never ask experts to estimate the costs 

for anything outside the bounds of their expertise, and they should always validate experts’ credentials 

before relying on their opinions.

The advantages of using an expert’s opinion are that

■ it can be used when no historical data are available;

■ it takes minimal time and is easy to implement, once experts are assembled;

■ an expert may give a different perspective or identify facets not previously considered, leading to a 

better understanding of the program; 

■ it can help in cross-checking for CERs that require data significantly beyond the data range;

■ it can be blended with other estimation techniques within the same WBS element; and 

■ it can be applied in all acquisition phases.

Disadvantages associated with using an expert’s opinion include

■ its lack of objectivity, 

■ the risk that one expert will try to dominate a discussion to sway the group or that the group will 

succumb to the urge to agree, and 

■ it is not very accurate or valid as a primary estimating method.

The bottom line is that because of its subjectivity and lack of supporting documentation, expert opinion 

should be used sparingly and only as a sanity check. Case study 35 shows how relying on expert opinion as 

a main source for a cost estimate is unwise.
118 Chapter 11 GAO-09-3SP

Case Study 35: Expert Opinion, from Customs Service Modernization, 

GAO/AIMD-99-41

The U.S. Customs Service Automated Commercial Environment (ACE), a major information 

technology systems modernization effort, was estimated in November 1997 to cost $1.05 

billion to develop, operate, and maintain between 1994 and 2008. GAO’s 1999 review 

found that the agency lacked a reliable estimate of what ACE would cost to build, deploy, 

and maintain. Instead of using a cost model, Customs had used an unsophisticated 

spreadsheet to extrapolate the cost of each ACE software increment. 

Further, Customs’ approach to determining software size and reuse was not well 

supported or convincing and had not been documented. For example, Customs had 

estimated the size of each ACE software increment—most increments had still been 

undefined—by extrapolating from the estimated size of the first increment, based on 

individuals’ undocumented best judgments about functionality and complexity. 

Last, Customs did not have any historical project cost data when it developed the $1.05 

billion estimate, and it had not accounted for relevant, measured, and normalized 

differences in the increments. For instance, it had not accounted for the change in ACE’s 

architecture from a mainframe system that had been written in COBOL and C++ to a 

combined mainframe and Internet-based system that was to be written in C++ and Java. 

Such a fundamental change would clearly have a dramatic effect on system costs and 

should have been explicitly addressed in Customs’ cost estimates.

GAO, Customs Service Modernization: Serious Management and Technical 

Weaknesses Must Be Corrected, GAO/AIMD-99-41 (Washington, D.C.: Feb. 26, 

1999).

Other Estimating Methods: Extrapolation from Actual Costs

Extrapolation uses the actual past or current costs of an item to estimate its future costs. The several 

variants of extrapolation include

■ averages, the most basic variant, a method that uses simple or moving averages to determine the 

average actual costs of units that have been produced to predict the cost of future units;

■ learning curves, which account for cost improvement and are the most common variant; and

■ estimates at completion, which use actual cost and schedule data to develop estimates of costs at 

completion with EVM techniques; EACs can be calculated with various EVM forecast techniques 

to take into account factors such as current performance. 

Extrapolation is best suited for estimating follow-on units of the same item when there are actual data 

from current or past production lots. This method is valid when the product design or manufacturing 

process has changed little. If major changes have occurred, careful adjustments will have to be made or 

another method will have to be used. When using extrapolation techniques, it is essential to have accurate 

data at the appropriate level of detail, and the cost estimator must ensure that the data have been validated 

and properly normalized. When such data exist, they form the best basis for cost estimates. Advantages 

associated with extrapolating from actual costs include their

■ reliance on historical costs to predict future costs,

■ great credibility and reliability for estimating costs, and
GAO-09-3SP Chapter 11 119

■ ability to be applied at whatever level of data—labor hours, material dollars, total costs.

The disadvantages associated with extrapolating from actual costs are that

■ changes in the accounting of actual costs can affect the results,

■ obtaining access to actual costs can be difficult,

■ results will be invalid if the production process or configuration is not stable, and 

■ it should not be used for items outside the actual cost data range.

Other Estimating Methods: Learning Curves

Using the cost estimating methods discussed in this chapter can generate the cost of a single item. 

However, a cost estimator needs to determine whether that cost is for the first unit, the average unit, or 

every unit. And given the cost for one unit, how should a cost estimator determine the appropriate costs 

for other units? The answer is in the use of learning curves. Sometimes called progress or improvement 

curves, learning curve theory is based on the premise that people and organizations learn to do things 

better and more efficiently when they perform repetitive tasks. A continuous reduction in labor hours 

from repetitive performance in producing an item often results from more efficient use of resources, 

employee learning, new equipment and facilities, or improved flow of materials. This improvement can be 

modeled with a mathematical CER that assumes that as the quantity of units to be produced doubles, the 

amount of effort declines by a constant percentage. 

Workers gain efficiencies in a number of areas as items are repeatedly produced. The most commonly 

recognized area of improvement is worker learning. Improvement occurs because as a process is repeated, 

workers tend to become physically and mentally more adept at it. Supervisors, in addition to realizing 

these gains, become more efficient in using their people, as they learn their strengths and weaknesses. 

Improvements in the work environment also translate into worker and supervisory improvement: Studies 

show that changes in climate, lighting, and general working conditions motivate people to improve. 

Cost improvement also results from changes to the production process that optimize placement of tools 

and material and simplify tasks. In the same vein, organizational changes can lead to lower recurring costs, 

such as instituting a just-in-time inventory or centralizing tasks (heat and chemical treatment processes, 

tool bins, and the like). Another example of organizational change is a manufacturer’s agreeing to give a 

vendor preferred status if it is able to limit defective parts to some percentage. The reduction in defective 

parts can translate into savings in scrap rates, quality control hours, and recurring manufacturing labor, all 

of which can result in valuable time savings. In general, it appears that more complex manufacturing tasks 

tend to improve faster than simpler tasks. The more steps in a process, the more opportunity there is to 

learn how to do them better and faster.

Another reason for contractor improvement is that in competitive business environments, market forces 

require suppliers to improve efficiency to survive. As a result, some suppliers may competitively price 

their initial product release at a loss, with the expectation that future cost improvements will make up the 

difference. This strategy can also discourage competitors from entering new markets. For the strategy to 

work, however, the assumed improvements must materialize or the supplier may cease to exist because of 

high losses.
120 Chapter 11 GAO-09-3SP

In observing production data (for example, manufacturing labor hours), early analysts noted that labor 

hours per unit decreased over time. This observation led to the formulation of the learning curve equation 

Y = AXb and the concept of a constant learning curve slope (b) that captures the change in Y given a 

change in X.41 The unit formulation states that “as the number of units doubles, the cost decreases by a 

constant percent.” In other words, every time the total quantity doubles, the cost decreases by some fixed 

percentage. Figure 13 illustrates how a learning curve works.

Figure 13: A Learning Curve

0

20

40

60

80

100

120%

Cumulative number of units

90% ratio

curve

1st unit

2nd unit

4th

8th

16th

32nd

64th

128th

80% ratio

curve

0 20 40 60 80 100 120 140

Cumulative average hours per unit (as a percent of first unit)

Source: © 1994, R. Max Wideman, FCSCE, “A Pragmatic Approach to Using Resource Loading, Production and Learning Curves on Construction Projects.”

Figure 13 shows how an item’s cost gets cheaper as its quantities increase. For example, if the learning 

curve slope is 90 percent and it takes 1,000 hours to produce the first unit, then it will take 900 hours to 

produce the second unit. Every time the quantity doubles—for example, from 2 to 4, 4 to 8, 8 to 16—the 

resource requirements will reduce according to the learning curve slope. 

Determining the learning curve slope is an important effort and requires analyzing historical data. If 

several production lots of an item have been produced, the slope can be derived from the trend in the data. 

Another way to determine the slope would be to look at company history for similar efforts and calculate 

it from those efforts. Or the slope could be derived from an analogous program. The analyst could look at 

slopes for a particular industry—aircraft, electronics, shipbuilding—sometimes reported in organizational 

studies, research reports, or estimating handbooks. Slopes can be specific to functional areas such as 

manufacturing, tooling, and engineering, or they may be composite slopes calculated at the system level, 

such as aircraft, radar, tank, or missiles.

The first unit cost might be arrived at by analogy, engineering build-up, a cost estimating relationship, 

fitting the actual data, or another method. In some cases, the first unit cost is not available. Sometimes 

41 b = log (slope)/log (2).
GAO-09-3SP Chapter 11 121

work measurement standards might provide the hours for the 5th unit, or a cost estimating relationship 

might predict the 100th unit cost. This is not a problem as long as the cost estimator understands the 

point on the learning curve that the unit cost is from and what learning curve slope applies. With this 

information, the cost estimator can easily solve for the 1st unit cost using the standard learning curve 

formula Y = AXb.

Because learning can reduce the cost of an item over time, cost estimators should be aware that if multiple 

units are to be bought from one contractor as part of the program’s acquisition strategy, reduced costs can 

be anticipated. Thus, knowledge of the acquisition plan is paramount in deciding if learning curve theory 

can be applied. If so, careful consideration must be given to determining the appropriate learning curve 

slope for both labor hours and material costs. In addition, learning curves are based on recurring costs, 

so cost estimators need to separate recurring from nonrecurring costs if the results are not to be skewed. 

Finally, these circumstances should be satisfied before deciding to use learning curves:42

■ much manual labor is required to produce the item;

■ the production of items is continuous and, if not, then adjustments are made; 

■ the items to be produced require complex processes;

■ technological change is minimal between production lots;

■ the contractor’s business process is being continually improved; and

■ the government program office culture (or environment) is sufficiently known. 

Particular care should be taken for early contracts, in which the cost estimator may not yet be familiar 

enough with program office habits to address the risk accurately (for example, high staff turnover, 

propensity for scope creep, or excessive schedule delays).

Production Rate Effects on Learning

It is reasonable to expect that unit costs decrease not only as more units are produced but also as the 

production rate increases. This theory accounts for cost reductions that are achieved through economies 

of scale. Some examples are quantity discounts and reduced ordering, processing, shipping, receiving, and 

inspection costs. Conversely, if the number of quantities to be produced decreases, then unit costs can be 

expected to increase, because certain fixed costs have to be spread over fewer items. At times, an increase 

in production rate does not result in reduced costs, as when a manufacturer’s nominal capacity is exceeded. 

In such cases, unit costs increase because of such factors as overtime, capital purchases, hiring actions, and 

training costs.

Another aspect of improvement is the continuity of the production line. Production breaks may occur 

because of program delays (budgetary or technical), time lapses between initial and follow-on orders, or 

labor disputes. They may occur as a result of design changes that may require a production line to shut 

down so it can be modified with new tools and equipment or a new configuration. Production lines can 

also shut down for unexpected recalls that require repairs for previously produced items. How much 

learning is lost depends on how long the production line is shut down. 

42Appendix XI has more detail on learning and learning curves.
122 Chapter 11 GAO-09-3SP

To determine the effect of a production break on the unit cost two questions need answering:

1. How much learning has been lost (or forgotten) because of the break in production? 

2. How will this loss of learning affect the costs of future production items?

The cost estimator should always consider the effect of a production break on the cost estimate. (See case 

study 36.)

Case Study 36: Production Rate, from Defense Acquisitions, 

GAO-05-183

Costs on the CVN 76 and CVN 77 Nimitz aircraft carriers grew because of additional labor 

hours required to construct the ships. At delivery, CVN 76 had required 8 million additional 

labor hours to construct; CVN, 77, 4 million. As the number of hours increased, total 

labor costs grew because the shipbuilder was paying for additional wages and overhead 

costs. Increases in labor hours stemmed in part from underestimating the labor hours. 

The shipbuilder had negotiated CVN 76 for approximately 39 million labor hours—only 

2.7 million more labor hours than the previous ship—CVN 75. However, CVN 75 had 

been constructed more efficiently, because it was the fourth ship of two concurrent ship 

procurements. CVN 76 and CVN 77, in contrast, were procured as single ships. 

Single ship procurements have historically been less efficient than two-ship procurements. 

The last time the Navy procured a carrier as a single-ship procurement, 7.9 million more 

hours were required—almost 3 times the number estimated for CVN 76 (2.7 million more 

hours). In addition, a 4-month strike in 1999, during the construction of CVN 76, had led to 

employee shortages in key trades and learning losses, because many employees were not 

returning to the shipyard. According to Navy officials, the shipbuilder was given $51 million 

to offset the strike’s effect.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

Pulling the Point Estimate Together

After each WBS element has been estimated with one of the methods discussed in this chapter, the 

elements should be added together to arrive at the total point estimate. The cost estimator should validate 

the estimate by looking for errors like double-counting and omitted costs. The cost estimator should also 

perform, as a best practice, cross-checks on various cost drivers to see if similar results can be produced. 

This helps validate the estimate. The cost estimator should also compare the estimate to an independent 

cost estimate. The estimate and the independent cost estimate should also be reconciled at this time. 

(Chapter 15 discusses validating the estimate.)

DOD’s major defense acquisition programs are required to develop independent cost estimates for major 

program milestones; other agencies may not require this practice. An independent cost estimate gives 

an objective measure of whether the point estimate is reasonable. Differences between them should be 

examined and discussed to achieve understanding of overall program risk and to adjust risk around the 

point estimate. 
GAO-09-3SP Chapter 11 123

Finally, as the program matures through its life cycle, as more data become available, or as changes occur, 

the cost estimator should update the point estimate. The updated point estimate should be compared 

against previous estimates, and lessons learned should be documented. (More detail is in chapter 20.)

8. Best Practices Checklist: Developing a Point Estimate

 The cost estimator considered various cost estimating methods:

3 Analogy, early in the life cycle, when little was known about the system 

being developed: 

ù Adjustments were based on program information, physical and 

performance characteristics, contract type. 

3 Expert opinion, very early in the life cycle, if an estimate could be 

derived no other way.

3 The build-up method later, in acquisition, when the scope of work was 

well defined and a complete WBS could be determined. 

3 Parametrics, if a database of sufficient size, quality, and homogeneity 

was available for developing valid CERs and the data were normalized 

correctly. 

ù Parametric models were calibrated and validated using historical 

data.

3 Extrapolating from actual cost data, at the start of production.

 Cost estimating relationships were considered: 

3 Statistical techniques were used to develop CERs: 

ù Higher R-squared; 

3 Statistical significance, for determining the validity of statistical 

relationships; 

ù Significance levels of F and t statistics. 

3 Before using a CER, the cost estimator 

ù Examined the underlying data set to understand anomalies;

ù Checked equations to ensure logical relationships;

ù Normalized the data;

ù Ensured that CER inputs were within the valid dataset range;

ù Checked modeling assumptions to ensure they applied to the 

program.

3 Learning curve theory was applied if

ù Much manual labor was required for production;

ù Production was continuous or adjustments had to be made;

ù Items to be produced required complex processes;

ù Technological change was minimal between production lots;

ù The contractor’s business process was being continually improved.

 Production rate and breaks in production were considered. 

 The point estimate was developed by aggregating the WBS element cost 

estimates by one of the cost estimating methods.

3 Results were checked for accuracy, double-counting, and omissions and 

were validated with cross-checks and independent cost estimates.
124 Chapter 11 GAO-09-3SP
GAO-09-3SP Chapter 12 125

Estimating Software Costs

Chapter 12

Software is a key component in almost all major systems the federal government acquires. Estimating 

software development, however, can be difficult and complex. To illustrate, consider some statistics: a 

Standish Group International 2000 report showed that 31 percent of software programs were canceled, 

more than 50 percent overran original cost estimates by almost 90 percent, and schedule delays averaged 

almost 240 percent.43 Moreover, the Standish Group reported that the number of software development 

projects that are completed successfully on time and on budget, with all features and functions as 

originally specified, rose only from 16 percent in 1994 to 28 percent in 2000. 44

Most often, creating an estimate based on an unachievable schedule causes software cost estimates to be 

far off target. Playing into this problem is an overwhelming optimism about how quickly software can 

be developed. This optimism stems from a lack of understanding of how staffing, schedule, software 

complexity, and technology all interrelate. Furthermore, optimism about how much savings new 

technology can offer and the amount of reuse that can be leveraged from existing programs also cause 

software estimates to be underestimated. Case study 37 gives an example.

Case Study 37: Underestimating Software, from Space Acquisitions, 

GAO-07-96

The original estimate for the Space Based Infrared System for nonrecurring engineering, 

based on actual experience in legacy sensor development and assumed software reuse, 

was significantly underestimated. Nonrecurring costs should have been two to three times 

higher, according to historical data and independent cost estimators. Program officials also 

planned on savings from simply rehosting existing legacy software, but those savings were 

not realized because all the software was eventually rewritten. It took 2 years longer than 

planned to complete the first increment of software.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Our work has also shown that the ability of government program offices to estimate software costs and 

develop critical software is often immature. Therefore, we highlight software estimation as a special case 

of cost estimation because of its significance and complexity in acquiring major systems. This chapter 

43Daniel D. Galorath, Software Projects on Time and within Budget—Galorath: The Power of Parametrics, PowerPoint presentation, El Segundo, 

California, n.d., p. 3. http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php.

44 Jim Johnson and others, “Collaboration: Development and Management—Collaborating on Project Success,” Software Magazine, 

Sponsored Supplement, February–March 2001, p. 2.
126 Chapter 12 GAO-09-3SP

supplements the steps in cost estimating with what is unique in the software development environment, so 

that auditors can better understand the factors that can lead to software cost overruns and failure to deliver 

required functionality on time. Auditors should remember that all the steps of cost estimating have to be 

performed for software just as they have to be performed for hardware. 

The 12 steps of cost estimating described in chapter 1 and summarized in table 15 also apply to software. 

That is, the purpose of the estimate and the estimating plan should be defined in steps 1 and 2, software 

requirements should be defined in step 3, the effort to develop the software should be defined in step 4, 

GR&As should be established in step 5, relevant technical and cost data should be collected in step 6, 

and a method for estimating the cost for software development and maintenance should be part of the 

point estimate in step 7. Moreover, sensitivity analysis in step 8, risk and uncertainty analysis in step 9, 

documenting the estimate in step 10, presenting results to management in step 11, and updating estimates 

with actual costs in step 12 are all relevant for software cost estimates.

Table 15: The Twelve Steps of High-Quality Cost Estimating Summarized

Step Summary

1 Define the estimate’s purpose

2 Develop the estimating plan 

3 Define the program characteristics, the technical baseline

4 Determine the estimating structure, the WBS

5 Identify ground rules and assumptions

6 Obtain the data

7 Develop the point estimate and compare it to an independent cost estimate

8 Conduct sensitivity analysis

9 Conduct a risk and uncertainty analysis

10 Document the estimate

11 Present the estimate to management for approval

12 Update the estimate to reflect actual costs and changes

Source: GAO.

In this chapter, we discuss some of the best practices for developing reliable and credible software 

cost estimates and fully understanding typical cost drivers and risk elements associated with software 

development. 

Unique Components of Software Estimation

Since software is not tangible like hardware, it can be more ambiguous and difficult to comprehend. In 

addition, software is built only once, whereas hardware is often mass produced, once design and testing 

are complete. Unlike hardware, for which the industry changes more slowly, software changes constantly, 

making it difficult to collect good data for cost estimating. Despite these differences, software estimating 

is otherwise similar to hardware estimating in that it follows the same basic development process.45 For 

instance, both use the same types of estimating methods—analogy, engineering build-up, parametric. 

45A source for more information on hardware cost estimating is the International Society of Parametric Analysts, Parametric 

Estimating Handbook, 4th ed. 
GAO-09-3SP Chapter 12 127

Size and complexity are cost drivers for both. Finally, how quickly hardware and software can be produced 

depends on the developer’s capability, available resources, and familiarity with the environment. 

Software is mainly labor intensive, and all the tasks associated with developing it are nonrecurring—there 

is no production phase. That is, once the software is developed, it is simple to produce a copy of it. How 

much effort is required to develop software depends on its size and complexity. Thus, estimating software 

costs has two basic elements—the software to be developed and the development effort to accomplish it. 

Estimating Software Size

Cost estimators begin a software estimate by predicting the sizes of the deliverables that must be 

constructed. Software sizing is the process of determining how big the application being developed will be. 

The size depends on many factors. For example, software programs that are more complex, perform many 

functions, have safety-of-life requirements, and require high reliability are typically bigger than simpler 

programs.

Estimating software size is not easy and depends on having a detailed knowledge about a program’s 

functions in terms of scope, complexity, and interactions. Not only is it hard to generate a size estimate 

for an application that has not yet been developed, but the software process also often experiences 

requirements growth and scope creep that can significantly affect size and the resulting cost and schedule 

estimates. 

Programs that do not track and control these trends typically overrun their costs and experience 

schedule delays. Methods for measuring size data include COSMIC (Common Software Measurement 

International Consortium) Functional Sizing Method, function point analysis, object point analysis, 

source lines of code, and use case (described in table 16).

Table 16: Sizing Metrics and Commonly Associated Issues 

Metric Advantages Disadvantages

COSMIC functional sizing 

Measures the size of software based on 

functional user requirements; sizes software 

independently of the technology to be 

used to implement it, focusing on practices 

and procedures the software must follow to 

meet user needs. COSMIC points are based 

on four different data movements: entry, 

exit, read, and write. Each one constitutes a 

COSMIC function point.

The method can be used to determine 

the software size of various applications 

including business, real-time 

(telecommunications, process control), 

embedded software (cellular phones, 

electronics), and infrastructure software 

(operating system software)

Sizing is easily understood and 

simplified because all data 

movements have the same 

value; sizing does not depend 

on data attributes; 

It applies to real-time and 

embedded systems and

allows for end-user and 

developer viewpoints;

standards exist for counting

Recently developed, so 

benchmarking data are 

limited; not accurate for 

counting highly algorithmic 

software; detailed information 

about data movements takes 

time to collect; automated 

counting does not exist
128 Chapter 12 GAO-09-3SP

Metric Advantages Disadvantages

Function point analysis

Considers how many functions a program 

does rather than how many instructions 

it contains; functions typically include 

user inputs (add, change, delete), outputs 

(reports), data files to be updated by 

the application, interfaces with other 

applications, and inquiries (searches or 

retrievals). 

Each function is weighted for complexity 

and total count is adjusted for the 

effect of 14 characteristics such as data 

communications, transaction rate, 

installation ease, and whether there are 

multiple sites. Accurate counting requires 

in-depth knowledge of standards, 

experience, and, preferably, function point 

certification. Function point analysis is 

linked directly to system requirements and 

functionality, so size analysis is measured 

in terms users can understand. The size 

estimates (and resulting cost and schedule 

estimates) can be based on quantifiable 

analysis through the project life cycle as 

requirements change. Function points are 

particularly useful in many development 

environments that might use unified 

modeling language, commercial off-theshelf components, or object-oriented 

approaches to software development and 

implementation

Many types of data sources 

can be used throughout 

development: user or estimator 

interviews, requirements 

and design documents, data 

dictionaries and models, end 

user guides, screen captures;

not dependent on language 

or technology; count is 

unaffected by language or 

tools used to develop the 

software; counts are available 

early in development from 

requirements and design 

specifications; nontechnical 

users can understand what 

function points are measuring; 

function points can be used to 

determine requirements creep; 

counts are fully documented 

and auditable;

standards are established 

and reviewed often by the 

International Function Point 

Users Group; counting can be 

quick and efficient

Counting involves subjectivity; 

difficult to derive requirements 

from top-level specifications;

does not capture technical and 

design constraints; untrained 

or inexperienced people can 

develop inconsistent function 

point counts; definitions can 

be confusing; automated 

function point analysis 

counting does not exist; 

database is not as big as for 

source line of code counts;

counts tend to underestimate 

algorithmic intensive systems

Object point analysis

Uses integrated computer-aided software 

engineering tools (CASE) to count number 

of screens, reports, and third-generation 

modules for basic sizing; CASE tools 

take over the job of manually writing 

software code by using graphical user 

interface generators, libraries of reusable 

components, and other design tools. Object 

points focus on actors involved in the 

solution and any actions they must take.

One benefit of using objects (i.e., actors) 

is that similar behaviors can be grouped 

into classes, allowing for behaviors from 

upper classes (parent) to be inherited by 

lower classes (children). Inheritance results 

in reduced coding effort; each count is 

weighted for complexity, summed to a total 

count, and adjusted for reuse

Relies on a graphical user 

interface; automates manual 

activities; objective measures; 

easier calculations; accounts for 

reuse through inheritance

Counts occur at the end of 

design; no standards for 

counting; and not widely 

used and therefore validated 

productivity metrics are not 

available
GAO-09-3SP Chapter 12 129

Metric Advantages Disadvantages

Reports, interfaces, conversions, extensions, and forms/workflows (RICEF/W)

Commonly used to size the effort associated 

with implementing Enterprise Resource 

Planning (ERP) systems; identifies changes 

that need to be made to configure the ERP 

system so that it satisfies user needs and fits 

within the target operating environment. 

Can be used to add functionality through 

custom development. RICEF/W needs to be 

adjusted for complexity

Represents ERP modifications 

and enhancements that do not 

require custom development

Specific to ERP systems;

no standards for counting; 

does not capture costs 

for integrating bolt-on 

functionality

Source lines of code (SLOC)

Considers the volume of code required to 

develop the software; includes executable 

instructions and data declarations and 

normally excludes comments and blanks. 

Estimation is by analogy, engineering 

expertise, or automated code counters. 

SLOC sizing is particularly appropriate 

for projects preceded by similar ones 

(e.g., same language, developers, type of 

application); helps ensure that experience 

is aligned to future development. When 

developing lines of code counts, it is critical 

to define what is and is not included. 

When developing databases or relying 

on software cost models, consistency in 

defining what the lines of code include is 

key

Widely used for many years;

can be used to estimate real 

time systems easily counted, 

manually or by automated 

code counter; objective;

large databases of historical 

program sizes are available;

can obtain precise counts of 

existing software using the 

USC Code Counter

No standard definition of what 

should be counted as lines 

of code (e.g., physical line vs. 

logical statement); different 

lines of code count for the 

same function, depending on 

language and programmer’s 

style; hard to capture lines of 

code for commercial off-theshelf systems; hard to translate 

lines of code counts between 

other programming languages 

such as object oriented 

code; variations in definition 

make it hard to compare 

studies using SLOC; hard to 

estimate program SLOC early; 

emphasizes coding effort, 

which is small compared to 

overall software development 

effort

Use cases and use case points

Defines interactions between external 

users and the system to achieve a 

goal (e.g., capture fingerprint or facial 

biometric to enroll applicants). A use case 

model describes a system’s functional 

requirements, consists of all users and use 

cases (tasks performed by the end user of 

a system that has a useful outcome), and 

identifies reuse by use case inclusions and 

extensions. Sizing count is arrived at by 

categorizing use cases as small, medium, 

or large and applying an average “use case 

points per category.” Adding a complexity 

factor to the sizing count based on number 

and types of users and transactions 

improves the count accuracy

Applies to interactive enduser applications and devices 

users interact with; intuitive to 

stakeholders and development 

team; identifies opportunities 

for software reuse; traceable to 

development team’s plans and 

output; increasingly applied 

to real-time systems; can be 

mapped to test cases and 

business scenarios, which helps 

in staggered deployment

Often yields an inaccurate 

final estimate if the system 

engineering process is 

immature and historical data 

are lacking; no standards for 

counting; developer must 

be using object oriented 

design techniques so required 

documentation is available; 

estimate cannot be done until 

design document with the 

defined use case is available; 

requires a design team with a 

great deal of experience with 

object oriented design

Source: DOD, NASA, SCEA, and industry.
130 Chapter 12 GAO-09-3SP

While software sizing can be approached in many ways, none are accurate because the “size” of software is 

an abstract concept. Moreover, with the exception of COSMIC and function points, none of the methods 

table 16 describes has a controlling body for internationally standardizing the counting rules. In the 

absence of a universal counting convention, different places may take one of the source definitions for the 

basic approach and then “standardize” the rules internally. This can result in different counts. Therefore, 

it is critical that the sizing method used is consistent. The test of a good sizing method is that two separate 

individuals can apply the same rules to the same problem and yield almost the same result. Before 

choosing a sizing approach, one must consider the following questions of maturity and applicability:

■ Are the rules for the sizing technique rigorously defined in a widely accepted format?

■ Are they under the control of a recognized, independent controlling body?

■ Are they updated from time to time by the recognized, independent controlling body?

■ Does the controlling body certify the competency (and, hence, consistency) of counters who use 

their rules?

■ Are statistical data available to support claims for the consistency of counting by certified counters?

■ How long have the rules been stable?

Auditors should know a few things about software sizing. The first is that reused and autogenerated 

software source lines of code should be differentiated from the total count. Reused software (code used 

verbatim with no modifications), adapted software (code that needs to be redesigned, may need to be 

converted, and may need some code added), and autogenerated software provide the developer with code 

that can be used in a new program, but none of these comes for free, and additional effort is usually 

associated with incorporating them into a new program. For instance, the effort associated with reused 

code depends on whether significant integration, reverse engineering, and additional design, validation, 

and testing are required. But if the effort to incorporate reused software is too great, it may be cheaper 

to write the code from scratch. As a result, the size of the software should reflect the amount of effort 

expected with incorporating code from another source. This can be accomplished by calculating the 

equivalent source lines of code, which adjusts the software size count to reflect the fact that some effort is 

required. 

Software porting is a special case of software reuse that is getting increasing visibility in cost estimation 

with respect to specific technologies, such as communications systems (waveforms). Porting represents 

hidden pitfalls, depending on the amount of capability to be transferred from special purpose processors 

(such as field-programmable gate arrays). Also, the quality of software commenting and documentation 

and the modularity of the initial code’s design and implementation greatly affect the porting of standard 

code in general purpose processors. Therefore, assumptions regarding savings (for example, assume less 

effort is required and no testing is necessary) from reused, adapted, and autogenerated software code 

should be looked at skeptically because of the additional work to research the code and provide necessary 

quality checks. As a minimum, regression testing will be required before integrating the software with the 

hardware for this type of code.

Second, while function points generate counts for real-time software, like missile systems, they are not 

optimal in capturing the complexity associated with high levels of algorithmic software. Therefore, for 
GAO-09-3SP Chapter 12 131

programs that require high levels of complex processing like operating systems, telephone switching 

systems, navigation systems, and process control systems, estimators should base the count on COSMIC 

points or SLOC rather than function points to adequately capture the additional effort associated with 

developing algorithmic software.

Finally, choosing a sizing metric depends on the software application (purpose of the software and level of 

reliability needed) and the information that is available. Since no one way is best, cost estimators should 

work with software engineers to determine which metric is most appropriate. Since SLOCs have been 

used widely for years as a software sizing metric, many organizations have databases of historical SLOC 

counts for various completed programs. Thus, source lines of code tend to be the most predominant 

method for sizing software. If the decision is made to use historical source lines of code for estimating 

software size, however, the cost estimator needs to make sure that the program being estimated is similar 

in size, language, and application to the historical data. For programs for which no analogous data are 

available but detailed requirements and specifications have been developed, function point counting is 

appropriate, as long as the software does not contain many algorithms; if it does, then COSMIC points 

or SLOC should be used. And, if computer-assisted software engineering tools are being used to develop 

the software, then object point analysis is appropriate. No matter which metric is chosen, however, the 

actual results can vary widely from the estimate, so that any point estimate should be accompanied by 

an estimated range of probability. (We discuss software and other cost estimating risk and uncertainty 

analyses in chapter 14.) 

When completing a software size estimate, it is preferable to use two different methodologies, if available, 

rather than relying on a single approach. Software estimates based on several different approaches that 

are compared and merge toward a consensus is the best practice. In addition, it is extremely important 

to include the expected growth in software size from requirements growth or underestimation (that 

is, optimism). Adjusting the software size to reflect expected growth from requirements being refined, 

changed, or added or initial size estimates being too optimistic and less reuse than expected is a best 

practice. This growth adjustment should be made before performing an uncertainty analysis (discussed in 

chapter 14). Understanding that software will usually grow, and accounting for it by using historical data, 

will result in more accurate software sizing estimates. Moreover, no matter what sizing convention is used, 

it is a best practice to continually update the size estimate as data become available so that growth can be 

monitored and accounted for. 

Estimating Software Development Effort

Once the initial software sizing is complete, it can be converted into software development effort—that is, 

an estimate of the human resources needed for the software’s development. It is important to note whether 

the effort accounts only for the WBS elements associated with the actual development of the software or 

also includes all the other nondevelopment activities. 

Table 53 in appendix IX, for example, shows a typical WBS for ground software development. The 

table shows that many other activities outside the actual coding of software are part of a typical software 

acquisition. These activities should also be estimated as part of the development effort. In particular, 

software management and control, software systems engineering, test-bed development, system integration 
132 Chapter 12 GAO-09-3SP

and testing, quality assurance, and training are all activities that should be performed in a customized 

software solution acquisition. 

The level of effort required for each activity depends on the type of system being developed. For example, 

military and systems software programs require more effort than Web programs of the same size. Since 

variations in activities can affect overall costs, schedules, and productivity rates by significant amounts, it 

is critical to appropriately match activities to the type of software project being estimated. For example, 

safety critical software applications composed of complex mathematical algorithms require higher levels 

of effort because stringent quality and certification testing must be satisfied. Moreover, operating systems 

that must reflect real time updates and great reliability will need more careful design, development, and 

testing than software systems that rely on simple calculations.

To convert software size into software development effort, the size is usually divided by a productivity 

factor like number of source lines of code, or function points, developed per labor work month. The 

productivity factor depends on several aspects, like the language used; whether the code is new, reused, 

or autogenerated; the developer’s capability; and the development tools used. It is best to use historical 

data from a similar program to develop the productivity factor, so that it best represents the development 

environment. If historical productivity factors are not available, an estimator can use a factor based on 

industry averages, but this will add more uncertainty to the estimate. It is important to note, however, that 

a productivity factor—based on the coding phase only—cannot be used to estimate the entire software 

development effort. When a productivity factor is used, all parameters associated with its computation 

need to be considered. Once the productivity factor has been selected, the corresponding labor hours can 

be generated. 

Some considerations in converting labor hours to cost are, first, that a cost estimator needs to determine 

how many productive hours are being assumed in a typical developer’s work day. This is important 

because assuming 8 hours of productive coding is unrealistic: staff meetings and training classes cut into 

valuable programming time, so that the number of effective work hours per day is typically 6 hours rather 

than 8. Further, the number of work days per year is not the same from company to company because of 

differences in vacation and sick leave offered and the country the developers live in. In the United States, 

fewer vacation days tend to be provided than in countries in Europe, but in other countries like Japan less 

time is provided. All these issues need to be considered and calibrated to the program being estimated. In 

fact, multiple studies on the impact of overtime have shown that except for a short increase in effort over 

the first 1 or 2 months, overtime does not have a significant impact on the life of the program.

The sizing value usually represents only the actual software development effort, so the cost estimator needs 

to use other methods to estimate all the other activities related to developing the software. Sometimes 

factors (such as percentage of development effort) are available for estimating these additional costs. 

Software cost estimating models often provide estimates for these activities. If a model is not used or 

not available, then the cost estimator must account for the cost of the other labor as well as nonlabor 

costs, such as hardware and licenses. Accurately estimating all these tasks is challenging, because 

they are affected by a number of risks. (Some are identified in table 17; appendix XV contains a more 

comprehensive list of risks.) 
GAO-09-3SP Chapter 12 133

Table 17: Common Software Risks That Affect Cost and Schedule

Risk Typical cost and schedule element

Sizing and 

technology

■ Overly optimistic software engineers tending to underestimate the amount of code 

needed

■ Poor assumptions on the use of reused code (requiring no modification) or adapted 

code (requiring some redesign, recoding, and retesting)

■ Vague or incomplete requirements, leading to uncertain size counts

■ Not planning for additional effort associated with commercial off-the-shelf software 

(e.g., systems engineering, performance testing, developing glue code)

Complexity ■ Programming language: the amount of design, coding, and testing (e.g., objectoriented languages require more up-front design but result in less coding and 

testing)

■ Applications: software purpose and reliability (e.g., criticality of failure, loss of life)

■ Hardware limitations with respect to the need for more efficient code

■ Number of modules affecting integration effort

■ Amount of new code to be developed 

■ Higher quality requiring more development and testing but resulting in less and 

easier-to-perform maintenance

■ Safety critical software requires more design, coding, and testing

Capability ■ Developers with better skill can deliver more effective software with fewer defects, 

allowing for faster software delivery

■ Optimistic assumption that a new development tool will increase productivity

■ Optimistic assumption about developer’s productivity, leading to cost growth, even 

if sizing is accurate

■ Geographically dispersed development locations, making communication and 

coordination more difficult

Management and 

executive oversight

■ Management’s dictating an unrealistic schedule

■ A decision to concurrently develop hardware and software, increasing risk

■ Incorporating a new method, language, tool, or process for the first time

■ Incomplete or inaccurate definition of system requirements

■ Not handling creeping requirements proactively

■ Inadequate quality control, causing delays in fixing unexpected defects

■ Unanticipated risks associated with commercial off-the-shelf software upgrades and 

lack of support

Source: SCEA and industry.

Scheduling Software Development

The schedule for getting the work accomplished should also be estimated. Too often, software 

development programs tend to run late because of requirements creep or poor quality control. Other 

times, the schedule is driven by some arbitrary date dictated by management or the customer. Optimism 

may be based on management’s thinking that if more people are added to the development team, the 

product can be developed faster. Unfortunately, the opposite usually happens: the larger the development 

team, the less its members are able to communicate with one another or work effectively. In addition, the 

more complex the software development effort is, the harder it will be to find the right staff for the job.

Scheduling is complicated and is affected by many factors. A cost estimator should understand the 

intricate interdependencies that affect the schedule:

■ staff availability;

■ an activity’s dependence on prior tasks;
134 Chapter 12 GAO-09-3SP

■ the concurrence of scheduled activities;

■ the activities that make up the critical path;

■ the number of shifts working and effective work hours per shift;

■ available budget;

■ whether overtime can be authorized;

■ downtime from meetings, travel, sickness; 

■ geographic location of workers, including time zones.

Significantly large software development efforts frequently experience cost and schedule growth. 

This is because of the complexities inherent in managing configuration, communications, and design 

assumptions that typically hinder software development productivity. In addition, increased software 

schedule has a ripple effect on other collateral support efforts such as program management and systems 

engineering. Hardware programs experience the same problems.

Management pressure on software developers to keep to an unrealistic schedule presents other problems. 

For example, to meet schedule constraints, the developer may minimize the time for requirements analysis, 

which can affect the quality of the software developed. In addition, developers may skip documentation, 

which could result in higher software maintenance costs. Moreover, developers may decide to build more 

components in parallel, defer functionality, postpone rework, or minimize functional testing, all to reduce 

schedule time. While these actions may save some time up front, they result in additional time, effort, and 

risk for the program. 

Rework should be included in every software development schedule because it is unwise to assume that 

software can be delivered without any defects. Therefore, if rework is not accounted for in the schedule, it 

will have to be accounted for when it occurs, which will cause problems in the sequencing of remaining 

tasks. It should be noted that if a software schedule does not include effort for rework, then the schedule 

will be unexecutable, and the maturity of the developing organization is questionable for assuming that 

all requirements will pass testing the very first time. Rework effort should include the time and resources 

associated with diagnosing the problem, designing and coding the fix, and then retesting until the 

problem is resolved. To adequately account for rework, the schedule should anticipate a certain number of 

defects based on historical experience, and time and effort should be allocated for fixing them.

We discuss scheduling more thoroughly in chapter 18, including how to account for these risks so that 

schedule is realistic. 

Software Maintenance

Once the software has been developed, tested, and installed in its intended location, it must be 

maintained, just like hardware. Often called the operational phase for software, its costs must be 

accounted for in the LCCE. During this phase, software is maintained by fixing any defects not 

discovered in testing (known as corrective maintenance), modifying the software to work with any 

changes to its physical environment (adaptive maintenance), and adding new functionality (perfective 

maintenance). When adding capability, the effort is similar to a minidevelopment effort and the cost 

drivers are the same as in development. Software maintenance may also be driven by technology upgrades 
GAO-09-3SP Chapter 12 135

(adaptive maintenance) and users requesting enhancements (perfective maintenance). In addition to 

providing help desk support to users of the software, perfective maintenance often makes up the bulk of 

the software maintenance effort. 

The level of maintenance required depends on several factors. How complex the software is will determine 

how much maintenance is needed. In addition, if requirements from development were deferred until the 

software was in maintenance mode, or the requirements were too vague and not well understood, then 

additional perfective maintenance will be necessary. The quality of the developed software will also affect 

maintenance. If the software was rigorously tested, then less corrective maintenance will be needed. In 

addition, software that is well documented will be easier to de-bug and will provide maintainers a better 

understanding of how the software was designed, making modifications more streamlined. 

In addition to the need to maintain the software code, costs are associated with help desk support that 

need to be included in the software’s operation and support phase. Effort will be spent on trouble calls 

and generating defect tickets for software maintenance and should be included as part of the software cost 

estimate. 

Parametric Software Estimation

Software development cost estimating tools—or parametric tools—can be used to estimate the cost to 

develop and maintain software. Parametric tools are based on historical data collected from hundreds 

of actual projects that can generate cost, schedule, effort, and risk estimates based on inputs provided 

by the tool user. Among other things, these inputs generally include the size of the software, personnel 

capabilities, experience, development environment, amount of code reuse, programming language, and 

labor rates. Once the data have been input, the tool relies on cost estimating relationships and analogies to 

past projects to calculate the software cost and schedule estimates. When these data are not available to the 

cost estimator, most tools have default values that can be used instead.

Parametric tools should be used throughout the development life cycle of the software. They are especially 

beneficial in the early stages of the software life cycle, when requirement specifications and design are still 

vague. For example, these tools provide flexibility by accepting multiple sizing metrics, so that estimators 

can apply different sizing methods and examine the results. Additionally, parametric-based estimates 

can be used to understand tradeoffs by analyzing the relative effects of different development scenarios, 

determine risk areas that can be managed, and provide the information necessary for monitoring and 

control of the program.

The tools allow estimators to manipulate various inputs to gauge the overall sensitivity to parameter 

assumptions and then assess the overall risk, based on the certainty of those inputs. Developers who use 

tools in development can discover potential problems early enough to mitigate their impact.

As the project matures and actual data become available, the precision of the cost estimates produced by 

a parametric tool are likely to improve. For this to happen, the tool must be calibrated with actual data 

from completed programs so it can be adjusted to reflect the actual development environment. Since most 

models are built on industry averages, simply using default values in the tool may lead to skewed results. 

Calibration avoids this by using known inputs and outcomes to adjust the relationships in the model. 

Therefore, calibration is necessary for ensuring more accurate estimates. 
136 Chapter 12 GAO-09-3SP

When a parametric tool is used, it is essential to ensure that the estimators are trained and experienced in 

applying it and interpreting the results. Simply using a tool does not enhance the estimate’s validity. Using 

a tool correctly by calibrating it to the specific program is necessary for developing a reliable estimate. In 

addition, the following issues should be well understood before unquestioningly accepting the results of a 

parametric tool: 

■ Ensure that autogenerated code is properly captured by the model, in terms of increased 

productivity and the effort required to design, develop, document, and produce the code.

■ Output from the tool may include different cost and effort estimates or activities and phases that 

would have to be mapped or deleted to conform to the specific program. Not understanding what 

is in the output could lead to overestimating or underestimating the program.

■ Some models limit the size of the development program for which they can forecast the effort. Sizes 

outside of the tool range may not fit the program being estimated.

■ Data are often proprietary so the models are only as accurate as their underlying data allow them to 

be. Therefore, results from the model should be cross-checked. 

■ Each model has different sensitivities to certain parameters and “opinions” on desirable staff levels. 

Therefore, various models offer different schedule duration results. For particularly small or large 

software programs, a schedule predicted by a commercial parametric model needs to be crosschecked. 

■ Where a detailed build structure or spiral development is to be modeled, the commercial 

model implementation and results should be closely monitored. The same is true for significant 

integration of commercial off-the-shelf software (COTS) or government off-the-shelf software 

(GOTS) with development software (or hardware).

In addition to these issues, it is important to note that many models do not address the costs associated 

with database development. If databases will be required as part of the software solution, and the model 

used to estimate the software does not account for the cost of database development, then this cost must 

be estimated separately. The cost for database development will depend on the size and complexity of the 

source data. Cost drivers for database development include the number of feeder systems, data elements, 

and users as well as the software to be used to develop the new database.

Commercial Off-the-Shelf Software

Using commercial off-the-shelf software has advantages and disadvantages, and auditors need to 

understand the risks that come with relying on it. One advantage is that development time can be faster. 

The software can provide more user functionality than custom software and may be flexible enough to 

accommodate multiple hardware and operating environments. Also, help desk support can be purchased 

with the commercial license, which can help reduce software maintenance costs. 

Among the drawbacks to off-the-shelf software is the learning curve associated with its use, as well as 

integrating it into the new program’s environment. In addition, most commercial software is developed 

for a broad spectrum of users, so it tends to address only general functions. More specific functions 

must be customized and added, and glue-code may be required to enable the software to interact with 

other applications. And, because the source code is usually not provided to customers of commercial off-
GAO-09-3SP Chapter 12 137

the-shelf software, it can be hard to support the software in-house. When upgrades occur, the software 

may have to be reintegrated with existing custom code. Thus, it can be wrong to think that commercial 

software will necessarily be an inexpensive solution. 

Estimators tend to underestimate the effort that comes before and after implementing off-the-shelf 

software. For example, requirements definition, design, and testing of the overall system must still be 

conducted. Poorly defined requirements can result in less than optimal software selection, necessitating 

the development of new code to satisfy all requirements. This unexpected effort will raise costs and cause 

program delays. In addition, adequate training and access to detailed documentation are important for 

effectively using the software. 

Furthermore, since commercial software is subject to intense market forces, upgrades can be released with 

minimal testing, causing unpredictable problems, such as defects and systems incompatibilities. When 

this happens, additional time is needed to analyze the cause of failures and fix them. Finally, interfaces 

between the software and other applications may need to be rewritten every time the software is upgraded. 

While software developers can address all these issues, they take some time to accomplish. Therefore, 

adequate planning should be identified and estimated by the cost estimator to ensure that enough time 

and resources are available to perform them. 

Enterprise Resource Planning Software

Enterprise resource planning (ERP) refers to the implementation of an administrative software system 

based on commercial off-the-shelf software throughout an organization. ERP’s objective is to integrate 

information and business processes—including human resources, finance, manufacturing, and sales—to 

allow information entered once into the system to be shared throughout an organization. ERP systems 

force business process reengineering, allowing for improved operations that can lead to savings down the 

road. To achieve savings requires an extensive knowledge of business processes so that users will optimize 

automation, programming skills, and change management in the new work processes. Although an ERP 

system is configured commercial software and should be treated as such, we highlight this type of effort 

because of the unique difficulty of estimating its implementation costs and duration. 

Organizations implementing ERP systems risk cost overruns and missed deadlines. According to a 

Gartner report, “For 40 percent of enterprises deploying ERP systems through 2009, the actual time 

and money spent on these implementations will exceed original estimates by at least 50 percent (0.7 

probability).”46

At the heart of an ERP system are thousands of packages—built from database tables—that need to 

be configured to match end business processes. Each table has a decision switch that opens a specific 

decision path. By confining themselves to only one way to do a task, stove-piped units become integrated 

under one system. Deciding which switches in the tables to choose requires a deep understanding of the 

existing business operating processes. Thus, as table switches are picked, these business processes become 

reengineered to conform to the ERP’s way of doing business. As a result, change management and buy-in 

from the end users are crucial to the ERP system’s ultimate success.

46Pat Phelan, Estimating the Time and Cost of ERP Implementation Projects Is a 10-Step Process (Stamford, Conn.: Gartner Inc., Apr. 

10, 2006), p. 3.
138 Chapter 12 GAO-09-3SP

Cost estimators and auditors need to be aware of the additional risks associated with ERP implementation. 

Table 18 describes some of these risks and best practices for avoiding them. 

Table 18: Best Practices Associated with Risks in Implementing ERP

Risk Best practice

Training Staff are trained in the new ERP system’s software and the new processes; 

agencies teach workers how the ERP system will affect their business processes, 

developing their own training programs if necessary; providing mentoring and 

support for the first year of implementation eases the transition to the new 

system; obtaining user buy-in can be accomplished by communicating and 

marketing the benefits and new capabilities the ERP system will offer

Integrating and testing 

software links

Agencies build and test links from their established software to the new ERP 

system or buy add-ons that are already integrated with the new system; they 

estimate and budget costs carefully, planning either way to test ERP integration 

from a process-oriented perspective

Interfacing with legacy 

systems

Since interfacing the ERP’s system software with legacy systems can be very 

expensive, carefully determining early on how both systems will pass data is 

paramount; preparing a business case to evaluate whether to maintain the 

legacy system is worth the added costs

Customizing Customizing core ERP software can be costly, especially since the ERP system’s 

elements are linked; perhaps use commercial add-ons if the software cannot 

handle at least one business process 

Converting and analyzing 

data 

Cost estimators look at the agency’s data conversion and analysis needs to see 

whether, for example, the cost of converting data to a new client server setup 

is accounted for, data from the ERP system and external systems have to be 

combined for analysis, the ERP budget should include data warehouse costs, or 

programming has to be customized

Following up installation Agencies plan for follow-up activities after installation, building them into 

their budget, keeping the team who implemented the ERP system onboard 

to keep the agency informed of its progress, and providing management with 

knowledge of the ERP project’s benefits

Source: GAO, DOD, and Derek Slater, “The Hidden Costs of Enterprise Software,” CIO Enterprise Magazine, Jan. 15, 1998.

Other costs associated with ERP system implementations include costs for adding “bolt-ons,” which are 

separate supplemental software packages that deliver capability not offered by the ERP system. Bolt-ons 

connect to the ERP system using standard application programming interfaces or extensible markup 

language schema, which allow for data to pass between both systems. Costs for interfacing the bolt-on 

with the ERP system need to be identified and estimated. In addition, the number of bolt-ons that need to 

be integrated, as well as the type and size of the bolt-on functionality, will drive the cost of the interface. 

Experts agree that the ERP postimplementation stabilization period tends to be underestimated, because 

people tend to be too optimistic about how long training and the transition period will last. As a result, 

there is a risk for cost growth if management does not do a good job of selling the benefits of ERP. To 

successfully implement an ERP system, management has to be committed to freeing up resources to get 

the job done. This means that seasoned staff will need to be pulled away from their day jobs to focus on 

the effort to be fully effective. In addition, training tends to be underestimated in terms of both length 

and timing. To better plan for this effort, management needs to create a sense of urgency for change and 

provide early communication and adequate training in order to ensure successful implementation.
GAO-09-3SP Chapter 12 139

Software Costs Must also Account for Information Technology

Infrastructure and Services

Studies have shown that information technology (IT) services outside software development and 

maintenance (for example, hardware cost, help desk, upgrade installation, training) can make up a 

majority of total ownership costs. In fact, OMB reports that 77 percent of the overall IT budget for fiscal 

year 2009 will support steady state IT operations while only 23 percent will be used for development, 

modernization, and enhancement.

Even systems such as ships, aircraft, and mission control centers have major IT infrastructure and 

services components to them. In fact, some IT systems encounter over 90 percent of their costs in the 

infrastructure and services required to support and run them. Yet when we read of costs, successes, 

failures, and challenges in IT systems, the vast majority of the systems typically refer to the software 

portions only, ignoring the IT services and infrastructure components. Making matters more difficult for 

those estimating IT systems are the numerous definitions of IT infrastructure. One useful definition is 

that it consists of the equipment, systems, software, and services used in common across an organization, 

regardless of mission, program, or project. IT infrastructure also serves as the foundation on which 

mission, program, or project-specific systems and capabilities are built.

While we have already discussed software development and maintenance, we discuss in this section 

estimating the information technology services, hardware systems, and facilities required to support 

software and systems.

Unique Components of IT Estimation

Unlike software, IT estimation is in some ways simpler than software development estimation, since IT 

infrastructure and services are more tangible. However, IT estimation is fraught with issues such as

■ What is the cost of the system engineering to define the IT system?

■ How much computing power is needed to support a system?

■ How many help desk personnel are needed to support X users?

■ How can costs be contained while still achieving innovation?

■ How can the value of the IT investment be quantified against its costs?

■ How do buy and lease decisions affect expenses and profitability?

■ How can we make tradeoffs between technology and costs?

■ What kind of application initiatives are needed to support the business?

■ How many vendors and how much vendor interface is required to run the IT operation?

■ How many sites does the IT infrastructure support?

■ How many and how clearly defined or stable are the requirements for the IT to align itself with the 

business goals?
140 Chapter 12 GAO-09-3SP

Simply getting a quote from a vendor for an IT system is rarely sufficient for IT cost estimation. While 

quotes often do not include many important cost elements, the cost estimator will still need to consider 

these elements. They include

■ help desk support services supplied internally for applications and equipment; 

■ facilities costs; 

■ costs of on-going installation, maintenance, repair, and trouble shooting;

■ employee training, both formal training and self-training.

To further complicate the effort, many vendors offer IT infrastructure either as a “software as a service” 

platform or as just “cloud computing.”47 Vendor-operated IT infrastructure hardware can be viable if 

issues such as loss of control, security, and potential resource sharing are acceptable. However, such 

vendor-operated infrastructure does not usually eliminate the costs of ongoing IT services to provide users 

help desk support, local computing, setup training, and other infrastructure services. The cost estimator 

must be aware that these costs should be considered, whether the infrastructure is to be owned by the 

government, leased, or owned and operated by vendors under contract with the government.

Major Cost Drivers Associated with IT Estimation

Many factors that affect IT costs need to be considered when developing an IT cost estimate. Various 

examples of cost drivers, organized by physical attributes of the IT infrastructure, are listed next, along 

with performance and complexity requirements and economic considerations.

1. Physical attributes that drive IT costs:

■ Application software, system software, and database storage size;

■ End user hardware list (e.g., laptops, CPU, printers);

■ Facility requirements (power, cooling);

■ Infrastructure hardware list (UNIX Servers, Windows servers, WAN/LAN equipment);

■ Number of application software, system software, and database items; 

■ Number of application software, system software, and database users (concurrent, causal);

■ Number of inbound and outbound application software and database interfaces;

■ Number of unique platforms supported;

■ Operating locations;

■ Physical and organizational entities. 

2. Performance and complexity attributes:

■ Business requirements;

■ Complexity of infrastructure environment (e.g., disparate platforms, loose vs. tight coupling);

■ User type (professional, concurrent, casual);

47Cloud computing refers to information that resides in servers on the Internet and is downloaded temporarily onto various 

hardware devices such as desktop and notebook computers, entertainment centers, and handheld telephones.
GAO-09-3SP Chapter 12 141

■ Criticality and reliability of systems;

■ Expected service level (system administration, database administration, help desk Tier I, Help 

Desk Tier II, Help Desk Tier III);

■ Experience with systems;

■ Infrastructure hardware complexity (small, medium, large);

■ IT project type (ERP, SOA, Web application, data mart);

■ Number of transactions per second;

■ Number of vendors;

■ Process experience and rigor;

■ Security requirements;

■ System complexity (hardware or software);

■ Usage patterns (transaction rates).

3. Economic factors and considerations:

■ Acquisition strategy; 

■ Hardware leasing and purchasing agreements;

■ Labor rates;

■ Sourcing strategy;

■ Replacement and upgrade policies;

■ Software leasing and purchasing agreements (enterprise, user based);

■ Test plan;

■ Training strategy;

■ Years of operating.

Common Risks for IT Infrastructure 

Many of the risks that affect software cost estimating apply to IT infrastructure. For example, in 

estimating the costs of any effort, a consideration should be made whether the risks of the investment 

justify the inclusion of an independent verification and validation contractor. In situations where the risks 

are very high, such as potential loss of life, the overall schedule may need to be extended to accommodate 

the additional reviews and testing required. For IT infrastructure, the set of risks in table 19 should be 

considered.
142 Chapter 12 GAO-09-3SP

Table 19: Common IT Infrastructure Risks

Risk Technical, management, and logistic requirements that increase costs

Financial Cost overruns

Funding cuts and delays

Logistics and equipment Contingency equipment availability

Physical storage of equipment on arrival and security 

Supply availability

Schedule Unscheduled changes and delays 

Nonconformance, not starting, and failures 

Reliance on external subcontractors and organizations 

Personnel Changes of personnel among customer or vendor 

Lack of skills or knowledge 

Not aware of policy or procedures or inadequate personnel to support help 

desk and deployment 

Time lost for end user training, trouble shooting, and down time

Project management No quality control or management process built into plan 

Absence of issue, change request, or configuration management logs 

Inconsistent project documentation or lack of IT process model 

Information security

Lack of detailed site information 

Lack of issue identification or trends

Lack of reporting

Poor planning

Requirements not well defined

Role confusion

Unaware of customer site requirements

Technical Adequate capacity

Additional hardware or software requirements to fully support system

Compatibility or whether data in the relevant process flow from end to end

Disasters

Hardware or software failure

Incorrect images or version loaded 

Integration with existing systems 

New design not working

Unplanned or unapproved changes

Version control problems

User Confusion about customer and vendor responsibilities

Inability to perform core or noncore business activities 

Loss of data

Not aware of vendor schedule or activities

User expectations

Source: GAO.
GAO-09-3SP Chapter 12 143

Estimating Labor and Material Costs Associated with IT Infrastructure

Labor and material nonrecurring and recurring efforts are associated with IT infrastructure. For 

estimating the nonrecurring effort, staff loading of the IT infrastructure is similar to software 

development during early architecture and design. Once the design is complete, the recurring effort 

associated with actual implementation and deployment can be accomplished, based on a distribution of 

organizational demand for IT.

IT recurring operations costs include costs similar to the maintenance of general fixed facilities. For 

example, facilities costs such as power, security, and general facilities support apply to IT infrastructure 

recurring operations. Furthermore, costs for purchased software licenses, training, technical refreshment, 

and various service level agreements also need to be considered. Finally, since the cost of hardware changes 

daily as does the requirement for computing power in items like servers, designing with a 50 percent 

reserve in capacity is prudent since systems tend to grow. Many labor services categories need to be 

considered when developing an IT infrastructure labor cost estimate. Table 20 describes typical labor 

categories.48

Table 20: Common Labor Categories Described

Category Description Common titles

Project 

stakeholder

A person invested in the project’s success while not 

participating in its execution or implementation; includes 

end users, managers, and external clients whose success 

is somehow tied to the project’s success. Stakeholders 

work with the product management team to ensure that 

the solution developed meets the project’s original needs. 

Stakeholder participation and availability are vital to the 

success of any project

Management Performs project planning, staffing, and tracking; is involved 

with daily operational activities, ensuring that resources are 

used effectively and services are delivered

Configuration manager, 

database manager, IT 

manager, project manager

Analyst Generally involved in planning and defining needs and 

requirements for IT projects and related support systems 

and in ongoing systems support, often bridging the user or 

customer and the technical team. Generally has domain or 

specialty knowledge of a certain type of system, technology, 

or discipline used to apply technology to address business 

and user requirements

Business process, 

requirements, or system 

analyst; network or 

telecommunications analyst; 

support analyst; operations 

analyst; database analyst; UI 

analyst; security analyst

Architect Develops high-level system design plans to meet the 

organization’s needs and comply with its policies; can help 

formulate policies and plans that support the organization, 

particularly as they pertain to technologies used to carry out 

policies and procedures

Systems architect or 

engineer; IT or data 

architect; network architect; 

storage architect

48Appendix IX contains a sample IT infrastructure and IT services WBS; it is a supplement to the automated information system 

configuration, customization, development, and maintenance WBS discussed in chapter 8. 
144 Chapter 12 GAO-09-3SP

Category Description Common titles

Technician Involved primarily in the physical setup, support, and 

maintenance of systems according to well defined plans and 

procedures, including system setup, installation, upgrades, 

and troubleshooting 

Desktop or PC 

technician; network 

engineer or technician; 

hardware technician; 

telecommunications 

technician

Test/QA Primarily verifies the integrity and performance of systems 

being deployed and operated; develops test plans and 

procedures, collecting and tracking defect data and 

problem reports and serves an auditing function to ensure 

compliance with policies and procedures

IT auditor, QA analyst, 

application tester, call center 

agent

Documentation Prepares or maintains documentation pertaining to 

programming, systems operation, and user documentation, 

including user manuals and online help screens

Technical or report writer; 

online help publisher; 

content developer; 

documentation specialist

Training Prepares and updates courseware and training materials and 

conducts training classes or events

Instructor, training 

developer, instructional 

designer, end user

Administrator Generally involved with the ongoing administration, 

maintenance, and support of specific systems to ensure they 

operate properly and effectively; associated with a specific 

system or type of system such as a platform, database, 

network, or enterprise application

Network, system, or 

enterprise application 

administrator; system 

administrator; Web or 

telecommunications 

administrator; database 

administrator; security 

administrator; storage 

administrator; help desk 

specialist (tier I, tier II, tier III)

Computer 

operator

Computer operators not included in support of IT 

infrastructure and IT services

Indirect support Secretarial, reception, and other labor in support of IT 

services and infrastructure personnel and systems

Contract labor Vendors that provide services under contract to support IT 

infrastructure

Source: GAO.
GAO-09-3SP Chapter 12 145

9. Best Practices Checklist: Estimating Software Costs

 The software cost estimate followed the 12-step estimating process:

3 Software was sized with detailed knowledge of program scope, 

complexity, and interactions, and the cost estimators worked with 

software engineers to determine the appropriate sizing metric.

3 It was sized with source lines of code, function, object, feature point, or 

other counts.

 The software sizing method was appropriate:

3 Source lines of code were used if requirements were well defined and if 

there was a historical database of code counts for similar programs and 

a standard definition for a line of code.

3 Function points were used if detailed requirements and specifications 

were available, software did not contain many algorithmic functions, 

and an experienced and certified function point counter was available.

3 COSMIC points were used if functional user requirements are 

known and the application is for business, real-time, embedded, or 

infrastructure software.

3 Object points were used if computer-aided software engineering tools 

were used to develop the software.

3 Reports, interfaces, conversions, extensions and forms / workflow were 

used for ERP programs.

3 Use cases and use case points were used if system and user interactions 

were defined.

3 Autogenerated and reused source lines of code were identified 

separately from new and modified code to account for pre- and 

postimplementation efforts.

3 Several methods were used to size the software to increase the accuracy 

of the sizing estimate.

3 The final software size was adjusted for growth based on historical data, 

and growth is continually monitored over time.

 Software cost estimates included

3 Development labor costs for coding and testing, other labor supporting 

software development, and nonlabor costs like purchasing hardware 

and licenses.

3 Productivity factors for converting software size into labor effort, 

based on historical data and calibrated to match program size and 

development environment.

3 Industry average productivity factors and risk ranges, if no historical 

data were available.

3 Assumptions about productive labor hours in a day and work days in a 

year.

3 Development schedules accounting for staff availability, prior task 

dependencies, concurrent and critical path activities, number and 

length of shifts, overtime allowance, down time, and worker locations. 
146 Chapter 12 GAO-09-3SP

3 Costs for help desk support, database development, and corrective, 

adaptive, and preventive maintenance as part of the software’s life cycle 

cost. 

3 Time and effort associated with rework to fix defects.

3 Training cost estimators to calibrate parametric tools to match the 

program and model results cross-checking for accuracy.

3 Estimators’ accounting for integrating commercial off-the-shelf software 

into the system, including developing custom software and glue-code.

3 Impact of risks facing ERP system implementations as outlined in table 18.

3 Costs associated with interfacing bolt-on applications for ERP systems.

 IT infrastructure and services components of the software cost estimate 

included

3 Costs associated with the physical attributes of the IT infrastructure, 

the performance and complexity requirements, and economic 

considerations.

3 Impact of risks affecting IT infrastructure, as outlined in table 19.

3 Costs associated with labor and material nonrecurring and recurring 

efforts.
GAO-09-3SP Chapter 13 147

Sensitivity Analysis

Chapter 13

As a best practice, sensitivity analysis should be included in all cost estimates because it examines the 

effects of changing assumptions and ground rules. Since uncertainty cannot be avoided, it is necessary to 

identify the cost elements that represent the most risk and, if possible, cost estimators should quantify the 

risk. This can be done through both a sensitivity analysis and an uncertainty analysis (discussed in the 

next chapter). 

Sensitivity analysis helps decision makers choose the alternative. For example, it could allow a program 

manager to determine how sensitive a program is to changes in gasoline prices and at what gasoline price 

a program alternative is no longer attractive. By using information from a sensitivity analysis, a program 

manager can take certain risk mitigation steps, such as assigning someone to monitor gasoline price 

changes, deploying more vehicles with smaller payloads, or decreasing the number of patrols. 

For a sensitivity analysis to be useful in making informed decisions, however, carefully assessing the 

underlying risks and supporting data is necessary. In addition, the sources of the variation should be 

well documented and traceable. Simply varying the cost drivers by applying a subjective plus or minus 

percentage is not useful and does not constitute a valid sensitivity analysis. This is the case when the 

subjective percentage does not have a valid basis or is not based on historical data.

In order for sensitivity analysis to reveal how the cost estimate is affected by a change in a single 

assumption, the cost estimator must examine the effect of changing one assumption or cost driver at a 

time while holding all other variables constant. By doing so, it is easier to understand which variable most 

affects the cost estimate. In some cases, a sensitivity analysis can be conducted to examine the effect of 

multiple assumptions changing in relation to a specific scenario. 

Regardless of whether the analysis is performed on only one cost driver or several within a single scenario, 

the difference between sensitivity analysis and risk or uncertainty analysis is that sensitivity analysis tries to 

isolate the effects of changing one variable at a time, while risk or uncertainty analysis examines the effects 

of many variables changing all at once.

Typically performed on high-cost elements, sensitivity analysis examines how the cost estimate is affected 

by a change in a cost driver’s value. For example, it might evaluate how the number of maintenance staff 

varies with different assumptions about system reliability values or how system manufacturing labor and 

material costs vary in response to additional system weight growth. 

Sensitivity analysis involves recalculating the cost estimate with different quantitative values for selected 

input values, or parameters, in order to compare the results with the original estimate. If a small change 

in the value of a cost element’s parameter or assumption yields a large change in the overall cost estimate, 
148 Chapter 13 GAO-09-3SP

the results are considered sensitive to that parameter or assumption. Therefore, a sensitivity analysis can 

provide helpful information for the system designer because it highlights elements that are cost sensitive. 

In this way, sensitivity analysis can be useful for identifying areas where more design research could result 

in less production cost or where increased performance could be implemented without substantially 

increasing cost. This type of analysis is typically called a what-if analysis and is often used for optimizing 

cost estimate parameters.

Sensitivity Factors

Uncertainty about the values of some, if not most, of the technical parameters is common early in a 

program’s design and development. Many assumptions made at the start of a program turn out to be 

inaccurate. Therefore, once the point estimate has been developed, it is important to determine how 

sensitive the total cost estimate is to changes in the cost drivers. 

Some factors that are often varied in a sensitivity analysis are

■ a shorter or longer economic life;

■ the volume, mix, or pattern of workload;

■ potential requirements changes;

■ configuration changes in hardware, software, or facilities;

■ alternative assumptions about program operations, fielding strategy, inflation rate, technology 

heritage savings, and development time;

■ higher or lower learning curves; 

■ changes in performance characteristics;

■ testing requirements;

■ acquisition strategy, whether multiyear procurement, dual sourcing, or the like;

■ labor rates; 

■ growth in software size or amount of software reuse; and

■ down-scoping the program.

These are just some examples of potential cost drivers. Many factors that should be tested are determined 

by the assumptions and performance characteristics outlined in the technical baseline description and 

GR&As. Therefore, auditors should look for a link between the technical baseline parameters and the 

GR&As to see if the cost estimator examined those that had the greatest effect on the overall sensitivity of 

the cost estimate. 

In addition, the cost estimator should always include in a sensitivity analysis the assumptions that are most 

likely to change, such as an assumption that was made for lack of knowledge or one that is outside the 

control of the program office. Case study 38 shows some assumptions that can affect the cost of building a 

ship.
GAO-09-3SP Chapter 13 149

Case Study 38: Sensitivity Analysis, from Defense Acquisitions, 

GAO-05-183

Given the uncertainties inherent in ship acquisitions, such as introducing new technologies 

and volatile overhead rates over time, cost analysts face a significant challenge in 

developing credible initial cost estimates. The Navy must develop cost estimates as long 

as 10 years before ship construction begins, before many program details are known. Cost 

analysts therefore have to make a number of assumptions about ship parameters like 

weight, performance, and software and about market conditions, such as inflation rates, 

workforce attrition, and supplier base. 

In the eight case study ships we examined, other unknowns led to uncertain estimates. 

Labor hour and material costs were based on data from previous ships and on unproven 

efficiencies in ship construction. GAO found that analysts often factored in savings based 

on expected efficiencies that never materialized. For example, they anticipated savings 

from implementing computer-assisted design and computer-assisted manufacturing for 

the San Antonio class transport LPD 17, but the contractor had not made the requisite 

research investments to achieve the proposed savings. Similar unproven or unsupported 

efficiencies were estimated for the Arleigh Burke class destroyer DDG 92 and Nimitz 

class aircraft carrier CVN 76. Changes in the shipbuilders’ supplier base also created 

uncertainties in their overhead costs. 

Despite these uncertainties, the Navy did not test the validity of the cost analysts’ 

assumptions in estimating construction costs for the eight case study ships and did not 

identify a confidence level for estimates.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

Steps in Performing a Sensitivity Analysis

A sensitivity analysis addresses some of the estimating uncertainty by testing discrete cases of assumptions 

and other factors that could change. By examining each assumption or factor independently, while 

holding all others constant, the cost estimator can evaluate the results to discover which assumptions 

or factors most influence the estimate. A sensitivity analysis also requires estimating the high and low 

uncertainty ranges for significant cost driver input factors. To determine what the key cost drivers are, a 

cost estimator needs to determine the percentage of total cost that each cost element represents. The major 

contributing variables within the highest percentage cost elements are the key cost drivers that should be 

varied in a sensitivity analysis. A credible sensitivity analysis typically has five steps:

1. identify key cost drivers, ground rules, and assumptions for sensitivity testing;

2. reestimate the total cost by choosing one of these cost drivers to vary between two set amounts—

for example, maximum and minimum or performance thresholds;49

3. document the results;

4. repeat 2 and 3 until all factors identified in step 1 have been tested independently;

5. evaluate the results to determine which drivers affect the cost estimate most.

49The ranges should be documented during data collection and cost estimating (steps 6 and 7).
150 Chapter 13 GAO-09-3SP

Sensitivity analysis also provides important information for economic analyses that can end in the choice 

of a different alternative from the original recommendation. This can happen because, like a cost estimate, 

an economic analysis is based on assumptions and constraints that may change. Thus, before choosing an 

alternative, it is essential to test how sensitive the ranking of alternatives is to changes in assumptions. In 

an economic analysis, sensitivity is determined by how much an assumption must change to result in an 

alternative that differs from the one recommended. For example, an assumption is considered sensitive if a 

10–50 percent change yields a different alternative, very sensitive if the change is less than 10 percent.

Assumptions and cost drivers that have the most effect on the cost estimate warrant further study to 

ensure that the best possible value is used for that parameter. If the cost estimate is found to be sensitive to 

several parameters, all the GR&As should be reviewed, to assure decision makers that sensitive parameters 

have been carefully investigated and the best possible values have been used in the final point estimate. 

Sensitivity Analysis Benefits and Limitations

A sensitivity analysis provides a range of costs that span a best and worst case spread. In general, it is better 

for decision makers to know the range of potential costs that surround a point estimate and the reasons 

behind what drives that range than to just have a point estimate to make a decision from. Sensitivity 

analysis can provide a clear picture of both the high and low costs that can be expected, with discrete 

reasons for what drives them. Figure 14 shows how sensitivity analysis can give decision makers insight.

Figure 14: A Sensitivity Analysis That Creates a Range around a Point Estimate

$10

billion

$11

billion

$13

billion

$9

billion

$12

billion

Life-cycle cost

(FY 07 constant

dollars)

$10.040

$9.940

$10.090

$9.89

$11.099

$9.79

$11.121

$9.75

$12.789

$9.36

Increase the 

number of cost

penalties in

airframe 

development CER

Double the 

development 

testing

Increase airframe

weight

Eliminate 

concurrent 

production

quantities

Increase quality of

materials in aircraft

Use 88% learning

curve instead of 

91%

Eliminate 

integration and 

assembly cost 

add-on

Reduce airframe 

weight

Improve aircraft 

maintainability

Reduce peacetime 

flying hours

+$40.0 million (0.4%) +$50.0 million (0.5%) +$1,009 million (10.0%) +$22.0 million (0.2%) +$1,668 million (15.0%)

-$60.0 million (0.6%)

Increase estimate:

Decrease estimate:

Description:

Description:

-$50.0 million (0.5%) -$100.0 million (1.0%) -$40.0 million (0.4%) -$390.0 million (4.0%)

Increase in life-cycle

cost estimate

Decrease in life-cycle

cost estimate

Source: GAO.

(Point

estimate)
GAO-09-3SP Chapter 13 151

In figure 14, it is very apparent how certain assumptions affect the estimate. For example, increasing 

the quality of materials in the aircraft has the biggest effect on the highest cost estimate—adding 

$1,668 million to the point estimate—while reducing the number of flying hours is the biggest driver 

for reducing the cost estimate—reducing the flying hours saves $390 million. Using visuals like this can 

quickly display what-if analyses that can help management make informed decisions.

A sensitivity analysis also reveals critical assumptions and program cost drivers that most affect the results 

and can sometimes yield surprises. Therefore, the value of sensitivity analysis to decision makers lies in the 

additional information and understanding it brings to the final decision. Sensitivity analysis can also make 

for a more traceable estimate by providing ranges around the point estimate, accompanied by specific 

reasons for why the estimate could vary. This insight allows the cost estimator and program manager to 

further examine potential sources of risk and develop ways to mitigate them early. Sensitivity analysis 

permits decisions that influence the design, production, and operation of a system to focus on the elements 

that have the greatest effect on cost. 

Sensitivity analysis examines only the effect of changing one assumption or factor at a time. But the risk of 

several assumptions or factors varying simultaneously, and its effect on the overall point estimate, should 

be understood. In the next chapter, we discuss risks and uncertainty analyses.

10. Best Practices Checklist: Sensitivity Analysis

 The cost estimate was accompanied by a sensitivity analysis that identified the 

effects of changing key cost driver assumption and factors.

3 Well-documented sources supported the assumption or factor ranges.

3 The sensitivity analysis was part of a quantitative risk assessment and not based 

on arbitrary plus or minus percentages.

3 Cost-sensitive assumptions and factors were further examined to see whether 

design changes should be implemented to mitigate risk.

3 Sensitivity analysis was used to create a range of best and worst case costs.

3 Assumptions and performance characteristics listed in the technical baseline 

description and GR&As were tested for sensitivity, especially those least 

understood or at risk of changing.

3 Results were well documented and presented to management for decisions.

 The following steps were taken during the sensitivity analysis:

3 Key cost drivers were identified.

3 Cost elements representing the highest percentage of cost were determined 

and their parameters and assumptions were examined.

3 The total cost was reestimated by varying each parameter between its 

minimum and maximum range.

3 Results were documented and the reestimate was repeated for each parameter 

that was a key cost driver.

3 Outcomes were evaluated for parameters most sensitive to change.

 The sensitivity analysis provided a range of possible costs, a point estimate, and a 

method for performing what-if analysis.
152 Chapter 13 GAO-09-3SP
GAO-09-3SP Chapter 14 153

Cost Risk and Uncertainty

Chapter 14

In chapter 13, we discussed sensitivity analysis and how it is useful for performing what-if analysis, 

determining how sensitive the point estimate is to changes in the cost drivers, and developing ranges 

of potential costs. A drawback of sensitivity analysis is that it looks only at the effects of changing one 

parameter at a time. In reality, many parameters can change at the same time. Therefore, in addition 

to a sensitivity analysis, an uncertainty analysis should be performed to capture the cumulative effect of 

additional risks. 

Because cost estimates predict future program costs, uncertainty is always associated with them. For 

example, data from the past may not always be relevant in the future, because new manufacturing 

processes may change a learning curve slope or new composite materials may change the relationship 

between weight and cost. Moreover, a cost estimate is usually composed of many lower-level WBS 

elements, each of which comes with its own source of error. Once these elements are added together, the 

resulting cost estimate can contain a great deal of uncertainty.

The Difference Between Risk and Uncertainty

Risk and uncertainty refer to the fact that because a cost estimate is a forecast, there is always a chance 

that the actual cost will differ from the estimate. Moreover, lack of knowledge about the future is only one 

possible reason for the difference. Another equally important reason is the error resulting from historical 

data inconsistencies, assumptions, cost estimating equations, and factors typically used to develop an 

estimate. 

In addition, biases are often found in estimating program costs and developing program schedules. The 

biases may be cognitive—often based on estimators’ inexperience—or motivational, where management 

intentionally reduces the estimate or shortens the schedule to make the project look good to stakeholders. 

Recognizing the potential for error and deciding how best to quantify it is the purpose of risk and 

uncertainty analysis.50

It is inaccurate to add up the most likely WBS elements to derive a program cost estimate, since their sum 

is not usually the most likely estimate for the total program, even if they are estimated without bias.51 Yet 

summing costs estimated at the detailed level to derive a point estimate is the most common approach to 

estimating a total program. Simulation of program risks is a better way to estimate total program cost, as 

we discuss below.

50Many good references outline the cost risk and uncertainty modeling process. One example is the Air Force Cost Analysis 

Agency’s Cost Risk and Uncertainty Analysis Handbook (Goleta, Calif.: Tecolote Research, Inc., April 2007).

51 See Stephen A. Book, “Do Not Sum ‘Most Likely’ Costs,” presentation to American Society of Military Comptrollers, Los 

Angeles, California, April 30, 2002.
154 Chapter 14 GAO-09-3SP

Quantifying risk and uncertainty is a cost estimating best practice addressed in many guides and 

references. DOD specifically directs that uncertainty be identified and quantified. The Clinger-Cohen 

Act requires agencies to assess and manage the risks of major information systems, including the 

application of the risk-adjusted return on investment criterion in deciding whether to undertake particular 

investments.52

While risk and uncertainty are often used interchangeably, in statistics their definitions are distinct: 

Risk is the chance of loss or injury. In a situation that includes favorable and unfavorable 

events, risk is the probability that an unfavorable event will occur.

Uncertainty is the indefiniteness about the outcome of a situation. It is assessed in cost 

estimate models to estimate the risk (or probability) that a specific funding level will be 

exceeded.53

Therefore, while both risk and uncertainty can affect a program’s cost estimate, enough data will never be 

available in most situations to develop a known frequency distribution. Cost estimating is analyzed more 

often for uncertainty than risk, although many textbooks use both terms to describe the effort.

Point Estimates Alone Are Insufficient for Good Decisions

Since cost estimates are uncertain, making good predictions about how much funding a program needs to 

be successful is difficult. In a program’s early phases, knowledge about how well technology will perform, 

whether the estimates are unbiased, and how external events may affect the program is imperfect. For 

management to make good decisions, the program estimate must reflect the degree of uncertainty, so that 

a level of confidence can be given about the estimate.

Quantitative risk and uncertainty analysis provide a way to assess the variability in the point estimate. 

Using this type of analysis, a cost estimator can model such effects as schedules slipping, missions 

changing, and proposed solutions not meeting user needs, allowing for a known range of potential costs. 

Having a range of costs around a point estimate is more useful to decision makers, because it conveys 

the level of confidence in achieving the most likely cost and also informs them on cost, schedule, and 

technical risks.

Point estimates are more uncertain at the beginning of a program, because less is known about its detailed 

requirements and opportunity for change is greater. In addition, early in a program’s life cycle, only 

general statements can be made. As a program matures, general statements translate into clearer and more 

refined requirements that reduce the unknowns. However, more refined requirements often translate into 

additional costs, causing the distribution of potential costs to move further to the right, as illustrated in 

figure 15. 

52 40 U.S.C. § 11312 (Supp. IV 2004).

53© 2000 From Probability Methods for Cost Uncertainty Analysis by Paul Garvey. Reproduced by permission of Taylor and Francis 

Group, LLC, a division of Informa PLC.
GAO-09-3SP Chapter 14 155

Figure 15: Changes in Cost Estimate Uncertainty across the Acquisition Life Cycle

Cost

Concept

formulation Estimate Uncertainty Development Implementation

0

Cost

estimate:

$175 million

Cost

estimate:

$230 million

Cost

estimate:

$125 million

Source: GAO.

While the point estimate increases in figure 15, the uncertainty range around it decreases. More is learned 

as the project matures. First, a better understanding of the risks is achieved, and either some risk is retired 

or some form of risk handling lessens the potential cost or effect on schedule. Second, the program is 

understood better and, most probably, more requirements are added or overlooked as elements are added, 

which has a tendency to increase costs along with reducing the variance. Thus, a point estimate, by itself, 

provides no information about the underlying uncertainty other than that it is the value chosen as most 

likely. 

A confidence interval, in contrast, provides a range of possible costs, based on a specified probability 

level. For example, a program with a point estimate of $10 million could range in cost from $5 million to 

$15 million at the 95 percent confidence level. In addition, the probability distribution, usually in the form 

of a cumulative distribution or S curve (described below) can provide the decision maker with an estimate 

of the probability that the program’s cost will actually be at some value or lower. Conversely, 1.0 minus 

this probability is the probability that the project will overrun that value. 

Using an uncertainty analysis, a cost estimator can easily inform decision makers about a program’s 

potential range of costs. Management, in turn, can use these data to decide whether the program fits 

within the overall risk range of the agency’s portfolio. 

Budgeting to a Realistic Point Estimate

Over the years, GAO has reported that many programs overrun their budgets because original point 

estimates are unrealistic. Case studies 39 and 40 are examples.
156 Chapter 14 GAO-09-3SP

Case Study 39: Point Estimates, from Space Acquisitions, GAO-07-96

Estimated costs for DOD’s major space acquisitions increased about $12.2 billion, or 

nearly 44 percent, above initial estimates for fiscal year 2006 through fiscal year 2011. 

GAO identified a variety of reasons for this. The most notable are that weapons programs 

have incentives to produce and use optimistic cost and schedule estimates to compete 

successfully for funding and that DOD starts its space programs before it has assurance 

that the capabilities it is pursuing can be achieved within its resource and time constraints. 

At the same time, the cost growth resulted partly from DOD’s using low cost estimates to 

establish program budgets, finding it necessary later to make funding shifts with costly, 

reverberating effects. In 2003, a DOD study found that the space acquisitions system was 

strongly biased to produce unrealistically low cost estimates throughout the process. 

The study found that most programs at contract initiation had a predictable cost growth 

of 50 percent to 100 percent. It found that the unrealistically low projections of program 

cost and the lack of provisions for management reserve seriously distorted management 

decisions and program content, increased risks to mission success, and virtually 

guaranteed program delays. GAO found most of these conditions in many DOD programs.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Case Study 40: Point Estimates, from Defense Acquisitions, GAO-05-183

For several case study ships, the costs of materials increased dramatically above the 

shipbuilder’s initial plan. Materials cost was the most significant component of cost growth 

for the CVN 76 in the Nimitz class of aircraft carriers, the LPD 17 in the San Antonio class 

of transports, and the SSN 775 in the Virginia class of submarines. The growth in materials 

costs resulted, in part, from Navy and shipbuilders underbudgeting for these costs.

For example, the materials budget for the first four Virginia class submarines was $132 

million less than quotes received from vendors and subcontractors. The shipbuilder agreed 

to take on the challenge of achieving lower costs in exchange for providing in the contract 

that the shipbuilder would be reimbursed for cost growth in high-value, specialized 

materials. 

In addition, the materials budget for the CVN 76 and CVN 77 was based on an incomplete 

list of materials needed to construct the ships, leading to especially sharp increases in 

estimated materials costs. In this case, the Defense Contract Audit Agency criticized the 

shipbuilder’s estimating system, particularly the system for materials and subcontract 

costs, stating that the resulting estimates “do not provide an acceptable basis for 

negotiation of a fair and reasonable price.” Underbudgeting of materials contributed to 

cost growth, recognized in the fiscal year 2006 budget. 

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

We have found that budgeting programs to a risk-adjusted estimate that reflects a program’s risks is 

critical to its successfully achieving its objectives. However, programs have developed optimistic estimates 
GAO-09-3SP Chapter 14 157

for many reasons. Cost estimators may have ignored program risk, underestimated data outliers, relied 

on historical data that may be misleading for a new technology, or assumed better productivity than the 

historical data supported, causing narrow uncertainty ranges. Decision makers may add their own bias for 

political or budgetary reasons. For example, they may make optimistic assumptions by assuming that a 

new program will perform much better than its predecessor in order to justify a preconceived notion, to fit 

the program within unrealistic budgetary parameters, or just to sell the program. 

One way to determine whether a program is realistically budgeted is to perform an uncertainty analysis, 

so that the probability associated with achieving its point estimate can be determined. A cumulative 

probability distribution, more commonly known as an S curve—usually derived from a simulation such 

as Monte Carlo—can be particularly useful in portraying the uncertainty implications of various cost 

estimates. Figure 16 shows an example of a cumulative probability distribution with various cost estimates 

mapped to a certain probability level.

Figure 16: A Cumulative Probability Distribution, or S Curve 

Source: GAO and NASA.

0

0

Dollars in thousands

200 400 600 800 1,000 1,200 1,400 1,600 1,800

20

40

60

80

100

Probability of occurrence

70% probability = 1,096

50% probability = 907.9

The risk-adjusted primary

estimate = 825, or 40% probable

In figure 16, one can readily see that given what is known about program risks and uncertainties, 

the least this hypothetical program could cost is about $500,000, at about 5 percent probability; the 

most, $1,700,000 or less, at about 95 percent probability. Using an S curve, decision makers can easily 

understand what the likelihood of different funding alternatives will imply.54

For example, according to the S curve in figure 16, the point estimate has up to a 40 percent chance of 

being met, meaning there is a 60 percent chance that costs will be greater than $825,000. On the basis 

54The simulation quantifies the imperfectly understood risks in the program after any agreed-on mitigation has been incorporated. 

Unknown unknowns, risks that are not known when the analysis is done, may require periodic risk analysis leading to improvement 

of the estimate of uncertainty.
158 Chapter 14 GAO-09-3SP

of this information, management could decide to add $82,900 to the point estimate to increase the 

probability to 50 percent or $271,000 to increase the confidence level to 70 percent. The important thing 

to note, however, is the large cost increase between the 70 percent and 95 percent confidence levels—

about $600,000—indicating that a substantial investment would be necessary to reach a higher level of 

certainty. 

Management can use the data in an S curve to choose a defensible level of contingency reserves. While 

no specific confidence level is considered a best practice, experts agree that program cost estimates should 

be budgeted to at least the 50 percent confidence level, but budgeting to a higher level (for example, 

70 percent to 80 percent, or the mean) is now common practice. Moreover, they stress that contingency 

reserves are necessary to cover increased costs resulting from unexpected design complexity, incomplete 

requirements, technology uncertainty, and industrial base concerns, to name a few uncertainties that can 

affect programs. 

How much contingency reserve should be allocated to a program beyond the 50 percent confidence level 

depends on the program cost growth an agency is willing to risk. Some organizations adopt other levels 

like the 70th or 80th percentile (refer to the S curve above) to 

1. reduce their anxiety about success within budget, 

2. make some provision for risks unknown at the time but likely to appear as the project progresses, 

and 

3. reduce the probability that they will have to explain overruns or rebaseline because they ran out of 

reserve budget. 

The amount of contingency reserve should be based on the level of confidence with which management 

chooses to fund a program, based on the probabilities reported in the S curve. In figure 16, management 

might choose to award a contract for $907,900 but fund the program at $1,096,000. This alternative 

would provide an additional $188,000 in contingency reserve at the 70 percent confidence level. The 

result would be only a 30 percent chance that the program would need additional funding, given the 

identification and quantification of the risks at the time of the analysis.

Another benefit of using an S curve is that management can proactively monitor a program’s costs, 

because it knows the probability for incurring overruns. By understanding which input variables have a 

significant effect on a program’s final costs, management can devote resources to acquire better knowledge 

about them so that risks can be minimized. Finally, knowing early what the potential risks are enables 

management to prepare contingencies to monitor and mitigate them using an EVM system once the 

program is under contract. 

The bottom line is that management needs a risk-adjusted point estimate based on an estimate of the 

range of confidence to make wise decisions. Using information from an S curve with a realistic probability 

distribution, management can quantify the level of confidence in achieving a program within a certain 

funding level. It can also determine a defensible amount of contingency reserve to quickly mitigate risk.
GAO-09-3SP Chapter 14 159

Developing a Credible S Curve of Potential Program Costs

Since an S curve is vital to knowing how much confidence management can have in a given point 

estimate, it is important to know the activities in developing one. Seven steps are associated with 

developing a justifiable S curve:

1. determine the program cost drivers and associated risks;

2. develop probability distributions to model various types of uncertainty (for example, program, 

technical, external, organizational, program management including cost estimating and 

scheduling);

3. account for correlation between cost elements to properly capture risk; 

4. perform the uncertainty analysis using a Monte Carlo simulation model;

5. identify the probability level associated with the point estimate; 

6. recommend sufficient contingency reserves to achieve levels of confidence acceptable to the 

organization; and

7. allocate, phase, and convert a risk-adjusted cost estimate to then-year dollars and identify high-risk 

elements to help in risk mitigation efforts.

To take these steps, the cost estimator must work with the program office and technical experts to collect 

the proper information. Short-changing or merely guessing at the first two steps does not lead to a credible 

S curve and can give management a false sense of confidence in the information.

Step 1: Determine Program Cost Drivers and Associated Risks

In chapter 13, we noted that one of the benefits of a sensitivity analysis is a list of the program cost drivers. 

Since numerous risks can influence the estimate, they should be examined for their sources of uncertainty 

and potential effect, and they should be modeled to determine how they can affect the uncertainty of the 

cost estimate. For example, undefined or unknown technical information, uncertain economic conditions, 

unexpected schedule problems, requirements growth, security level changes, and political issues are 

often encountered during a program’s acquisition. Each of these risks can negatively or positively affect 

a program’s cost. This means that uncertainty can cause the actual cost or schedule to differ from any 

current plan either in a positive or beneficial direction or in a negative or harmful direction. In addition, 

new technologies may be proposed that can fail outright, causing rework and unexpected cost growth. 

Risks are also associated with the estimating process itself. For instance, historical data from which 

to make a credible estimate can be lacking. When this happens, a cost estimator has no choice but to 

extrapolate with existing methods or develop a new estimating approach. No matter the method, some 

error will be introduced into the estimate. 

Accounting for all possible risks is necessary to adequately capture the uncertainty associated with a 

program’s point estimate. Far from exhaustive, table 21 describes some of the many sources of risk. It is 

only a starting point, since each program is unique.
160 Chapter 14 GAO-09-3SP

Table 21: Potential Sources of Program Cost Estimate Uncertainty

Uncertainty Definition Example

Business or 

economic 

Variations from change 

in business or economic 

assumptions

Changes in labor rate assumptions—e.g., wages, overhead, 

general and administrative cost—supplier viability, inflation 

indexes, multiyear savings assumptions, market conditions, 

and competitive environment for future procurements

Cost estimating Variations in the cost estimate 

despite a fixed configuration 

baseline

Errors in historical data and cost estimating relationships, 

variation associated with input parameters, errors with 

analogies and data limitations, data extrapolation errors, 

optimistic learning and rate curve assumptions, using the 

wrong estimating technique, omission or lack of data, 

misinterpretation of data, incorrect escalation factors, 

overoptimism in contractor capabilities, optimistic savings 

associated with new ways of doing business, inadequate 

time to develop a cost estimate

Program Risks outside the program 

office control

Program decisions made at higher levels of authority, 

indirect events that adversely affect a program, directed 

funding cuts, multiple contractor teams, conflicting 

schedules and workload, lack of resources, organizational 

interface issues, lack of user input when developing 

requirements, personnel management issues, organization’s 

ability to accept change, other program dependencies

Requirements Variations in the cost estimate 

caused by change in the 

configuration baseline from 

unforeseen design shifts

Changes in system architecture (especially for system of 

systems programs), specifications, hardware and software 

requirements, deployment strategy, critical assumptions, 

program threat levels, procurement quantities, network 

security, data confidentiality

Schedule Any event that changes 

the schedule: stretching it 

out may increase funding 

requirements, delay delivery, 

and reduce mission benefits

Amount of concurrent development, changes in 

configuration, delayed milestone approval, testing failures 

requiring rework, infeasible schedule with no margin, overly 

optimistic task durations, unnecessary activities, omission of 

critical reviews

Software Cost growth from overly 

optimistic assumptions about 

software development

Underestimated software sizing, overly optimistic software 

productivity, optimistic savings associated with using 

commercial off-the-shelf software, underestimated 

integration effort, lack of commercial software 

documentation, underestimating the amount of glue 

code needed, configuration changes required to support 

commercial software upgrades, changes in licensing fees, 

lack of support for older software versions, lack of interface 

specification, lack of software metrics, low staff capability 

with development language and platform, underestimating 

software defects

Technology Variations from problems 

associated with technology 

maturity or availability

Uncertainty associated with unproven technology, 

obsolete parts, optimistic hardware or software heritage 

assumptions, feasibility of producing large technology 

leaps, relying on lower reliability components, design errors 

or omissions

 Source: DHS, DOD, DOE, NASA, OMB, SCEA, and industry. 
GAO-09-3SP Chapter 14 161

Collecting high-quality risk data is key to a successful analysis of risk. Often there are no historical data 

from which to derive the information needed as inputs to a risk analysis of cost or schedule. Usually most 

risk data are derived from in-depth interviews or in risk workshops. In other words, the data used in 

program risk analyses are often based on individuals’ expert judgment, which depends on the experience 

of the interviewees and may be biased. The success of data collection depends also on the risk maturity 

of the organization’s culture. It is difficult to collect useful risk analysis data when the organization is 

indifferent or even hostile to expressing risk in the program. Obtaining risk information from staff outside 

the acquisition program office can help balance potential optimism.

After identifying all possible risks, a cost estimator needs to define each one in a way that facilitates 

determining the probability of each risk occurring, along with the cost effect. To do this, the estimator 

needs to identify a range of values and their respective probabilities—either based on specific statistics 

or expressed as best case, worst case, and most likely—and the rationale for choosing the variability 

discussed. While the best practice is to rely on historical data, if these data are not available, how 

qualitative judgment was applied should be explained (e.g., not planning for first time success in testing). 

Because the quality and availability of the data affect the cost estimate’s uncertainty, these should be well 

documented and understood. For example, a cost estimate based on detailed actual data in significant 

quantities will yield a more confident estimate than one based on an analogy using only a few data points. 

Since collecting all this information can be formidable, it should be done when the data are collected to 

develop the estimate. Interviews with experts familiar with the program are good sources of how varied 

the risks are for a particular cost element. However, experts do not always think in extremes. They 

tend instead to estimate probability ranges that represent only 60 percent to 85 percent of the possible 

outcomes, so adjustments may have to be made to consider a wider universe of risks. In addition, the 

technical baseline description should address the minimum and maximum range, as well as the most 

likely value for critical program parameters.

Several approaches, ranging from subjective judgment to complex statistical techniques, are available 

for dealing with uncertainty. Here we describe different ways of determining the uncertainty of a cost 

estimate.

Cost Growth Factor

Using the cost growth factor, the cost estimator reflects on assumptions and judgments from the 

development of the cost estimate and then makes a final adjustment to the estimate. This is usually a 

percentage increase, based on historical data from similar programs, or an adjustment solicited from 

expert opinion and based on experience. This yields a revised cost estimate that explicitly recognizes the 

existence of uncertainty. It can be applied at the total program level or for one or more WBS elements. 

The advantages of this approach are that it is easy to implement, takes little time to perform, and requires 

minimal detail. Its several problems are that it requires access to a credible historical database, the selection 

of comparative projects and adjustment factors that can be subjective, and new technologies or lessons 

learned that may cause historical data to be less relevant.

Expert Opinion

An independent panel of experts can be gathered to review, understand, and discuss the system and its 

costs, in order to quantify the estimate’s uncertainty and adjust the point estimate. This approach is often 
162 Chapter 14 GAO-09-3SP

used in conjunction with the Delphi technique, in which several experts provide opinions independently 

and anonymously. The results are summarized and returned to the experts, who are then given the 

opportunity to change or modify their opinions, based on the opinions of the group as a whole. If 

successful, after several such iterations, the expert opinions converge. 

The strengths of this approach are directly related to the diversity and experience of the panel members. 

The major weaknesses are that it can be time consuming and experts can present biased opinions. For 

example, some of the largest risks facing a program may stem from a new technology for which there 

is little previous experience. If the risk distributions rest on the beliefs of the same experts who may be 

stakeholders, it could be difficult to truly capture the program risks. A typical rule of thumb is that 

lower and upper bounds estimated by experts should be interpreted as representing the 15 percent and 

85 percent levels, respectively, of all possible outcomes. Therefore, the cost estimator will need to adjust 

the distribution bounds to account for skew (see step 2 for more on this issue). Cost estimators can also 

mitigate bias by avoiding leading questions and by questioning all assumptions to see if they are backed by 

historical data.

The analytic hierarchy process, like the Delphi technique, is another approach to making the best 

of expert opinion. It can be applied to the opinion of either an individual or a panel of experts and 

mitigates the problems of bias that result from group think or dominating personalities. The analytic 

hierarchy process provides a structured way to address complicated decisions: it relies on a framework for 

quantifying decision elements and evaluating various alternatives. This process allows for effective decision 

making because it captures both subjective and objective evaluation parameters, which can lead to less bias 

and help determine the best overall decision. The approach relies on mathematics to organize pair-wise 

comparisons of decision components and prioritizes the results to arrive at a stable outcome. 

Mathematical Approaches

Mathematical approaches rely on statistics to describe the variance associated with an analogy or a cost 

estimating relationship. The most common approach is to collect data on the optimistic, most likely, 

and pessimistic range (the “3-point estimate”) for the risk or the cost element schedule activity duration. 

Statistics like the standard error of the estimate and confidence intervals are more difficult to collect 

from program participants and are not commonly used. Some distributions use more exotic inputs such 

as “shape parameters” that are often difficult to collect, even in the most in-depth interviews. Therefore, 

the 3-point estimate and an idea about the distribution shape can be used to define the probability 

distribution to be used in a simulation. Probability distributions are used either to characterize risks that 

are assigned to cost elements or activity durations or as estimates of uncertainty in costs or durations 

that may be affected by several risks. With either of these approaches, in the simulation the lower-level 

WBS element cost probabilities are combined to develop an overall program cost estimate probability 

distribution. 

A benefit of this approach is that it complements the decomposition approach to cost estimating. In 

addition, the emergence of commercial software models means that Monte Carlo simulation can be 

implemented quickly and easily, once all the data have been collected. Some drawbacks to the approach 

include that input distributions can be various, correlation between cost elements needs to be included, 

and decision makers may not always accept the output. In addition, high-quality risk data are sometimes 

difficult and may be expensive to collect.
GAO-09-3SP Chapter 14 163

Technology Readiness Levels

NASA and the Air Force Space Command, among other organizations, address uncertainty by applying 

readiness levels, which capture the risk associated with developing state-of-the-art technology. They have 

historically developed technology readiness levels to indicate how close a given technology is to being 

available. Technology readiness levels are rated on a scale from 1 to 9, with 1 representing paper studies of 

a technology’s feasibility and 9 representing technology completely integrated into a finished product. In 

appendix XII, we list and describe nine technology readiness levels. 

Knowing a technology’s readiness level allows a cost estimator to judge the risk inherent in assuming it 

will be available for a given program. For example, GAO has determined that level 7—demonstration 

of a prototype in an operational environment—is the level of technological maturity that constitutes low 

risk for starting a product development program. One needs to be cautious, since programs can inflate 

the level. There should be specific evidence that a program has achieved the claimed technology readiness 

level, such as that physical and functional interfaces are clearly defined, raw materials are available, and 

manufacturing procedures are set up and undergoing testing for proof of concept before accepting a claim 

as true. 

Software Engineering Institute Maturity Models

SEI has developed a variety of models that provide a logical framework for assessing whether an 

organization has the necessary process discipline to repeat earlier successes on similar projects. 

Organizations that do not satisfy the requirements for the “repeatable” level are by default judged to be at 

the initial level of maturity—meaning that their processes are ad hoc, sometimes even chaotic, with few 

of the processes defined and success dependent mainly on the heroic efforts of individuals. The lower the 

maturity, the higher the risk that a program will incur cost overruns. 

In addition to evaluating software risks, SEI’s risk evaluation method can be tailored to address hardware 

and organizational risks with a program. This method includes identifying and quantifying risk using a 

repeatable process for eliciting risks from experts. Furthermore, using SEI’s taxonomy, the risk evaluation 

method provides a consistent framework for employing risk management methods and mitigation 

techniques. 

Schedule Risk Analysis

Schedule risk analysis captures the risk that schedule durations may increase from technical challenges, 

lack of qualified personnel, and too few staff to do the work. Schedule risk analysis examines the effect 

of activities and events slipping on a program’s critical path or the longest path through the network 

schedule. A program schedule delay will have cost effects for all aspects of a program, including systems 

engineering and program management. It also analyzes how various activities affect one another because 

of precedence relationships—activity C cannot begin until activities A and B are finished—and how a slip 

in one activity affects the duration of other activities when concurrence is high among tasks. By applying 

probabilistic distributions to capture the uncertainty with traditional early start–late start and early finish–

late finish schedule durations, using optimistic, pessimistic, and most likely values, a cost estimator can 

draw a better picture of the true critical path and any cost effects to the program. In addition, this analysis 

addresses the feasibility of the program plan as well as the effect of not meeting the anticipated finish date.
164 Chapter 14 GAO-09-3SP

Risk Cube (Probability Impact Matrix) Method

The risk cube method prioritizes uncertainties that could jeopardize program cost, schedule, performance, 

and quality objectives in terms of probability of occurrence and cost effect. Subject matter experts, 

typically engineers and others familiar with the program, define the risk factors, probabilities, and cost 

effect for each identified risk. Using these data, the cost estimator develops the expected cost overrun 

by multiplying the cost impact by each risk factor’s probability of occurrence. A common technique for 

engaging those knowledgeable about the program is creating a two-dimensional matrix like the one in 

figure 17.

Figure 17: A Risk Cube Two-Dimensional Matrix

Risk

Consequence

The program penalty 

incurred if the objective is 

not obtained

Probability

The likelihood that 

an objective will not 

be met if the current 

plan is used

Source: GAO.

LOW

MEDIUM

HIGH

In the risk cube (P-I matrix) method, risks are mapped onto the matrix, based on the severity of the 

consequence—ranging from low risk = 1 to high risk = 5—and the likelihood of their occurring—

ranging from low likelihood = 1 to high = 5. Risks that fall in the upper right quadrant are the most likely 

to occur and have the greatest consequences for the program, compared to risks that fall into the lower left 

quadrant. 

When risks are plotted together, management can quickly determine which ones have top priority. For 

a risk cube (P-I matrix) analysis to be accurate, complete lists of all risks are needed, as well as accurate 

probabilities of occurrence and cost impacts. Determining the cost impact will vary by program and WBS 

element, but a cost impact could, for example, be categorized as “60 percent more funding is required to 

resolve a technical shortfall that has no acceptable workarounds.” Once the cost impacts are identified, 

they are mapped to the appropriate WBS elements to help identify risk mitigation steps that would be 

most beneficial. 

The advantages of using this approach are that those knowledgeable about the program can readily 

understand and relate to risks presented in this manner and that decision makers can understand the 

link between specific risks and consequences. A disadvantage is that engineers may not always know the 
GAO-09-3SP Chapter 14 165

cost impacts and may not account for the full spectrum of possible outcomes. Moreover, this method can 

underestimate total risk by omitting the correlation between technical risk and level of effort in activities 

like program management.

Risk Scoring

Risk scoring quantifies and translates risks into cost impacts. Risk scoring is used to determine the 

amount of risk, preferably using an objective method in which the intervals between a score have 

meaning—a score of 1 = low risk, a score of 5 = medium risk, and a score of 10 = high risk. This 

method is used most often to determine technical risk associated with hardware and software. The 

following categories are used for hardware: technology advancement (level of maturity), engineering 

development (current stage of development), reliability (operating time without failure), producibility (ease 

to manufacture), alternative item (availability of back-up item), and schedule (amount of aggressiveness). 

Table 22 is an example of the hardware risk scoring matrix.55

Table 22: A Hardware Risk Scoring Matrix 

Risk category

Risk score: 0 = low, 5 = medium, 10 = high

0 1–2 3–5 6–8 9–10

1. Technology 

advancement

Completed, 

state of the art

Minimum 

advancement 

required

Modest 

advancement 

required

Significant 

advancement 

required

New 

technology

2. Engineering 

development

Completed, fully 

tested

Prototype Hardware 

and software 

development

Detailed design Concept 

defined

3. 

Reliability

Historically high 

for same system

Historically 

high on similar 

systems

Modest problems 

known

Serious 

problems known

Unknown

4. Producibility Production and 

yield shown on 

same system

Production and 

yield shown on 

similar system

Production and 

yield feasible

Production 

feasible and 

yield problems

No known 

production 

experience

5.

Alternative 

item

Exists or 

availability on 

other items not 

important

Exists or 

availability on 

other items 

somewhat 

important

Potential 

alternative in 

development

Potential 

alternative in 

design

Alternative 

does not exist 

and is required

6.

Schedule 

Easily achieved Achievable Somewhat 

challenging

Challenging Very 

challenging

Source: © 2003, Society of Cost Estimating and Analysis (SCEA), “Cost Risk Analysis.”

In addition to hardware, categories for software include technology approach (level of innovation), design 

engineering (current stage of development), coding (code maturity), integrated software (based on the 

source lines of code count), testing (amount completed), alternatives (availability of back-up code), and 

schedule (amount of aggressiveness). A software risk scoring matrix is shown in table 23.

55The original approach to this impact-only assessment was Floyd Maxwell’s of the Aerospace Corporation. Since he used it for 

many years at Aerospace, it was originally called the “Maxwell Matrix.”
166 Chapter 14 GAO-09-3SP

Table 23: A Software Risk Scoring Matrix

Risk category

Risk score: 0 = low, 5 = medium, 10 = high

0 1–2 3–5 6–8 9–10

1. 

Technology 

advancement

Proven 

conventional 

analytic 

approach, 

standard 

methods

Undemonstrated 

conventional 

approach, 

standard 

methods

Emerging 

approaches, new 

applications

Unconventional 

approach, 

concept in 

development

Unconventional 

approach, 

concept 

unproven

2.

Design 

engineering

Design complete 

and validated

Specifications 

defined and 

validated

Specifications 

defined

Requirements 

defined

Requirements 

partly defined

3.

Coding

Fully integrated 

code available 

and validated

Fully integrated 

code available

Modules 

integrated

Modules 

exist but not 

integrated

Wholly new 

design, no 

modules exist

4.

Integrated 

software

Thousands of 

instructions

Tens of 

thousands of 

instructions

Hundreds of 

thousands of 

instructions

Millions of 

instructions

Tens of millions 

of instructions

5.

Testing

Tested with 

system

Tested by 

simulation

Structured 

walk-throughs 

conducted

Modules tested 

but not as a 

system

Untested 

modules

6.

Alternatives

Alternatives 

exist, alternative 

design not 

important

Alternatives 

exist; design 

somewhat 

important

Potential for 

alternatives in 

development

Potential 

alternatives 

being 

considered

Alternative does 

not exist but is 

required

7.

Schedule and 

management 

Relaxed 

schedule, serial 

activities, high 

review cycle 

frequency, early 

first review

Modest schedule, 

few concurrent 

activities, review 

cycle reasonable

Modest schedule, 

many concurrent 

activities, 

occasional 

reviews, late first 

review

Fast track on 

schedule, many 

concurrent 

activities

Fast track, 

missed 

milestones, 

review at 

demonstrations 

only, no periodic 

reviews

Source: U.S. Air Force.

Technical engineers score program elements between 0 and 10 for each category and then rank the 

categories according to the program’s effect. Next, each element’s risk score is translated into a cost impact 

by (1) multiplying a factor by an element’s estimated cost (for example, a score of 2 increases the cost of an 

element by 10 percent) or (2) multiplying a factor by predetermined costs (a score of 2 has a cost impact 

of $50,000) or (3) developing a weighted average risk assessment score that is mapped to a historical cost 

growth distribution. 

After using one or several of these methods to determine the cost risk, the estimator’s next step is to choose 

probability distributions to model the risk for each WBS cost element that has uncertainty.

Step 2: Develop Probability Distributions to Model Uncertainty

Uncertainty is best modeled with a probability distribution that accounts for all possible outcomes 

according to the probability that they will occur. Figure 18 gives an example of a known distribution that 

models all outcomes associated with rolling a pair of dice. 
GAO-09-3SP Chapter 14 167

Figure 18: The Distribution of Sums from Rolling Two Dice 

Probability

Value

0 probability

that outcome is

less than 2

100% probability

that outcome will

not exceed 12

50% probability

that outcome is

above or below 7

(this is the median)

Most likely

outcome

Source: GAO.

In figure 18, the horizontal axis shows the potential value of dice rolls, while the vertical axis shows the 

probability associated with each roll. The value at the midpoint of all rolls is the median. In the example, 

the median is also the most likely value (that is, average = a roll of 7), because the outcomes associated with 

rolling a pair of dice are symmetric. 

Besides descriptive statistics, probability distributions provide other useful information, such as the 

boundaries of an outcome. For example, the lower bound in figure 18 is 2 and the upper bound is 12. 

By examining the distribution, it is easy to see that both the upper and lower bounds have the lowest 

probability of occurring, while the chances of rolling a 6, 7, or 8 are much greater. 

It is difficult to pick an appropriate probability distribution for the point cost estimate as a whole, because 

it is composed of several subsidiary estimates based on the WBS. These WBS elements are often estimated 

with a variety of techniques, each with its own uncertainty distributions that may be asymmetrical. 

Therefore, just simply adding the most likely WBS element costs does not result in the most likely cost 

estimate because the risk distributions associated with the subelements differ. 

One way to resolve this issue is to create statistical probability distributions for each WBS element or risk 

by specifying the risk shape and bounds that reflect the relative spread and skewness of the distribution. 

The probability distribution represents the risk shape, and the tails of the distribution reflect the best 

and worst case outcomes. Even though the bounds are extremes and unlikely to occur, the distribution 

acknowledges the possibility and probability that they could happen. Probability distributions are typically 

determined using the 3-point estimates of optimistic, most likely, and pessimistic values to identify the 

amount of spread and skewness of the data. However, if risks are used directly, they will be assigned to 

specific cost elements or activities in a schedule and will perform appropriately in a simulation.56

56Risks can be entered directly or they can be assigned as multiplication factors to specific cost elements or schedule activities. If 

this “risk driver” approach is used, the data collected, including probability of occurrence and impact (typically a 3-point estimate), 

will be on the risks themselves. Hence, the focus is on the risks, not on their impact on activities or cost line items. This focus on the 

risks makes it easy to understand the results and to focus on mitigating risks directly.
168 Chapter 14 GAO-09-3SP

Using a simulation tool such as Monte Carlo, a cost estimator can develop a statistical summation of all 

probable costs, allowing for a better understanding of how likely it is that the point estimate can be met. 

A Monte Carlo simulation also does a better job of capturing risk, because it takes into consideration that 

some risks will occur while others may not. Furthermore, the simulation can adjust the risks beyond the 

upper and lower bounds to account for the fact that experts do not typically think in extremes. Figure 19

shows why different WBS element distributions need to be statistically summed in order to develop the 

overall point estimate probability distribution. 

Figure 19: A Point Estimate Probability Distribution Driven by WBS Distributions

+ ... =

RPE RPE RPE

Confidence level

+ +

6RPE Cost

Probability density

RPE

Bell curve S curve

Cost = x1 + x2 + x3 + ... + xn

x1 x2 xn

Inputs Outputs

Probability distributions for each cost element in a system’s work breakdown structure A cumulative probability distribution of the system’s total cost

20

50

70

85

100

Source: NASA.

Note: RPE = reference point estimate.

In figure 19, the sum of the reference point estimates has a low level of probability on the S curve. In other 

words, there is only a 20 percent chance or less of meeting the point estimate cost. Therefore, in order to 

increase the confidence in the program cost estimate, it will be necessary to add more funding to reach a 

higher level of confidence.

Next to knowing the bounds or 3-point estimates for the uncertainty of the WBS element or risk, 

choosing the right probability distribution for each WBS element is important for capturing the 

uncertainty correctly. For any WBS element, selecting the probability distribution should be based on 

how effectively it models actual outcomes. Since different distributions model different types of risk, 

knowing the shape of the distribution helps in visualizing how the risk will affect the overall cost estimate 

uncertainty. A variety of probability distribution shapes are available for modeling cost risk. Table 24 lists 

eight of the most common probability distributions used in cost estimating uncertainty analysis. 

Table 24: Eight Common Probability Distributions

Distribution Description Shape Typical application

Bernoulli Assigns probabilities of “p” for 

success and “1 – p” for failure; 

mean = “p”; variance = “1 – p” 

Probability

Values 0 1

With likelihood and consequence 

risk cube models; good for 

representing the probability of a 

risk occurring but not for the impact 

on the program

Beta Similar to normal distribution 

but does not allow for negative 

cost or duration, this continuous 

distribution can be symmetric or 

skewed Least

Likely

Most

Probability

Values

To capture outcomes biased toward 

the tail ends of a range; often used 

with engineering data or analogy 

estimates; the shape parameters 

usually cannot be collected from 

interviewees
GAO-09-3SP Chapter 14 169

Distribution Description Shape Typical application

Lognormal A continuous distribution 

positively skewed with a 

limitless upper bound and 

known lower bound; skewed to 

the right to reflect the tendency 

toward higher cost

Probability

Values

To characterize uncertainty 

in nonlinear cost estimating 

relationships; it is important to 

know how to scale the standard 

deviation, which is needed for this 

distribution

Normal Used for outcomes likely to 

occur on either side of the 

average value; symmetric 

and continuous, allowing for 

negative costs and durations. 

In a normal distribution, about 

68% of the values fall within one 

standard deviation of the mean

Least

Likely

Most

Probability

Values

To assess uncertainty with cost 

estimating methods; standard 

deviation or standard error of the 

estimate is used to determine 

dispersion. Since data must be 

symmetrical, it is not as useful 

for defining risk, which is usually 

asymmetrical, but can be useful for 

scaling estimating error

Poisson Peaks early and has a long tail 

compared to other distributions

Least

Likely

Most

Probability

Values

To predict all kinds of outcomes, like 

the number of software defects or 

test failures

Triangular Characterized by three points 

(most likely, pessimistic, and 

optimistic values), can be 

skewed or symmetric and is 

easy to understand because it 

is intuitive; one drawback is the 

absoluteness of the end points, 

although this is not a limitation 

in practice since it is used in a 

simulation 

Least

Likely

Most

Probability

Values

To express technical uncertainty, 

because it works for any system 

architecture or design; also used to 

determine schedule uncertainty

Uniform Has no peaks because all values, 

including highest and lowest 

possible values, are equally likely

Probability

Equally likely throughout

Values

With engineering data or analogy 

estimates

Weibull Versatile, can take on the 

characteristics of other 

distributions, based on the value 

of the shape parameter “b”—

e.g., Rayleigh and exponential 

distributions can be derived 

from ita

Probability

Values

In life data and reliability analysis 

because it can mimic other 

distributions and its objective 

relationship to reliability modeling

Source: DOD, NASA, SCEA, and Industry.

a

The Rayleigh and exponential distributions are a class of continuous probability distribution. 

The triangular, lognormal, beta, uniform, and normal distributions in table 24 are the most common 

distributions that cost estimators may use to perform an uncertainty analysis. They are generally sufficient, 

given the quality of the information derived from interviews and the granularity of the results. However, 
170 Chapter 14 GAO-09-3SP

many other types of distributions are discussed in myriad literature sources and are available through a 

variety of statistical tools.

The point to remember is that the shape of the distribution is determined by the characteristics of the 

risks they represent. If they are applied to WBS elements, they may combine the impact of several risks, 

so it may take some thought to determine the most appropriate distribution to use. For a CER, it is a 

best practice to use prediction interval statistical analysis to determine the bounds of the probability 

distribution because it is an objective method for determining variability. The prediction interval captures 

the error around a regression estimate and results in a wider variance for the CER.

When there is no objective way to pick the distribution bounds, a cost estimator will resort to interviewing 

several people—especially experienced personnel both within and outside the program—about what the 

distribution bounds should be. Promising anonymity to the interviewees may help secure their unbiased 

thoughts. Separating the risk analysis function organizationally from the program and program manager 

often provides the needed independence to withstand political and other pressures for biased results.

One way to avoid the potential for experts to be success oriented when choosing the upper and lower 

extremes of the distribution is to look for historical data that back up the distribution range. If historical 

data are not available, it may be necessary to adjust the tails to account for the fact that being overly 

optimistic usually results in programs costing more and taking longer than planned. Thus, it is necessary 

to skew the tails to account for this possibility in order to properly represent the overall risk. Organizations 

should, as a best practice, examine and publish default distribution bounds that cost estimators can use 

when the data cannot be obtained objectively. 

Once all cost element risks have been identified in step 1 and distributions have been chosen to model 

them in step 2, correlation between the cost elements must be examined in order to fully capture risk, 

especially risk related to level-of-effort cost elements.

Step 3: Account for Correlation between Cost Elements

Because different WBS elements’ costs may be affected by the same external factors, some degree of 

correlation exists between them. Correlation identifies the relationship between WBS elements such that 

when one WBS element’s cost is high within its own probability distribution, the other WBS element will 

also show a high cost in its own probability distribution. Thus, correlated cost elements should rise and 

fall together. Without correlating the two elements, inconsistent scenarios where one is high and the other 

is low could occur during the simulation, causing erroneous results. Therefore, a change in one WBS 

element’s cost will usually be found with a change in the same direction (if positive correlation) in another 

element’s cost. If this is so for many elements, the cumulative effect tends to increase the range of possible 

costs. Consider the following examples:

■ If a supplier delivers an item late, other scheduled deliveries could be missed, resulting in additional 

cost.

■ If technical performance problems occur, unexpected design changes and unplanned testing may 

result, affecting the final schedule and cost.
GAO-09-3SP Chapter 14 171

■ If concurrence is great between activities, a slip in one activity could have a cascading effect on 

others, resulting in a high degree of schedule and cost uncertainty.

■ If the number of software lines of code depends heavily on the software language and the 

definition of what constitutes a line of code, a change in the counting definition or software 

language will change the number of lines of code affecting both schedule and cost.

As these examples show, many parts of a cost estimate may move together, and when they do, summing 

their costs results in reinforcement in both negative and positive directions. Therefore, mitigating a risk 

that affects two or more WBS cost elements can reduce uncertainty on several cost elements. A case in 

point is the standing army effect, which occurs when a schedule slip in one element results in delays in 

many other cost elements as staff wait to complete their work. As such, interelement correlation must be 

addressed so that the total cost probability distribution properly reflects the risks. 

To properly capture functional correlation, the cost model should be structured with all dependencies 

intact. For instance, if the cost of training is modeled as a factor of hardware cost, then any uncertainty 

in the hardware cost will be positively correlated to the risk in training cost. Thus, when the simulation 

is run, risks fluctuating within main cost element probability distributions will accurately flow down to 

dependent WBS elements. 

One of the advantages of a cost estimating relationship based cost model is the manner in which the 

statistical analysis used to derive the CERs can also be drawn on to identify and, in some cases, quantify 

the correlations between various cost risk elements. It is also important to ensure that uncertain cost 

method inputs (weight, labor rates) are correlated appropriately.

In some cases, however, it may be necessary to inject correlation to “below the line” dependent elements 

to account for correlated risk. These elements are typically level-of-effort support activities, like systems 

engineering and program management. In addition, correlation may have to be injected into the cost 

model to account for effects that the model may not capture. For example, a program risk may be that the 

length of an aircraft wing increases. If that happens, a larger engine than was originally estimated would 

then be required. Because this risk effect is not correlated in the cost model, it must be injected into the 

risk scenario.

Estimators should examine the correlation coefficients from the simulation model to determine the 

amount of correlation that already exists in the cost model. As a rule of thumb, it is better to insert an 

overall nominal correlation of 0.25 than to show no correlation at all. This will prevent the simulation 

from drawing a low value for one element and a high value for another element, causing a cancellation of 

risk when both elements are positively correlated. 

Regardless of which approach is taken, it is important to note that correlation should never be ignored. 

Doing so can significantly affect the cost risk analysis, resulting in a dramatically understated probability 

distribution that can create a false sense of confidence in the resulting estimate. Therefore, highly risky 

programs should show a wide range of possible costs. (More information on correlation and how to 

account for schedule risk affecting the cost estimate is in appendix X.)
172 Chapter 14 GAO-09-3SP

Step 4: Perform Uncertainty Analysis with a Monte Carlo Simulation

The most common technique for combining the individual elements and their distributions is Monte 

Carlo simulation.57 In one approach, the distributions for each cost element are treated as individual 

populations from which random samples are taken. In another approach, each risk is modeled and 

assigned to the WBS elements it affects; in this approach, a risk may affect more than one WBS element’s 

cost, and a WBS element’s cost may be affected by more than one risk. In either case, during the 

simulation a cost model is recalculated thousands of times by repeatedly drawing random values from each 

WBS distribution or distribution of risk factors, so that many, thousands of or nearly all possible outcomes 

are taken into account. The simulation’s output illustrates (1) the likelihood of achieving the program’s 

cost objectives, given the current plan and risks as they are known and quantified; (2) the likelihood of 

other possible outcomes, which can be a way to determine the cost value that has an acceptable probability 

of being exceeded; and (3) by sensitivity, the high-priority risks or WBS elements as a guide to effective 

risk mitigation. 

Not a new concept, Monte Carlo simulation has been a respected method of analyzing risk in engineering 

and science for more than 60 years. Mathematicians working on the Manhattan project used it during 

World War II and this technique was used to determine the value of pi (π) to within 6 decimal points. 

Developed by a mathematician who pondered the probabilities associated with winning a card game of 

solitaire, Monte Carlo simulation is used to approximate the probability outcomes of multiple trials by 

generating random numbers. In determining the uncertainty associated with a program’s point estimate, 

a Monte Carlo simulation randomly generates values for uncertain variables over and over to simulate a 

model. 

Without the aid of simulation, the analyst generally produces a single outcome for the total program cost, 

usually by adding up the individual WBS element cost estimates. This value is not necessarily the most 

likely or average scenario. In fact, without a risk analysis, it is not known how adequate this single-point 

estimate is likely to be for handling the program risks. But after hundreds or thousands of trials, one can 

view the frequency distribution of the results and determine the certainty of any outcome. Performing an 

uncertainty analysis using Monte Carlo simulation quantifies the amount of cost risk within a program. 

Only by quantifying the cost risk can management make informed decisions about risk mitigation 

strategies and provide a benchmark against which to measure progress.

To perform an uncertainty analysis, each WBS element’s risk or risk factor is assigned a specific probability 

distribution of feasible values. In setting up the simulation, any identified causality may be modeled. 

Also, correlations are specified, including identified correlated elements and estimated strength of the 

correlation. These are automatically taken into account by the software during the simulation, where a 

random draw from each distribution is taken and the results are added up. This random drawing among 

distributions is repeated thousands of times with statistical software in order to determine the frequency 

distribution. Since the simulation’s inputs are probability distributions, the outputs are also distributions. 

The result is a distribution of random total program costs based on the overall mean and standard 

deviation. Rather than being normal, the total cost distribution is usually lognormal. This happens 

because the overall cost distribution is derived from the lower-level WBS elements, each of which has 

57Latin hypercube simulation can also be used. This method partitions the “simulation draw area” into equal area segments and 

results in convergence to the “correct” answer with fewer iterations. 
GAO-09-3SP Chapter 14 173

unique distributions. Since many of these underlying distributions tend to be skewed to the right, the 

overall distribution is typically lognormal. This makes sense since most cost estimates tend to overrun 

rather than underrun. This distribution can also be converted to an S curve like the S curves shown in 

figures 16 and 19. 

An advantage of using a Monte Carlo simulation is that both good and bad effects can be modeled, as well 

as any offsets that occur when both happen at the same time. In addition, Monte Carlo simulation not 

only recognizes the uncertainty inherent in the point estimate but also captures the uncertainty with all 

other possible estimates, allowing for a better analysis of alternatives. Using this technique, management 

can base decisions on cost estimate probabilities rather than relying on a single point estimate with no level 

of confidence attached.

Step 5: Identify the Probability Associated with the Point Estimate

After the simulation has been run and causality and correlation have been accounted for, the next step is 

to determine the probability associated with the point estimate. The cumulative probability distribution 

resulting from the Monte Carlo simulation provides the cost estimator and management with riskadjusted estimates and corresponding probabilities. The output of the simulation is useful for determining 

the level of probability in achieving the point estimate, along with a range of possible outcomes bounded 

by minimum and maximum costs. This probability can then be weighed against available funding to 

understand the confidence one can place in the program’s meeting its objectives.

Uncertainty analysis using a Monte Carlo simulation communicates to stakeholders how likely a program 

is to finish at the estimated cost and schedule, how much cost contingency reserve is needed to provide 

the desired degree of certainty that the estimate will be adequate, and the likely risks so that proactive 

responses can be developed.58 It also determines how different two competing alternatives are in terms of 

cost. In addition, estimating future costs with probabilities is better than just relying on a point estimate, 

because informed decisions can be made regarding all possible outcomes.

Because we can never know all the risks until the program is finally complete, the risk analysis and cost 

risk simulation exercise should be conducted periodically through the life of the program. Organizations 

often require such an analysis before major milestone decision points.

Step 6: Recommend Sufficient Contingency Reserves

The main purpose of risk and uncertainty analysis is to ensure that a program’s cost, schedule, and 

performance goals can be met. The analysis also communicates to decision makers the specific risks that 

contribute to a program’s cost estimate uncertainty. Without this knowledge, a program’s estimated cost 

could be understated and subject to underfunding and cost overruns, putting it at risk of being reduced 

in scope or requiring additional funding to meet its objectives. Moreover, probability data from an 

uncertainty analysis can result in more equitable distribution of budget in an EVM system, ensuring that 

the most risky cost elements receive adequate budget up front.

58Cost and schedule Monte Carlo simulations tend to be performed separately by different specialists. Cost uncertainty models 

seldom address schedule risk issues. Performing a schedule risk analysis can more adequately address schedule risk issues. (More 

detail is in appendix X.) 
174 Chapter 14 GAO-09-3SP

Using information from the S curve, management can determine the contingency reserves needed to 

reach a specified level of confidence. The difference in cost between the point estimate and the desired 

confidence level determines the required contingency reserve. Because cost distributions tend to be right 

skewed (that is, the tendency is for costs to overrun rather than underrun), the mean of the distribution 

tends to fall somewhere between the 55 percent and 65 percent confidence levels. Therefore, if it is decided 

to fund a program at the 50 percent confidence level, there is still a chance that the program will need 

additional funding because the expected value is at a higher confidence level. Moreover, extremely risky 

programs will require funding at a level closer to the 65 percent confidence level or higher. Since each 

program is unique and so are its risks, there are no set rules as to what level of contingency is sufficient.

Decision makers have to decide the level of confidence at which to set the budget. Having adequate 

funding is paramount for optimal program execution, since it can take many months to obtain necessary 

funding to address an emergent program issue. Without available risk funding, cost growth is likely. 

We caution that the validity of the results depends on the knowledge, experience, and data regarding a 

program’s risks. When the uncertainty analysis has been poorly executed, management may have a false 

sense of security that all risks have been accounted for and that the analysis was based on sound data. 

When this happens, program decisions will be based on bad information. Thus, it is imperative that the 

cost estimators properly correlate cost elements and consider a broad range of potential program risks 

rather than simply focusing on the risks that most concern the program office or contractor. Furthermore, 

to ensure that best practices have been followed and to prevent errors such as not properly accounting for 

correlation between cost elements, it is a best practice to vet the uncertainty analysis through a core group 

of experts to ensure that results are robust and valid. 

In addition, to ensure that accurate information is available for performing uncertainty analysis, the 

estimate should be continually updated with actual costs and any variances recorded. This will enable 

organizations to identify areas where estimating was difficult or sources of risk were not considered. 

Doing so will guard against presenting misleading results to management and will result in continuous 

improvements in the uncertainty analysis process.

A program’s early phases entail a lot of uncertainty, and the amount of contingency funding required may 

exceed acceptable levels. Management may gain insight from the uncertainty analysis by acting to reduce 

risk to keep the program affordable. It may also examine different levels of contingency reserve funds to 

understand what level of confidence the program can afford. Most importantly, management needs to 

understand that any uncertainty analysis or risk mitigation is only as good as the comprehensiveness of 

risks and uncertainties identified. Unknown risks could still cause problems, and these are difficult, if not 

impossible, to quantify. 

Step 7: Allocate, Phase, and Convert a Risk-Adjusted Cost Estimate to Then-Year 

Dollars and Identify High-Risk Elements

Uncertainty is calculated on the total cost estimate results, not year by year. Therefore, since a budget is 

requested in then-year dollars, it is necessary to convert the cost estimate into then-year dollars by phasing 

the WBS element costs over time. Because WBS element results at a specific confidence level will not 

sum to the parent levels, it will be necessary to pick the level in the WBS from which risk dollars are to be 

managed. The difference between the point estimate and the risk result at the selected confidence level is 

the amount of contingency reserve to be set aside for mitigating risks in lower WBS level elements. 
GAO-09-3SP Chapter 14 175

Once the amount of contingency reserve has been identified, reserves need to be identified and set aside 

for the WBS elements that harbor the most risks so that funding will be available to mitigate risks quickly. 

To identify which WBS elements may need contingency reserve, results from the uncertainty analysis 

are used to prioritize risks, based on probability and impact as they affected the cost estimate during the 

simulation. Knowing which risks are important will guide the allocation of contingency reserve. 

Risk Management

Risk and uncertainty analysis is just the beginning of the overall risk management process. Risk 

management is a structured and efficient process for identifying risks, assessing their effect, and 

developing ways to reduce or eliminate risk. It is a continuous process that constantly monitors a program’s 

health. In this process, program management develops risk handling plans and continually tracks them to 

assess the status of program risk mitigation activities and abatement plans. In addition, risk management 

anticipates what can go wrong before it becomes necessary to react to a problem that has already occurred. 

Identifying and measuring risk by evaluating the likelihood and consequences of an undesirable event are 

key steps in risk management. The risk management process should address five steps:

1. identify risks,

2. analyze risks (that is, assess their severity and prioritize them),

3. plan for risk mitigation,

4. implement a risk mitigation plan, and

5. track risks.

Steps 1 and 2 should have already been taken during the risk and uncertainty analysis. Steps 3–5 should 

begin before the actual work starts and continue throughout the life of the program. Over time, some 

risks will be realized, others will be retired, and some will be discovered: Risk management never ends. 

Establishing a baseline of risk expectations early provides a reference from which actual cost risk can be 

measured. The baseline helps program managers track and defend the need to apply risk reserves to resolve 

problems.

Integrating risk management with a program’s systems engineering and program management process 

permits enhanced root cause analysis and consequence management, and it ensures that risks are handled 

at the appropriate management level. Furthermore, successful risk mitigation requires communication and 

coordination between government and the contractor to identify and address risks. A common database 

of risks available to both is a valuable tool for mitigating risk so that performance and cost are monitored 

continually. 

Regular event-driven reviews are also helpful in defining a program that meets users’ needs while 

minimizing risk. Similarly, relying on technology demonstrations, modeling and simulation, and 

prototyping can be effective in containing risk. When risks materialize, risk management should provide a 

structure for identifying and analyzing root causes. 

Effective risk management depends on identifying and analyzing risk early, while there is still time to 

make corrections. By developing a watch list of risk issues that may cause future problems, management 

can monitor and detect potential risks once the program is under contract. Programs that have an EVM 

system can provide early warning of emerging risk items and worsening performance trends, allowing for 

implementing corrections quickly. 
176 Chapter 14 GAO-09-3SP

EVM systems also require the contractor to provide an estimate at completion and written corrective 

action plans for any variances that can be assessed for realism, using risk management data and techniques. 

Moreover, during IBR, the joint government and contractor team evaluates program risks associated with 

work definition, schedule, and the adequacy of budgets. This review enhances mutual understanding of 

risks facing the program and lays the foundation for tracking them in the EVM system. It also establishes a 

realistic baseline from which to measure performance and identify risk early.

Risk management is continual because risks change significantly during a program’s life. A risk event’s 

likelihood and consequences may change as the program matures and more information becomes known. 

Program management needs always to reevaluate the risk watch list to keep it current and examine new root 

causes. Successful risk management requires timely reporting to alert management to risks that are surfacing, 

so that mitigation action can be approved quickly. Having an active risk management process in place is a 

best practice: When it is implemented correctly, it minimizes risks and maximizes a program’s chances of 

being delivered on time, within budget, and with the promised functionality.

11. Best Practices Checklist: Cost Risk and Uncertainty

 A risk and uncertainty analysis quantified the imperfectly understood risks that are 

in the program and identified the effects of changing key cost driver assumptions 

and factors.

3 Management was given a range of possible costs and the level of certainty in 

achieving the point estimate.

3 A risk adjusted estimate that reflects the program’s risks was determined.

3 A cumulative probability density function, an S curve, mapped various cost 

estimates to a certain probability level and defensible contingency reserves 

were developed.

3 Periodic risk and uncertainty analysis was conducted to improve estimate 

uncertainty.

 The following steps were taken in performing an uncertainty analysis:

3 Program cost drivers and associated risks were determined, including those 

related to changing requirements, cost estimating errors, business or economic 

uncertainty, and technology, schedule, program, and software uncertainty.

ù All risks were documented for source, data quality and availability, and 

probability and consequence.

ù Risks were collected from staff within and outside the program to counter 

optimism.

ù Uncertainty was determined by cost growth factor, expert opinion (adjusted 

to consider a wider range of risks), statistics and Monte Carlo simulation, 

technology readiness levels, software engineering maturity models and risk 

evaluation methods, schedule risk analysis, risk cube (P-I matrix) method, or 

risk scoring.

3 A probability distribution modeled each cost element’s uncertainty based on 

data availability, reliability, and variability.

ù A range of values and their respective probabilities were determined either 

based on statistics or expressed as 3-point estimates (best case, most likely, 

and worst case), and rationale for choosing which method was discussed.

ù Documentation of the rationale for choosing the probability distributions 

should be provided.
GAO-09-3SP Chapter 14 177

ù Probability distribution reflects the risk shape and the tails of the 

distribution reflect the best and worst case spread as well as any skewness. 

Distribution bounds were adjusted to account for stakeholder bias using 

organization default values when data specific to the program are not 

available. 

ù If the risk driver approach is used, the data collected, including probability 

of occurrence and impact, were applied to the risks themselves.

ù Prediction interval statistical analysis was used for CER distribution bounds. 

3 The correlation between cost elements was accounted for to capture risk.

ù The correlation ensures that related cost elements move together during 

the simulation, resulting in reinforcement of the risks.

ù Cost estimators examined the amount of correlation already existing in 

the model. If no correlation is present, an insertion of 0.25 correlation was 

added.

3 A Monte Carlo simulation model was used to develop a distribution of total 

possible costs and an S curve showing alternative cost estimate probabilities.

ù High-priority risks were examined and identified for risk mitigation.

ù Strength of correlated cost elements were examined and additional 

correlation added if necessary to account for risk.

3 The probability associated with the point estimate was identified.

3 Contingency reserves were recommended for achieving the desired confidence 

level.

ù The mean of the distribution tends to fall around the 55%–65% confidence 

level because the total cost distribution follows a lognormal trend (i.e., 

tendency to overrun rather than underrun costs).

ù Budgeting to at least the mean of the distribution or higher is necessary to 

guard against potential risk.

ù The cost risk and uncertainty results were vetted through a core group of 

experts to ensure that the proper steps were followed.

ù The estimate is continually updated with actual costs and any variances 

recorded to identify areas where estimating was difficult or sources of risks 

were not considered.

3 The risk-adjusted cost estimate was allocated, phased, and converted to thenyear dollars for budgeting, and high-risk elements were identified to mitigate 

risks.

ù Results from the uncertainty analysis were used to prioritize risks based on 

probability and impacts as they affected the cost estimate.

 A risk management plan was implemented jointly with the contractor to identify 

and analyze risk, plan for risk mitigation, and continually track risk.

3 A risk database watch list was developed, and a contractor’s EVM system 

was used for root cause analysis of cost and schedule variances, monitoring 

worsening trends, and providing early risk warning. 

3 Event-driven reviews, technology demonstrations, modeling and simulation, 

and risk-mitigation prototyping were implemented.
178 Chapter 14 GAO-09-3SP
GAO-09-3SP Chapter 15 179

Validating the Estimate

Chapter 15

It is important that cost estimators and organizations independent of the program office validate that 

all cost elements are credible and can be justified by acceptable estimating methods, adequate data, and 

detailed documentation. This crucial step ensures that a high-quality cost estimate is developed, presented, 

and defended to management. This process verifies that the cost estimate adequately reflects the program 

baseline and provides a reasonable estimate of how much it will cost to accomplish all tasks. It also 

confirms that the program cost estimate is traceable and accurate and reflects realistic assumptions. 

Validating the point estimate is considered a best practice. One reason for this is that independent 

cost estimators typically rely on historical data and therefore tend to estimate more realistic program 

schedules and costs for state-of-the-art technologies. Moreover, independent cost estimators are less 

likely to automatically accept unproven assumptions associated with anticipated savings. That is, they 

bring more objectivity to their analyses, resulting in estimates that are less optimistic and higher in 

cost. An independent view provides a reality check of the point estimate and helps reduce the odds that 

management will invest in an unrealistic program that is bound to fail. 

The Cost Estimating Community’s Best Practices for Validating Estimates

OMB’s Circular No. A-94 and best practices established by professional cost analysis organizations, such 

as SCEA, identify four characteristics of a high-quality, reliable cost estimate.59 It is well-documented, 

comprehensive, accurate, and credible. 

By well documented is meant that an estimate is thoroughly documented, including source data and 

significance, clearly detailed calculations and results, and explanations of why particular methods and 

references were chosen. Data can be traced to their source documents.

An estimate is comprehensive if it has enough detail to ensure that cost elements are neither omitted 

nor double counted. All cost-influencing ground rules and assumptions are detailed in the estimate’s 

documentation.

An estimate that is accurate is unbiased, the work is not overly conservative or overly optimistic, and is based 

on an assessment of most likely costs. Few, if any, mathematical mistakes are present those that are are minor.

As for credibility, any limitations of the analysis because of uncertainty or bias surrounding data or 

assumptions are discussed. Major assumptions are varied, and other outcomes are recomputed to 

determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis is 

59For the OMB guidelines, see Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Circular No. A-94 

(Washington, D.C.: Oct. 29, 1992), and Director, OMB, “2009 Discount Rates for OMB Circular No. A-94,” memorandum for 

the heads of departments and agencies, Executive Office of the President, OMB, Washington, D.C., Dec. 12, 2008. 
180 Chapter 15 GAO-09-3SP

performed to determine the level of risk associated with the estimate. The estimate’s results are crosschecked, and an independent cost estimate (ICE) conducted by a group outside the acquiring organization 

is developed to determine whether other estimating methods produce similar results.

Table 25 shows how the 12 steps of a high-quality cost estimating process, described in table 2, can be 

mapped to these four characteristics of a high-quality, reliable cost estimate.

Table 25: The Twelve Steps of High-Quality Cost Estimating, Mapped to the Characteristics of a HighQuality Cost Estimate

Cost estimate characteristic Cost estimating step

Well documented

The estimate is thoroughly documented, including source data and 

significance, clearly detailed calculations and results, and explanations 

for choosing a particular method or reference

■ Data are traced back to the source documentation 

■ Includes a technical baseline description 

■ Documents all steps in developing the estimate so that a cost analyst 

unfamiliar with the program can recreate it quickly with the same 

result

■ Documents all data sources for how the data were normalized 

■ Describes in detail the estimating methodology and rationale used to 

derive each WBS element’s cost 

1. Define the estimate’s purpose 

3. Define the program 

5. Identify ground rules and 

assumptions 

6. Obtain the data

10.Document the estimate

11. Present the estimate to 

management

Comprehensive

The estimate’s level of detail ensures that cost elements are neither 

omitted nor double counted

■ Details all cost-influencing ground rules and assumptions 

■ Defines the WBS and describes each element in a WBS dictionary 

■ A major automated information system program may have only a 

cost element structure

2. Develop the estimating plan 

4. Determine the estimating 

approach

Accurate

The estimate is unbiased, not overly conservative or overly optimistic, 

and based on an assessment of most likely costs 

■ It has few, if any, mathematical mistakes; its mistakes are minor

■ It has been validated for errors like double counting and omitted 

costs

■ Cost drivers have been cross-checked to see if results are similar

■ It is timely

■ It is updated to reflect changes in technical or program assumptions 

and new phases or milestones

■ Estimates are replaced with EVM EAC and the independent EAC from 

the integrated EVM system

7. Develop the point estimate and 

compare it to an independent 

cost estimate

12.Update the estimate to reflect 

actual costs and changes

Credible

Discusses any limitations of the analysis from uncertainty or biases 

surrounding data or assumptions 

■ Major assumptions are varied and other outcomes recomputed to 

determine their sensitivity to changes in assumptions

■ Risk and uncertainty analysis is performed to determine the level of 

risk associated with the estimate

■ An independent cost estimate is developed to determine if other 

estimating methods produce similar results

7. Develop the point estimate and 

compare it to an independent 

cost estimate

8. Conduct sensitivity analysis

9. Conduct risk and uncertainty 

analysis

Source: GAO.
GAO-09-3SP Chapter 15 181

It is important that cost estimates be validated, because lessons learned have shown that cost estimates 

tend to be deficient in this area (see case study 41). 

Case Study 41: Validating the Estimate, from Chemical Demilitarization, 

GAO-07-240R

GAO reviewed and evaluated the cost analyses that the U.S. Army used to prepare its 

cost-benefit report on the DuPont plan of treatment and disposal options for the VX 

nerve agent stockpile at the Newport, Indiana, depot. GAO also interviewed Army and 

contractor officials on the data and assumptions they had used to prepare their analyses. 

To determine the accuracy of the underlying data, GAO independently calculated values 

based on provided assumptions to compare with values in the supporting spreadsheets. 

GAO compared values from the supporting spreadsheets with summary data in the 

supporting posttreatment estimate report that the Shaw Environmental Group had 

prepared, Shaw being the contractor that helped perform the analysis for the U.S. Army 

Chemical Materials Agency report. 

GAO found, based on OMB criteria and criteria approved by the cost estimating 

community, that the underlying cost estimates in the Army’s report were not reliable and 

that the effect of this on the Army’s finding that the DuPont plan had “significant cost 

savings over the three considered alternatives” was uncertain. GAO’s finding of unreliable 

cost estimates included (1) the quantity and magnitude of errors, (2) quality control 

weaknesses, (3) questionable or inadequate supporting source data and documentation, 

and (4) the undetermined sensitivity of key assumptions. Neither the Army nor the 

contractor had a system for cross-checking costs, underlying assumptions, or technical 

parameters that went into the estimates. 

Moreover, GAO determined that the results from the Army’s program risk analysis were 

unreliable because they had been generated from previously discussed, unreliable 

cost estimates and because the Army attributed no risk to potential permit, legal, or 

other challenges to the DuPont plan. It was unclear whether the program risks of other 

alternatives were understated or overstated. 

Overall, GAO could not determine the cumulative effect of these problems on the outcome 

or results of the Army’s analysis, largely because GAO had no confidence in much of the 

supporting data, given these problems. Without reliable underlying cost estimates, the 

Army, the Congress, and the public could not have confidence that the most cost-effective 

solution had been selected. 

GAO’s recommendations were that the Army conduct its cost-benefit analysis again, 

using best practices, so that its data and conclusions would be comprehensive, traceable, 

accurate, and credible; that it correct any technical and mathematical errors in the cost 

estimate; that it establish quality control and independent review processes to check data 

sources, calculations, and assumptions; and that it perform a sensitivity analysis of key 

assumptions.

GAO, Chemical Demilitarization: Actions Needed to Improve the Reliability of 

the Army’s Cost Comparison Analysis for Treatment and Disposal Options for 

Newport’s VX Hydrolysate, GAO-07-240R (Washington, D.C.: Jan. 6, 2007).

Too often, we have reported that program cost estimates are unrealistic and that, as a result, they 

cost more than originally promised. One way to avoid this predicament is to ensure that program 

cost estimates are both internally and externally validated—that is, that they are comprehensive, 
182 Chapter 15 GAO-09-3SP

well documented, accurate, and credible. This increases the confidence that an estimate is reasonable and 

as accurate as possible. A detailed review of these characteristics follows.

1. Determine That the Estimate Is Well Documented

Cost estimates are considered valid if they are well documented to the point at which they can be easily 

repeated or updated and can be traced to original sources through auditing. Rigorous documentation 

also increases an estimate’s credibility and helps support an organization’s decision making. The 

documentation should explicitly identify the primary methods, calculations, results, rationales or 

assumptions, and sources of the data used to generate each cost element. 

Cost estimate documentation should be detailed enough to provide an accurate assessment of the 

cost estimate’s quality. For example, it should identify the data sources, justify all assumptions, and 

describe each estimating method (including any cost estimating relationships) for every WBS cost 

element. Further, schedule milestones and deliverables should be traceable and consistent with the cost 

estimate documentation. Finally, estimating methods used to develop each WBS cost element should be 

thoroughly documented so that their derivation can be traced to all sources, allowing for the estimate to 

be easily replicated and updated.

2. Determine That the Estimate Is Comprehensive

Analysts should make sure that the cost estimate is complete and accounts for all possible costs. They 

should confirm its completeness, its consistency, and the realism of its information to ensure that all 

pertinent costs are included. Comprehensive cost estimates completely define the program, reflect the 

current schedule, and are technically reasonable. In addition, cost estimates should be structured in 

sufficient detail to ensure that cost elements are neither omitted nor double-counted. For example, if it is 

assumed that software will be reused, the estimate should account for all associated costs, such as interface 

design, modification, integration, testing, and documentation.

To determine whether an estimate is comprehensive, an objective review must be performed to certify that 

the estimate’s criteria and requirements have been met, since they create the estimate’s framework. This 

step also infuses quality assurance practices into the cost estimate. In this effort, the reviewer checks that 

the estimate captures the complete technical scope of the work to be performed, using a logical WBS that 

accounts for all performance criteria and requirements. In addition, the reviewer must determine that all 

assumptions and exclusions the estimate is based on are clearly identified, explained, and reasonable.

3. Determine That the Estimate Is Accurate

Estimates are accurate when they are not overly conservative or too optimistic, based on an assessment 

of most likely costs, adjusted properly for inflation, and contain few, if any, minor mistakes. In addition, 

when schedules or other assumptions change, cost estimates should be revised to reflect their current 

status.

Validating that a cost estimate is accurate requires thoroughly understanding and investigating how 

the cost model was constructed. For example, all WBS cost estimates should be checked to verify that 

calculations are accurate and account for all costs, including indirect costs. Moreover, proper escalation 

factors should be used to inflate costs so that they are expressed consistently and accurately. Finally, 

rechecking spreadsheet formulas and data input is imperative to validate cost model accuracy.
GAO-09-3SP Chapter 15 183

Besides these basic checks for accuracy, the estimating technique used for each cost element should be 

reviewed. Depending on the analytical method chosen, several questions should be answered to ensure 

accuracy. Table 26 outlines typical questions associated with various estimating techniques.

Table 26: Questions for Checking the Accuracy of Estimating Techniques

Technique Question

Analogy ■ What heritage programs and scaling factors were used to create the analogy?

■ Are the analogous data from reliable sources? 

■ Did technical experts validate the scaling factor?

■ Can any unusual requirements invalidate the analogy?

■ Are the parameters used to develop an analogous factor similar to the program being 

estimated?

■ How were adjustments made to account for differences between existing and new 

systems? Were they logical, credible, and acceptable? 

Data collection ■ How old are the data? Are they still relevant to the new program?

■ Is there enough knowledge about the data source to determine if it can be used to 

estimate accurate costs for the new program?

■ Has a data scatter plot been developed to determine whether any outliers, 

relationships, and trends exist?

■ Were descriptive statistics generated to describe the data, including the historical 

average, mean, standard deviation, and coefficient of variation?

■ If data outliers were removed, did the data fall outside three standard deviations?

■ Were comparisons made to historical data to show they were an anomaly?

■ Were the data properly normalized so that comparisons and projections are valid?

■ Were the cost data adjusted for inflation so that they could be described in like terms?

Engineering build-up ■ Was each WBS cost element defined in enough detail to use this method correctly?

■ Are data adequate to accurately estimate the cost of each WBS element?

■ Did experienced experts help determine a reasonable cost estimate?

■ Was the estimate based on specific quantities that would be ordered at one time, 

allowing for quantity discounts?

■ Did the estimate account for contractor material handling overhead?

■ Is there a definitive understanding of each WBS cost element’s composition?

■ Were labor rates based on auditable sources? Did they include all applicable 

overhead, general and administrative costs, and fees? Were they consistent with 

industry standards?

■ Is a detailed and accurate materials and parts list available?

Expert opinion ■ Do quantitative historical data back up the expert opinion?

■ How did the estimate account for the possibility that bias influenced the results?

Extrapolate from 

actuals (averages, 

learning curves, 

estimates at 

completion)

■ Were cost reports used for extrapolation validated as accurate?

■ Was the cost element at least 25% complete before using its data as an extrapolation?

■ Were functional experts consulted to validate the reported percentage as complete?

■ Were contractors interviewed to ensure the cost data’s validity?

■ Were recurring and nonrecurring costs separated to avoid double counting?

■ How were first unit costs of the learning curve determined? What historical data were 

used to determine the learning curve slope?

■ Were recurring and nonrecurring costs separated when the learning curve was 

developed?

■ How were partial units treated in the learning curve equation?

■ Were production rate effects considered? How were production break effects 

determined?
184 Chapter 15 GAO-09-3SP

Technique Question

Parametric ■ Was a valid statistical relationship, or CER, between historical costs and program, 

physical, and performance characteristics established?

■ How logical is the relationship between key cost drivers and cost?

■ Was the CER used to develop the estimate validated and accepted?

■ How old are the data in the CER database? Are they still relevant for the program 

being estimated?

■ Do the independent variables for the program fall within the CER data range?

■ What is the level of variation in the CER? How well does the CER explain the 

variation (R2) and how much of the variation does the model not explain? 

■ Do any outliers affect the overall fit?

■ How significant is the relationship between cost and its independent variables?

■ How well does the CER predict costs?

Software estimating ■ Was the software estimate broken into unique categories: new development, reuse, 

commercial off-the-shelf, modified code, glue code, integration?

■ What input parameters—programmer skills, applications experience, development 

language, environment, process—were used for commercial software cost models, 

and how were they justified?

■ How was the software effort sized? Was the sizing method reasonable?

■ How were productivity factors determined?

■ How were labor hours converted to cost? How many productive hours were 

assumed in each day?

■ How were savings from autogenerated code and commercial off-the-shelf software 

estimated? Are the savings reasonable?

■ What were the assumptions behind the amount of code reuse? Were they 

supported?

■ How was the integration between the software, commercial software, system, and 

hardware estimated, and what historical data supported the results?

■ Were software license costs based on actual or historical data?

■ Were software maintenance costs adequately identified and reasonable?

Source: DOD, SCEA, and industry.

CERs and cost models also need to be validated to demonstrate that they can predict costs within an 

acceptable range of accuracy. To do this, data from historical programs similar to the new program should 

be collected to determine whether the CER selected is a reliable predictor of costs. In this review, technical 

parameters for the historical programs should be examined to determine whether they are similar to the 

program being estimated. For the CER to be accurate, the new and historical programs should have 

similar functions, objectives, and program factors, like acquisition strategy, or results could be misleading. 

Equally important, CERs should be developed with established and enforced policies and procedures that 

require staff to have proper experience and training to ensure the model’s continued integrity.

Before a parametric model is used to develop an estimate, the model should be calibrated and validated 

to ensure that it is based on current, accurate, and complete data and is therefore a good predictor of cost. 

Like a CER, a parametric model is validated by determining that its users have enough experience and 

training and that formal estimating system policies and procedures have been established. The procedures 

focus on the model’s background and history, identifying key cost drivers and recommending steps for 

calibrating and developing the estimate. To stay current, parametric models should be continually updated 

and calibrated. 
GAO-09-3SP Chapter 15 185

Validation with calibration gives confidence that the model is a reliable estimating technique. To evaluate 

a model’s ability to predict costs, a variety of assessment tests can be performed. One is to compare 

calibrated values with independent data that were not included in the model’s calibration. Comparing 

the model’s results to the independent test data’s “known value” provides a useful benchmark for how 

accurately the model can predict costs. An alternative is to use the model to prepare an estimate and then 

compare its result with an independent estimate based on another estimating technique.

The accuracy of both CERs and parametric models can be verified with regression statistics, which 

measure the accuracy and goodness of fit, such as the coefficient of determination (R2). CERs with an 

R2 equal to 1.0 would indicate that the CER predicts the sample data perfectly. While this is hardly ever 

the case, an R2 close to 1.0 is more accurate than an R2 that is less than 0.70, meaning 30 percent of the 

variation is unexplained.

4. Determine That the Estimate Is Credible

Credible cost estimates clearly identify limitations because of uncertainty or bias surrounding the data 

or assumptions. Major assumptions should be varied and other outcomes recomputed to determine how 

sensitive outcomes are to changes in the assumptions. In addition, a risk and uncertainty analysis should 

be performed to determine the level of risk associated with the estimate. Finally, the results of the estimate 

should be cross-checked and an ICE performed to determine whether alternative estimate views produce 

similar results.

To determine an estimate’s credibility, key cost elements should be tested for sensitivity, and other cost 

estimating techniques should be used to cross-check the reasonableness of GR&As. It is also important to 

determine how sensitive the final results are to changes in key assumptions and parameters. A sensitivity 

analysis identifies key elements that drive cost and permits what-if analysis, often used to develop cost 

ranges and risk reserves. This enables management to know the potential for cost growth and the reasons 

behind it.

Along with a sensitivity analysis, a risk and uncertainty analysis adds to the credibility of the cost 

estimate, because it identifies the level of confidence associated with achieving the cost estimate. Risk and 

uncertainty analysis produces more realistic results, because it assesses the variability in the cost estimate 

from such effects as schedules slipping, missions changing, and proposed solutions not meeting users’ 

needs. An uncertainty analysis gives decision makers perspective on the potential variability of the estimate 

should facts, circumstances, and assumptions change. By examining the effects of varying the estimate’s 

elements, a degree of uncertainty about the estimate can be expressed with a range of potential costs that is 

qualified by a factor of confidence.

Another way to reinforce the credibility of the cost estimate is to see whether applying a different method 

produces similar results. In addition, industry rules of thumb can constitute a sanity check. The main 

purpose of cross-checking is to determine whether alternative methods produce similar results. If so, then 

confidence in the estimate increases, leading to greater credibility. If not, then the cost estimator should 

examine and explain the reason for the difference and determine whether it is acceptable.
186 Chapter 15 GAO-09-3SP

An ICE is considered one of the best and most reliable validation methods. ICEs are typically performed 

by organizations higher in the decision-making process than the office performing the baseline estimate. 

They provide an independent view of expected program costs that tests the program office’s estimate for 

reasonableness. Therefore, ICEs can provide decision makers with additional insight into a program’s 

potential costs—in part, because they frequently use different methods and are less burdened with 

organizational bias. Moreover, ICEs tend to incorporate adequate risk and, therefore, tend to be more 

conservative by forecasting higher costs than the program office.

The ICE is usually developed from the same technical baseline description the program office used so 

that the estimates are comparable. An ICE’s major benefit is that it provides an objective and unbiased 

assessment of whether the program estimate can be achieved, reducing the risk that the program will 

proceed underfunded. It also can be used as a benchmark to assess the reasonableness of a contractor’s 

proposed costs, improving management’s ability to make sound investment decisions, and accurately assess 

the contractor’s performance. 

In most cases, the ICE team does not have insight into daily program events, so it is usually forced to 

estimate at a higher level or use analogous estimating techniques. It is, in fact, expected that the ICE team 

will use different estimating techniques and, where possible, data sources from those used to develop the 

baseline estimate. It is important for the ICE team and the program’s cost estimate team to reconcile the 

two estimates. 

Two issues with ICEs are the degree of independence and the depth of the analysis. Degree of 

independence depends on how far removed the estimator is from the program office. The greater the 

independence, the more detached and disinterested the cost estimator is in the program’s success. The 

basic test for independence, therefore, is whether the cost estimator can be influenced by the program 

office. 

Thus, independence is determined by the position of the cost estimator in relation to the program office 

and whether there is a common superior between the two. For example, if an independent cost estimator 

is hired by the program office, the estimator may be susceptible to success-oriented bias. When this 

happens, the ICE can end up too optimistic. 

History has shown a clear pattern of higher cost estimates the further away from the program office that 

the ICE is created. This is because the ICE team is more objective and less prone to accept optimistic 

assumptions. To be of value, however, an ICE must not only be performed by entities far removed from 

the acquiring program office but must also be accepted by management as a valuable risk reduction 

resource that can be used to minimize unrealistic expectations. The second issue with an ICE is the depth 

of the review. 

While an ICE reveals for decision makers any optimistic assumptions or items that may have been 

overlooked, in some cases management may choose to ignore it because the estimate is too high, as in 

case study 42.
GAO-09-3SP Chapter 15 187

Case Study 42: Independent Cost Estimates, from Space Acquisitions, 

GAO-07-96

In a review of the Advanced Extremely High Frequency (AEHF) satellite program, the 

National Polar-orbiting Operational Environmental Satellite System (NPOESS), and the 

Space Based Infrared System (SBIRS) High program, GAO found examples of program 

decision makers’ not relying on independent cost estimates (ICE). Independent estimates 

had forecast considerably higher costs and lengthier schedules than program office or 

service cost estimates. To establish budgets for their programs, however, the milestone 

decision authorities had used program office estimates, or even lower estimates, rather 

than the independent estimates. 

DOD’s space acquisition policy required that ICEs be prepared outside the acquisition 

chain of command and that program and DOD decision makers consider them at key 

acquisition decision points. The policy did not require, however, that the independent 

estimates be relied on for setting budgets. 

In 2004, AEHF program decision makers relied on the program office cost estimate rather 

than the independent estimate the CAIG had developed to support the production 

decision. The program office had estimated that AEHF would cost about $6 billion; the 

CAIG had estimated $8.7 billion, some $2.7 billion more. 

The program office estimate was based on the assumption that AEHF would have ten 

times more capacity than Milstar, the predecessor satellite, at half the cost and weight. 

The CAIG believed that this assumption was overly optimistic, given that since AEHF 

began in 1999, its weight had more than doubled to obtain the desired increase in data 

rate. 

NPOESS was another example of large differences between program office and 

independent cost estimates. In 2003, government decision makers relied on the 

program office’s $7.2 billion cost estimate rather than the $8.8 billion independent cost 

estimate that the Air Force Cost Analysis Agency (AFCAA) had presented to support the 

development contract award. Program officials and decision makers had preferred the 

more optimistic assumptions and costs of the program office estimate, having viewed 

the independent estimate as too high.

The SBIRS High program office and AFCAA predicted program cost growth as early 

as 1996, when the program began. While the two estimates, in 2006 dollars, were 

close—$5.7 billion by the program office and $5.6 billion by AFCAA—both were much 

more than the contractor’s estimate. Nevertheless, the program office budgeted SBIRS 

High at $3.6 billion, almost $2 billion less than either the program office or AFCAA had 

estimated.

GAO, Space Acquisitions: DOD Needs to Take More Action to Address 

Unrealistic Initial Cost Estimates of Space Systems, GAO-07-96 (Washington, 

D.C.: Nov. 17, 2006).

Table 27 lists eight types of ICE reviews and describes what they entail.
188 Chapter 15 GAO-09-3SP

Table 27: Eight Types of Independent Cost Estimate Reviews

Review Description

Document review It is an inventory of existing documentation to determine whether information 

is missing and an assessment of the available documentation to support the 

estimate 

Independent cost 

assessment

An outside evaluation of a program’s cost estimate that examines its quality and 

accuracy, with emphasis on specific cost and technical risks, it involves the same 

procedures as those of the program estimate but using different methods and 

techniques

Independent cost estimate Conducted by an organization outside the acquisition chain, using the same 

detailed technical information as the program estimate, it is a comparison with 

the program estimate to determine whether it is accurate and realistic

Independent government 

cost estimate 

Analyzing contractors’ prices or cost proposals, it estimates the cost of activities 

outlined in the statement of work; does not include all costs associated with a 

program and can only reflect costs from a contractor’s viewpoint. Assumes that 

all technical challenges can be met as outlined in the proposal, meaning that it 

cannot account for potential risks associated with design problems

Nonadvocate review Performed by experienced but independent internal nonadvocate staff, it 

ascertains the adequacy and accuracy of a program’s estimated budget; 

assesses the validity of program scope, requirements, capabilities, acquisition 

strategy, and estimated life-cycle costs

Parametric estimating 

technique

Usually performed at the summary WBS level, it includes all activities associated 

with a reasonableness review and incorporates cross-checks using parametric 

techniques and factors based on historical data to analyze the estimate’s 

validity

Reasonableness, or 

sufficiency, review

It is a review of all documentation by an independent cost team, meeting with 

staff responsible for developing the program estimate, to analyze whether 

the estimate is sufficient with regard to the validity of cost and schedule 

assumptions and cost estimate methodology rationale and whether it is 

complete

Sampling technique It is an independent estimate of key cost drivers of major WBS elements whose 

sensitivity affects the overall estimate; detailed independent government cost 

estimates developed for these key drivers include vendor quotes and material, 

labor, and subcontractor costs. Other program costs are estimated using the 

program estimate, as long as a reasonableness review has been conducted to 

ensure their validity

Source: DOD, DOE, and NASA.

As the table shows, the most rigorous independent review is an ICE. Other independent cost reviews 

address only a program’s high-value, high-risk, and high-interest elements and simply pass through 

program estimate values for the other costs. While they are useful to management, not all provide the 

objectivity necessary to ensure that the estimate going forward for a decision is valid.

After an ICE or independent review is completed, it is reconciled to the baseline estimate to ensure that 

both estimates are based on the same GR&As. A synopsis of the estimates and their differences is then 

presented to management. Using this information, decision makers use the ICE or independent review to 

validate whether the program estimate is reasonable. 
GAO-09-3SP Chapter 15 189

Since the ICE team is outside the acquisition chain, is not associated with the program, and has nothing 

at stake with regard to program outcome or funding decisions, its estimate is usually considered more 

accurate. Some ICEs are mandated by law, such as those for DOD’s major acquisition programs. 

Nevertheless, the history of myriad DOD programs clearly shows that ICEs are usually higher, and more 

accurate, than baseline estimates. Thus, if a program cost estimate is close to ICE results, one can be more 

confident that it is accurate and more likely to result in funding at a reasonable level.

12. Best Practices Checklist: Validating the Estimate

The cost estimate was validated against four characteristics:

 It is comprehensive, includes all possible costs, ensures that no costs were 

omitted or double-counted, and explains and documents key assumptions. 

3 It completely defines the program, reflects the current schedule, and 

contains technically reasonable assumptions.

3 It captures the complete technical scope of the work to be performed, 

using a logical WBS that accounts for all performance criteria and 

requirements.

 It was documented so well that it can easily be repeated or updated and 

traced to original sources by auditing.

3 Supporting documentation identifies data sources, justifies all 

assumptions, and describes all estimating methods (including 

relationships) for all WBS elements. 

3 Schedule milestones and deliverables can be traced and are consistent 

with the documentation. 

 It is accurate, not too conservative or too optimistic; is based on an 

assessment of most likely costs, adjusted properly for inflation; and contains 

few minor mistakes. 

3 WBS estimates were checked to verify that calculations were accurate 

and accounted for all costs and that proper escalation factors were used 

to inflate costs so they were expressed consistently and accurately.

3 Questions associated with estimating techniques were answered to 

determine the estimate’s accuracy.

3 CERs and parametric cost models were validated to ensure that they 

were good predictors of costs, their data were current and applied 

to the program, the relationships between technical parameters 

were logical and statistically significant, and results were tested with 

independent data.

 Data limitations from uncertainty or bias were identified; results were crosschecked; an ICE was developed to see if results were similar.

3 Major assumptions were varied and other outcomes recomputed to 

determine their sensitivity to changes in the assumptions. 

3 Risk and uncertainty analysis was conducted.
190 Chapter 15 GAO-09-3SP
GAO-09-3SP Chapter 16 191

Documenting the Estimate 

Chapter 16

Well-documented cost estimates are considered a best practice for high-quality cost estimates, for several 

reasons. 

■ First, thorough documentation is essential for validating and defending a cost estimate. That is, a 

well documented estimate can present a convincing argument of an estimate’s validity and can help 

answer decision makers’ and oversight groups’ probing questions. 

■ Second, documenting the estimate in detail, step by step, provides enough information so that 

someone unfamiliar with the program could easily recreate or update it. 

■ Third, good documentation helps with analyzing changes in program costs and contributes to the 

collection of cost and technical data that can be used to support future cost estimates. 

■ Finally, a well-documented cost estimate is essential if an effective independent review is to ensure 

that it is valid and credible. It also supports reconciling differences with an independent cost 

estimate, improving understanding of the cost elements and their differences so that decision 

makers can be better informed.

Documentation provides total recall of the estimate’s detail so that it can be replicated by someone other 

than those who prepared it. It also serves as a reference to support future estimates. Documenting the cost 

estimate makes available a written justification showing how it was developed and aiding in updating it as 

key assumptions change and more information becomes available. 

Estimates should be documented to show all parameters, assumptions, descriptions, methods, and 

calculations used to develop a cost estimate. A best practice is to use both a narrative and cost tables 

to describe the basis for the estimate, with a focus on the methods and calculations used to derive the 

estimate. With this standard approach, the documentation provides a clear understanding of how the cost 

estimate was constructed. Moreover, cost estimate documentation should explain why particular methods 

and data sets were chosen and why these choices are reasonable. It should also reveal the pros and cons of 

each method selected. Finally, there should be enough detail so that the documentation serves as an audit 

trail of backup data, methods, and results, allowing for clear tracking of a program’s costs as it moves 

through its various life-cycle phases.

Estimates that lack documentation are not useful for updates or information sharing and can hinder 

understanding and proper use. Experience shows that poorly documented estimates can cause a program’s 

credibility to suffer because the documentation cannot explain the rationale of the underlying cost 

elements. Case study 43 takes a closer look at the effect a poorly documented cost estimate can have on 

decision making.
192 Chapter 16 GAO-09-3SP

Case Study 43: Documenting the Estimate, from Telecommunications, 

GAO-07-268

The General Services Administration (GSA) provided GAO with documentation of 

its method, the calculations it used to derive each cost element, its results, and 

many of the previous transition costs for Networx—its program of governmentwide 

telecommunications contracts enabling agencies to make a transition to new, innovative 

services and operations. It had not, however, documented significant assumptions. 

Specifically, GSA had not documented the rationale behind its 76 percent transition traffic 

factor or why it had chosen 30 months for the transition—two key assumptions of its 

analysis. 

GSA also did not provide documentation of certain data sources. Specifically, program 

officials could not provide supporting data for the estimate of an agency transition 

cost valued at $4.7 million. Likewise, GSA could not document the data sources used to 

estimate costs for contractor support in planning and implementing the transition. While 

many costs in its estimate were based on charges incurred during the previous transition, 

GSA officials stated that it was not appropriate to use previous costs as a basis for the 

contractor cost element. 

They explained that unlike the previous transition, GSA would not provide agencies with 

on-site contractor support. They had made this decision because, in part, the 2-1/2 years 

of transition planning that had taken place was expected to result in the agencies’ better 

preparation and ability to facilitate making their transition without direct assistance from 

GSA or its contractors. 

Instead of basing projection of contractor costs on prior charges, program officials told 

GAO that GSA management had decided that contractor support costs should not exceed 

$35 million. Program officials could not provide any data or analysis to support this 

decision.

GSA had not used sound analysis when estimating the funds needed to meet its transitionrelated commitments. These weaknesses could be attributed, in part, to the lack of a cost 

estimation policy that reflected best practices. While GSA’s intentionally conservative 

approach minimized the risk that it would have inadequate funds to pay for committed 

transition costs, it increased the risk that GSA would retain excess funds that could be used 

for other purposes.

GAO, Telecommunications: GSA Has Accumulated Adequate Funding for 

Transition to New Contracts but Needs Cost Estimation Policy, GAO-07-268

(Washington, D.C.: Feb. 23, 2007).

In addition to these requirements, good documentation is necessary to

■ satisfy policy requirements for properly recording the basis of the estimate,

■ convince management and oversight staff that the estimate is credible,

■ provide supporting data that can be used to create a historical database,

■ help answer questions about the approach or data used to create the estimate,

■ record lessons learned and provide a history for tracking why costs changed,

■ define the scope of the analysis,
GAO-09-3SP Chapter 16 193

■ allow for replication so that an analyst unfamiliar with the program can understand the logic 

behind the estimate, and

■ help conduct future cost estimates and train junior analysts.

Elements of Cost Estimate Documentation

Two important criteria should be kept in mind when generating high-quality cost estimate 

documentation. First, it should describe the cost estimating process, data sources, and methods and should 

be clearly detailed to allow anyone to easily reconstruct the estimate. Second, the results of the estimating 

process should be presented in a format that makes it easy to prepare reports and briefings to upper 

management. 

Cost estimators should document all the steps used to develop the estimate. As a best practice, the cost 

estimate documentation should address how the estimate satisfies the 12-step process and corresponding 

best practices identified in this guide for creating high-quality cost estimates. Table 28 describes the 

various sections of proper documentation and what they should include.

Table 28: What Cost Estimate Documentation Includes

Document 

section and cost 

estimating step Description

Cover page and table of contents

2–3 ■ Names the cost estimators, the organization they belong to, etc. 

■ Gives the program’s name, date, and milestones

■ Lists the document’s contents, including supporting appendixes

Executive summary

6–9 ■ Summarizes clearly and concisely the cost estimate results, with enough information 

about cost drivers and high-risk areas for management to make informed decisions 

■ Presents a time-phased display of the LCCE in constant and current year dollars, 

broken out by major WBS cost elements; if an update, tracks the results and discusses 

lessons learned

■ Identifies critical ground rules and assumptions

■ Identifies data sources and methods used to develop major WBS cost elements and 

reasons for each approach

■ Discusses ICE results and differences and explains whether the point estimate can be 

considered reasonable

■ Discusses the results of a sensitivity analysis, the level of uncertainty associated with 

the point estimate, and any contingency reserve recommendations and compares 

them to the funding profile

Introduction

1–5 ■ Gives a program overview: who estimated it, how cost was estimated, the date 

associated with the estimate

■ Addresses the estimate’s purpose, need, and whether it is an initial estimate or update

■ Names the requester, citing tasks assigned and related correspondence (in an 

appendix, if necessary)

■ Gives the estimate’s scope, describing major program phases and their estimated time 

periods, and what the estimate includes and excludes, with reasons

■ Describes GR&As and technical and program assumptions, such as inflation rates
194 Chapter 16 GAO-09-3SP

Document 

section and cost 

estimating step Description

System description

5 ■ Describes the program background and system, with detailed technical and program 

data, major system components, performance parameters, and support requirements

■ Describes contract type, acquisition strategy, and other information in the technical 

baseline description

Program inputs

1–3 ■ Gives the team composition—names, organizational affiliations, who was responsible 

for developing the estimate

■ Details the program schedule, including master schedule and deliverables

■ Describes the acquisition strategy

Estimating method and data by WBS cost element

6, 7, 10 ■ The bulk of the documentation, describing in a logical flow how each WBS cost 

element in the executive summary was estimated; details each cost element enough 

that someone independent of the program recreating the estimate could arrive at 

the same results. Supporting information too detailed for this section is placed in an 

appendix

■ Defines the cost element and describes how it was derived

■ Summarizes costs spread by fiscal year in constant year dollars, matching the current 

program schedule

■ Details the method, sources, models, and calculations for developing the estimate; 

fully documents CERs, including the rationale for the relationship between cost and 

the independent variables, the applicable range for independent variables, and the 

process for validating the CER, including descriptive statistics associated with the 

relationship

■ If cost models were used, documents input and output data and any calibrations to 

the model; the cost model, data input, and results are in an appendix

■ Documents the data in detail with a display of all database information used for 

parametric or analogy-based estimates; describes judgments about parametric 

variables, analogy scaling, or complexity factors and adjustments of the data; 

identifies data limitations and qualifies the data, based on sources (historical data, 

budget estimates), time periods they represent, and adjustments to normalize them or 

account for significant events like production breaks

6, 7, 10 ■ Identifies direct and indirect labor rates, labor hours, material and subcontractor costs, 

overhead rates, learning curves, inflation indexes, and factors, including their basis

■ Shows the calculation of the cost estimate, with a logical link to input data

■ Identifies and discusses significant cost drivers; identifies specialists whose judgments 

were used and their qualifications

■ Discusses the cross-check approach for validating the estimate

■ Discusses the ICE’s results and differences and whether it corroborates the point 

estimate as reasonable

Sensitivity analysis

8 ■ Describes the effect of changing key cost drivers and assumptions independently 

■ Identifies the major cost drivers that should be closely monitored
GAO-09-3SP Chapter 16 195

Document 

section and cost 

estimating step Description

Risk and uncertainty analysis 

9 ■ Discusses sources of risk and uncertainty, including critical assumptions, associated 

with the estimate

■ The effect of uncertainty associated with the point estimate is quantified with 

probability distributions, and the resulting S curve is fully documented; the method 

for quantifying uncertainty is discussed and backed up by supporting data

■ The basis for contingency reserves and how they were calculated is fully documented

Management approval

11 ■ Includes briefings presenting the LCCE to management for approval, explaining the 

technical and program baseline, estimating approach, sensitivity analysis, risk and 

uncertainty analysis, ICE results and reasons for differences, and an affordability 

analysis to identify any funding shortfalls

■ Presents the estimate’s limitations and strengths

■ Includes management approval memorandums, recommendations for change, and 

feedback

Updates reflecting actual costs and changes

12 ■ Reflects changes in technical or program assumptions or new program phases or 

milestones

■ Replaces estimates with actual costs from the EVM system and reports progress on 

meeting cost and schedule estimates

■ Includes results of post mortems and lessons learned, with precise reasons for why 

actual costs or schedules differ from the estimate 

Source: DHS, DOD, DOE, NASA, SCEA, and industry.

While documentation of the cost estimate is typically in the form of a written document, the 

documentation can be completed in other acceptable ways. For example, some organizations rely on 

cost models that automatically develop documentation, while others use detailed MS Excel spreadsheets 

with cell notes and hyperlinks to other documents. The important thing to consider is whether the 

documentation allows someone to trace the data, calculations, modeling assumptions, and rationale back 

to a source document for verification and validation. In addition, it should also address the reconciliation 

with the independent cost estimate so that others can understand areas of risk. 

Other Considerations

Documenting the cost estimate should not be a last-minute effort. If documentation is left untouched 

until the end of the estimating process, it will be much harder to recapture the rationale and judgments 

that formed the cost estimate and will increase the chance of overlooking important information that can 

cause credibility issues. Documentation should be done in parallel with the estimate’s development, so that 

the quality of the data, methods, and rationale are fully justified. More information is preferred over too 

little, since the purpose of documenting the estimate is to allow for recreating it or updating it by someone 

else who knows nothing about the program or estimate. Consequently, documentation should be written 

step by step and should include everything necessary for another analyst to easily and quickly replicate 

the estimate and arrive at the same results. In addition, access to an electronic copy of the cost model 
196 Chapter 16 GAO-09-3SP

supporting the estimate should be available with the documentation so that updates can be performed 

efficiently. Finally, the cost estimate and documentation need to be stored so that authorized personnel 

can easily find it and use it for future estimates.

13. Best Practices Checklist: Documenting the Estimate

 The documentation describes the cost estimating process, data sources, and 

methods step by step so that a cost analyst unfamiliar with the program could 

understand what was done and replicate it.

3 Supporting data are adequate for easily updating the estimate to reflect actual 

costs or program changes and using them for future estimates.

3 The documentation describes the estimate with narrative and cost tables. 

3 It contains an executive summary, introduction, and descriptions of methods, 

with data broken out by WBS cost elements, sensitivity analysis, risk and 

uncertainty analysis, management approval, and updates that reflect actual 

costs and changes.

3 Detail addresses best practices and the 12 steps of high-quality estimates.

3 The documentation is mathematically sensible and logical.

3 It discusses contingency reserves and how they were derived from risk and 

uncertainty analysis and the LCCE funding profile. 

 It includes access to an electronic copy, and both are stored so that authorized 

personnel can easily find and use them for other cost estimates.
GAO-09-3SP Chapter 17 197

Presenting the Estimate to Management 

Chapter 17

A cost estimate is not considered valid until management has approved it. Since many cost estimates are 

developed to support a budget request or make a decision between competing alternatives, it is vital that 

management is briefed on how the estimate was developed, including risks associated with the underlying 

data and methods. Therefore, the cost estimator should prepare a briefing for management with enough 

detail to easily defend the estimate by showing how it is accurate, complete, and high in quality. The 

briefing should present the documented LCCE with an explanation of the program’s technical and 

program baseline.

The briefing should be clear and complete, making it easy for those unfamiliar with the estimate to 

comprehend its level of competence. The briefing should focus on illustrating to management, in a logical 

manner, what the largest cost drivers are. Slides with visuals should be available to answer more probing 

questions. A best practice is to present the briefing in a consistent format to facilitate management’s 

understanding the completeness of the cost estimate, as well as its quality. Moreover, decision makers who 

are familiar with a standard briefing format will be better able to concentrate on the briefing’s contents, 

and on the cost estimate, rather than focusing on the format itself.

The cost estimate briefing should succinctly illustrate key points that center on the main cost drivers and 

the final cost estimate’s outcome. Communicating results simply and clearly engenders management 

confidence in the ground rules, methods, and results and in the process that was followed to develop the 

estimate. The presentation must include program and technical information specific to the program, along 

with displays of budget implications, contractor staffing levels, and industrial base considerations, to name 

a few. These items should be included in the briefing:

■ The title page, briefing date, and the name of the person being briefed.

■ A top-level outline.

■ The estimate’s purpose: why it was developed and what approval is needed.

■ A brief program overview: its physical and performance characteristics and acquisition strategy, 

sufficient to understand its technical foundation and objectives.

■ Estimating ground rules and assumptions. 

■ Life-cycle cost estimate: time-phased in constant-year dollars and tracked to any previous estimate.

■ For each WBS cost element, show the estimating method for cost drivers and high-value items; show 

a breakout of cost elements and their percentage of the total cost estimate to identify key cost drivers.

■ Sensitivity analysis, interpreting results carefully if there is a high degree of sensitivity.

■ Discussion of risk and uncertainty analysis: (1) cost drivers, the magnitude of outside influences, 

contingencies, and the confidence interval surrounding the point estimate and the corresponding 
198 Chapter 17 GAO-09-3SP

S curve showing the range within which the actual estimate should fall; (2) other historic data for 

reality checks; and (3) how uncertainty, bounds, and distributions were defined.

■ Comparison to an independent cost estimate, explaining differences and discussing results.

■ Comparison of the LCCE, expressed in current-year dollars, to the funding profile, including 

contingency reserve based on the risk analysis and any budget shortfall and its effect.

■ Concerns or challenges the audience should be aware of.

■ Conclusions, recommendations, and associated level of confidence in the estimate. 

When briefing management on LCCEs, the presenter should include separate sections for each program 

phase—research and development, procurement, operations and support, disposal—and should provide 

the same type of information as the cost estimate documentation contains. In addition, the briefing 

should present the summary information, main conclusions, and recommendations first, followed by 

detailed explanations of the estimating process. 

This approach allows management to gain confidence in the estimating process and, thus, the estimate 

itself. At the conclusion of the briefing, the cost estimator should ask management whether it accepts 

the cost estimate. Acceptance, along with any feedback from management, should be acted on and 

documented in the cost estimate documentation package.

14. Best Practices Checklist: Presenting the Estimate to Management

 The briefing to management 

3 was simple, clear, and concise enough to convey its level of competence.

3 illustrated the largest cost drivers, presenting them logically, with 

backup charts for responding to more probing questions.

3 was consistent, allowing management to focus on the estimate’s content.

 The briefing contained 

3 A title page, outline, and brief statement of purpose of the estimate. 

3 An overview of the program’s technical foundation and objectives.

3 LCCE results in time-phased constant-year dollars, tracked to previous 

estimates.

3 A discussion of GR&As.

3 The method and process for each WBS cost element, with estimating 

techniques and data sources.

3 The results of sensitivity analysis and cost drivers that were identified.

3 The results of risk and uncertainty analysis with confidence interval, S 

curve analysis, and bounds and distributions.

3 The comparison of the point estimate to an ICE with discussion of 

differences and whether the point estimate was reasonable.

3 An affordability analysis based on funding and contingency reserves.

3 Discussion of any other concerns or challenges

3 Conclusions and recommendations.

 Feedback from the briefing, including management’s acceptance of the 

estimate, was acted on and recorded in the cost estimate documentation.
GAO-09-3SP Chapter 18 199

Managing Program Costs: Planning

Chapter 18

In this chapter, we review the importance of obtaining the best perspective on a program and its inherent 

risks by linking cost estimating and EVM. We describe a best practice for cost estimators and EVM 

analysts: sharing data to update program costs and examining differences between estimated and actual 

costs to present scope changes, risks, and other opportunities to management with sufficient lead time to 

plan for and mitigate their impact. Then we summarize the history and nature of EVM—its concepts, 

tools, and benefits. Finally, we describe EVM as managing program costs through proper planning.

Linking Cost Estimation as the Foundation for EVM Analysis

A credible cost estimate lies at the heart of EVM analysis. Figure 20 depicts how cost estimating supports 

the EVM process. It also lays out the specific flow of activity between key functions such as cost 

estimation, system development oversight, and risk management.

Figure 20: Integrating Cost Estimation, Systems Development Oversight, and Risk Management Systems acquisition

Systems maintenance

Risk

analysis

Life-cycle

cost estimate

Risk

analysis

Performance

measurement

baseline

Acquisition

cost

estimate

Integrated

baseline

review

Acquisition

plan

Work

breakdown

structure

Risk

analysis

Operational/

functional

concept

Surveillance

Risk

analysis

Risk

analysis

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

• Cost

• Schedule

• Technical

Program

management

Earned

value

management

Cost

analysis

Systems

engineering

Source: NDIA.

Presystems acquisition planning

Requirements

definition Analysis of

alternatives

Concept

definition

Work

breakdown

structure
200 Chapter 18 GAO-09-3SP

As the lower left of figure 20 shows, a program’s life cycle begins with planning, where systems 

engineering defines the program’s concept, requirements, and WBS. When these activities are complete, 

the information is passed on to the cost analysis team so that they can develop the program’s LCCE. 

Before a system is acquired, however, a risk analysis examining cost, schedule, and technical impacts 

is performed. The results of the LCCE and risk analysis are presented to executive management for an 

informed decision on whether the program should proceed to systems acquisition. 

If management approves the program for acquisition, then systems engineering and cost analyses continue, 

in conjunction with the development of the program’s EVM performance measurement baseline.60 This 

baseline is necessary for defining the time-phased budget plan from which actual program performance 

is measured. After the performance measurement baseline has been established, the program manager 

and supplier participate in an IBR to ensure mutual understanding of all the risks. This review also 

validates that the program baseline is adequate and realistically portrays all authorized work according to 

the schedule. When appropriate, an IBR may begin before contract award to mitigate risk. The Federal 

Acquisition Regulation (FAR) provides for a pre-award IBR as an option, in accordance with agency 

procedures.61

Preparing for and managing program risk occurs during both planning and system acquisition. In 

planning, a detailed WBS is developed that completely defines the program and encompasses all 

risks from program initiation through assigning adequate resources to perform the work. During 

acquisition, risks are linked to specific WBS elements so that they can be prioritized and tracked through 

risk management, using data from systems engineering, cost estimating, risk analysis, and program 

management. These efforts should result in an executable program baseline that is based on realistic cost, 

schedule, and technical goals and that provides a mechanism for addressing risks.

Cost Estimation and EVM in System Development Oversight

Government cost estimating and EVM are often conducted by different groups that barely interact during 

system development. As a result, program managers do not benefit from an integration of their efforts. 

Once the cost estimate has been developed and approved, cost estimators tend to move on to the next 

program, often not updating the cost estimate with actual costs after a contract has been awarded. In some 

cases, cost estimators do not update a cost estimate unless significant cost overruns or schedule delays have 

occurred or major requirements have changed. 

Also, EVM analysts are usually not that familiar with a program’s technical baseline document, GR&As, 

and cost estimate data or methodology. They tend to start monitoring programs without adequate 

knowledge of where and why risks are associated with the underlying cost estimate. Limited integration 

can mean that 

■ cost estimators may update the program estimate without fully understanding what the earned 

value data represent, 

60The system acquisition phase includes both contract and in-house organization efforts. If in-house staffing is selected, the effort 

should be managed in the same way as contract work. This means that in-house efforts are expected to meet the same cost, schedule, 

and technical performance goals that would be required for contract work to ensure the greatest probability of program success.

61Federal Acquisition Regulation (FAR), 48 C.F.R. § 34.202 (added by Federal Acquisition Circular 2005-11, July 5, 2006). 
GAO-09-3SP Chapter 18 201

■ EVM analysts do not benefit from cost estimators’ insight into the possible cost and schedule risks 

associated with the program, and

■ neither fully understands how risks identified with the cost estimate S curve (or cumulative 

probability distribution) translate into the program’s performance measurement baseline.

Therefore, it is considered a best practice to link cost estimating and EVM analysis. Joining forces, cost 

estimators and EVM analysts can use each other’s data to update program costs and examine differences 

between estimated and actual costs. Scope changes, risks, and opportunities can be presented to 

management in time to plan for or mitigate them. Program status can be compared to historical data to 

understand variances. Finally, cost estimators can help EVM analysts calculate a cumulative probability 

distribution to determine the confidence level in the baseline. 

EVM and Acquisition: A Baseline for Risk Management 

Using generally accepted risk management techniques, a program manager can decide how much 

management reserve budget to set aside to cover risks that were unknown at the program’s start. As the 

program develops according to the baseline plan, metrics from the EVM system can be analyzed to 

identify risks that have been realized, as well as emerging risks and opportunities. By integrating EVM 

data and risk management, program managers can develop EACs for all management levels, including 

OMB reporting requirements. In figure 21, EVM is integrated with risk management for a better program 

view.

Figure 21: Integrating EVM and Risk Management 

•

•

Use risks to establish

suitable management

reserve Ensure management

reserve sufficient to

handle high probability

of risk

• Incorporate risk impacts

into estimate at

completion

•

•

Use earned value to

monitor performance of

risk mitigation plans

Identify newly

developing risks and

opportunities

• Incorporate risk

mitigation plans into

program schedules and budgets

Integrated guidance

Earned value

management

• Define and organize

work

• Plan risk management

activities

•

•

•

•

Authorize work

Update schedules

Measure performance

Compute variances

•

•

Assign responsibility

Execute risk-handling

plans

•

•

•

Establish management

reserve

Issue target budgets

Authorize planning

•

•

•

Perform risk assessment

Identify and analyze risk

Determine risk exposure

•

•

•

Plan program

Schedule IMS

Establish performance

management baseline

• Develop risk-handling

plans

Risk management

Source: NDIA.

•

•

•

Analyze results

Plan corrective action

Update estimate at

completion

• Monitor and

communicate risks and

opportunities

Update risk register

Revisions and change control

Often, organizational barriers can keep the EVM and risk management processes independent of one 

another rather than tightly integrated. Senior management should encourage cross-organizational 

communication and training between these two disciplines to ensure that they are working together to 

better manage the risks facing a program. Doing so will promote a thorough understanding of program 
202 Chapter 18 GAO-09-3SP

risks and help improve risk mitigation. Additionally, addressing risk in the formulation of the program 

EVM baseline will result in higher credibility and the greater likelihood of success. Risk identification 

and mitigation plans should be provided to the IBR team before the IBR and assessed as part of the IBR 

process. Next, we turn to what EVM is, what some of its concepts are, and how to use its tools and gain 

from its benefits.

The Nature and History of EVM

What EVM Is

Earned value management goes beyond simply comparing budgeted costs to actual costs. It measures the 

value of work accomplished in a given period and compares it with the planned value of work scheduled 

for that period and with the actual cost of work accomplished. By using the metrics derived from these 

values to understand performance status and to estimate cost and time to complete, EVM can alert 

program managers to potential problems sooner than expenditures alone can.

Assume, for example, that a contract calls for 4 miles of railroad track to be laid in 4 weeks at a cost of 

$4 million. After 3 weeks of work, only $2 million has been spent. An analysis of planned versus actual 

expenditures suggests that the project is underrunning its estimated costs. However, an earned value 

analysis reveals that the project is in trouble because even though only $2 million has been spent, only 

1 mile of track has been laid and, therefore, the contract is only 25 percent complete. Given the value of 

work done, the project will cost the contractor $8 million ($2 million to complete each mile of track), and 

the 4 miles of track will take a total of 12 weeks to complete (3 weeks for each mile of track) instead of the 

originally estimated 4 weeks.

Thus, EVM is a means of cost and schedule performance analysis. By knowing what the planned cost 

is at any time and comparing that value to the planned cost of completed work and to the actual cost 

incurred, analysts can measure the program’s cost and schedule status. Without knowing the planned cost 

of completed work and work in progress (that is, earned value), true program status cannot be determined. 

Earned value provides the missing information necessary for understanding the health of a program; it 

provides an objective view of program status. Moreover, because EVM provides data in consistent units 

(usually labor hours or dollars), the progress of vastly different work efforts can be combined. For example, 

earned value can be used to combine feet of cabling, square feet of sheet metal, or tons of rebar with effort 

for systems design and development. That is, earned value can be employed as long as a program is broken 

down into well-defined tasks.

EVM’s History

EVM is not a new concept. It has been around in one form or another since the early 1900s, when 

industrial engineers used it to assess factory performance. They compared physical work output—earned 

value, or something gained through some effort—to the planned physical work and subsequent actual 

costs. In the 1920s, General Motors used a form of EVM called flexible budgets; by the early 1960s, 

EVM had graduated to the Program Evaluation and Review Technique, which relied on resource loaded 

networked schedules and budgets to plan and manage work. 

In 1967, DOD adopted EVM as Cost/Schedule and Control System Criteria (C/SCSC). These criteria, 

based on the best management practices used in American industry since the early 1900s, defined for 
GAO-09-3SP Chapter 18 203

defense contractors the minimum acceptable standards for providing the government with objective 

program performance reporting. C/SCSC also required contractors to integrate effort, schedule, and cost 

into a single plan. This was a broad divergence from DOD’s typical analysis of “spend plans”—comparing 

planned costs to actual costs—which gave no insight into what was actually accomplished for the money 

spent. 

Earned value technique now required contractors to report progress on cost, schedule, and technical 

achievement, giving managers access for the first time to timely and accurate status updates. The data gave 

managers the ability to confidently predict how much money it would cost and how long it would take 

to complete a contract. Rather than enforcing a particular system for contractors to implement, however, 

C/SCSC required them to develop their own management control systems that could satisfy the standards 

to use earned value effectively. 

Along with the many benefits to implementing C/SCSC came many problems. For instance, some 

programs found C/SCSC criteria overwhelming, causing them to maintain two sets of data—one for 

managing the program and one for reporting C/SCSC data. In other instances, EVM was viewed only 

as a financial management tool to be administered with audit-like rigor. A 1997 GAO report found that 

while EVM was intended to serve many different groups, program managers often ignored the data even 

though they could have benefited from responding to impending cost and schedule overruns on major 

contracts. 

To try to resolve these problems, the Office of the Secretary of Defense encouraged industry to define new 

EVM criteria that were more flexible and useful to industry and government. In 1996, DOD accepted 

industry’s revamped criteria, stating that they brought EVM back to its intended purposes of integrating 

cost, schedule, and technical effort for management and providing reliable data to decision makers.

EVM Guidelines in Practice Today

The new EVM approach encompasses 32 guidelines, organized into 5 categories of effort: (1) organizing, 

(2) planning and budgeting, (3) accounting, (4) analysis, and (5) making revisions. The guidelines define 

the major principles for managing programs, including, among other things,

■ defining and detailed planning of the scope of work using a WBS,

■ identifying organizational responsibility for doing the work,

■ scheduling authorized work,

■ applying realistic resources and budget to complete the work,

■ measuring the progress of work by objective indicators, 

■ developing a project measurement baseline,

■ collecting the cost of labor and materials associated with the work performed,

■ analyzing variances from planned cost and schedules,

■ forecasting costs at completion, 

■ taking management actions to control risk, and

■ controlling changes.
204 Chapter 18 GAO-09-3SP

The EVM guidelines today are often viewed as common sense program management practices that would 

be necessary to successfully manage any development program, regardless of size, cost, or complexity. 

Moreover, they have become the standard for EVM and have been adopted by industry, major U.S. 

government agencies, and government agencies in Australia, Canada, Japan, Sweden, and the United 

Kingdom. Furthermore, when reviewing agencies’ annual budget requests, OMB uses agency-reported 

EVM data to decide which acquisition programs to continue funding. Accordingly, government and 

industry consider EVM a worldwide best practice management tool for improving program performance.

As a key management concept, EVM has evolved from an industrial engineering tool to a government and 

industry best practice, providing improved oversight of acquisition programs. Using EVM is like forming 

an intelligent plan that first identifies what needs to be done and then uses objective measures of progress 

to predict future effort. Commercial firms told us that they use the earned value concept to manage their 

programs because they believe that good up-front technical planning and scheduling not only make sense 

but are essential for delivering successful programs.

Implementing EVM

For EVM to be effective, strong leadership from the top is necessary to create a shared vision of success 

that brings together areas often stove-piped by organizational boundaries. To accomplish this shared 

vision, senior management should set an expectation that reliable and credible data are key aspects to 

managing a successful program and show an active interest in program status to send a message to their 

staff that they are accountable and that results matter. Accordingly, stakeholders need to take an interest in 

and empower those doing the work and make sure that corporate practices are in place that allow them to 

know the truth about how a program is doing. Leadership must require information sharing in an open, 

honest, and timely fashion so it can provide resources and expertise immediately when problems begin to 

arise.

To ingrain this expectation, agencies should set forth policies that clearly define and require disciplined 

program management practices for planning and execution. As part of that policy, the focus should be 

on integrating cost, schedule, and technical performance data so that objective program progress can be 

measured and deviations from the baseline acted upon quickly. Moreover, the policy should also address 

the importance of continuous training in cost estimating, EVM, scheduling, and risk and uncertainty 

analysis that will provide the organization with high-performing and accountable people who are 

experienced in these essential disciplines. Training should be provided and enforced for all program 

personnel needing such training, not just those with program management responsibilities. While 

program managers and staff need to be able to interpret and validate EVM data to effectively manage 

deliverables, costs, and schedules, oversight personnel and decision-makers also need to understand 

EVM terms and analysis products in order to ask the right questions, obtain performance views into the 

program, and make sound investment decisions. 

The Purpose of Implementing an EVM System

Using the value of completed work for estimating the cost and time needed to complete a program should 

alert program managers to potential problems early in the program and reduce the chance and magnitude 

of cost overruns and schedule delays. EVM also provides program managers with early warning of 

developing trends—both problems and opportunities—allowing them to focus on the most critical issues.
GAO-09-3SP Chapter 18 205

The two main purposes for implementing an EVM system are to (1) encourage the use of effective 

internal cost and schedule management control systems and (2) allow the customer to rely on timely and 

accurate data for determining product-oriented contract status. To be effective, an EVM system should 

constitute management processes that serve as a comprehensive tool for integrating program planning and 

execution across cost, schedule, and technical disciplines. In essence, an EVM system should provide the 

means for planning, reporting, and analyzing program performance. 

EVM as a Planning Tool

EVM imposes the discipline of planning all work in sufficient detail so that the cost, technical effort, and 

schedule dependencies are known at the outset. When EVM is used as a planning tool, all work is planned 

from the beginning—current work in detail, future work outlined at higher levels. As the work is planned 

to a manageable level of detail, it is broken into descriptive work packages that are allocated a portion of 

the program budget. These units are then spread across the program schedule to form the performance 

measurement baseline, which is used to detect deviations from the plan and give insight into problems and 

potential impacts. 

EVM as a Management Reporting Tool

EVM objectively measures program status with objective methods such as discrete units and weighted 

milestones to determine work accomplished. These measures are based on specific criteria that are defined 

before the work starts. As work is accomplished, its value is measured against a time-phased schedule. 

While the guidelines require no specific scheduling technique, more complex programs typically use a 

networked schedule that highlights the program’s critical path.62 The earned value is measured in terms of 

the planned cost of work actually completed. This difference of including earned value allows for objective 

measurements of program status that other reporting systems cannot provide. 

EVM as an Analysis and Decision Support Tool 

EVM indicates how past performance may affect future performance. For example, EVM data isolate cost 

and schedule variances by WBS element, allowing an understanding of technical problems that may be 

causing the variances. Problems can be seen and mitigated early. In addition, opportunity can be taken in 

areas that are performing well to reallocate available budgets for work that has not yet started.63

Key Benefits of Implementing EVM

Table 29 describes some of the key benefits that can be derived from successfully implementing an EVM 

system, and figure 22 shows the expected inputs and outputs associated with tracking earned value.

62DOD interprets the guidelines to require a network schedule.

63We consulted the expert community on the issue of reallocation of budget for completed activities that underrun. The experts 

explained that while the term budget in EVM represents the plan, it is not the same thing as funding. Therefore, in EVM, a control 

account’s budget is fully earned once the effort is 100 percent complete, even if the actual cost of the effort was more or less than 

the budget. As a result, budget for past work, earned value, and actual costs need to stay together in an EVM system in order to 

maintain reporting integrity. However, if a WBS control account’s or work package’s actual cost (WBS) is under running the 

planned budget, this may suggest that the budget for future work packages may be over budgeted as well. If that is the case, then 

budget for future work could be recalled into management reserve to be available for critical path activities. According to the EVM 

guidelines, a contractor’s EVM system should allow for that. 
206 Chapter 18 GAO-09-3SP

Table 29: Key Benefits of Implementing EVM

Key benefit Description

Provides a single 

management control 

system

■ The criteria for developing an EVM system promote the integration of cost, 

schedule, and technical processes with risk management, improving the 

efficiency and effectiveness of program management; they require measuring 

progress, accumulating actual costs, analyzing variances, forecasting costs at 

completion, and incorporating changes in a timely manner

■ Implemented correctly, EVM provides a single management control system that 

prevents organizations from managing with one system and reporting from 

another. The concept that all work should be scheduled and traceable from the 

master plan to the details demonstrates that no specific scheduling software is 

required

Improves insight into 

program performance

■ Enhanced insight into program performance results from the upfront planning, 

scheduling, and control EVM requires; this is important since the window of 

opportunity for correcting project problems occurs very early in a program 

■ Studies based on the performance of over 700 contracts show that performance 

trends indicate final outcome once they are about 15% to 20% complete; thus, 

programs operating within an EVM system can quickly uncover, address, and 

resolve problems before they become out of control

Reduces cycle time to 

deliver a product

■ EVM imposes discipline and objective measurement and analysis on cost, 

schedule, and technical processes; planning and analysis often address and 

prevent problems from surfacing later

■ If costly and untimely rework can be circumvented, the time to deliver the end 

product may also be reduced

Promotes management by 

exception

■ EVM directs management attention to only the most critical problems, reducing 

information overload. Since EVM allows quick communication of cost and 

schedule variances relative to the baseline plan, management can focus on the 

most pressing problems first

Fosters accountability ■ EVM requires breaking a program down into sufficiently detailed tasks to 

clearly define what is expected and when; this allows those responsible for 

implementing specific tasks to better understand how their work fits into the 

overall program plan, establishes accountability, gives personnel a sense of 

ownership, and can result in more realistic estimates at completion of future 

tasks

■ When technical staff are held accountable for their performance, they tend to 

better understand the implications of how it affects overall program success; 

managers held accountable for their planning are more likely to implement a 

disciplined process for estimating work and tracking it through completion

Allows comparative 

analysis against completed 

projects

■ Consistent reporting of projects with EVM processes (following established 

guidelines) has for many decades resulted in a database useful for comparative 

analysis, giving managers insight into how their programs perform compared to 

historical program data

■ They can also use the data for planning programs, improving the cost 

estimating process, and determining which suppliers provided the best value in 

the past 

Provides objective 

information for managing 

the program

■ Measuring program performance gives objective information for identifying 

and managing risk; it allows early detection and resolution of problems by 

anticipating what could go wrong, based on past trends

■ Objective data obtained from an EVM system enable management to defend 

and justify decisions and determine the best course of action when problems 

arise

Source: GAO, DOD, NASA, SCEA, and industry.
GAO-09-3SP Chapter 18 207

Figure 22: Inputs and Outputs for Tracking Earned Value

Track earned

value Input Output

Performance based

specifications

Goal metric

approach

Binary quality

gates at the inch

pebble level

Establish clear

goals and

decision points

Acquisition

process

improvement

People aware

management

accountability

Configuration

management

Formal

inspections

Facilitate

establishing and

tracking earned

value metrics

Supports/

encourages

use of earned

value metrics

Are

necessary for

implementing

Provide objective

data for determining

earned value credits

Alerts program

managers to

potential schedule

and cost risks

early

Provides a

documented project

performance trail

Provides

quantitative

data for

decision making

Is a means of

communicating

project status

Demonstration

based

reviews

Formal risk

management

Acquisition process

improvement

Best value

awards

Metrics-based

scheduling and

management

Quantitative

program

measurement

Demonstrationbased reviews

Programwide

visibility of

progress vs. plan

Source: DOD and GAO.

Obstacles to EVM

Obstacles, real or imagined, stop many programs and organizations from implementing EVM. Table 30

describes ten common concerns about EVM implementation and discusses the basis of each one.

Table 30: Ten Common Concerns about EVM

Concern Basis for concern

1. EVM is too 

expensive to 

implement

■ It is expensive to implement EVM when no formal EVM system is in place. Some 

companies spend $1 million to $2 million to put a good system in place from scratch

■ Many have some elements in place and can get certified with less effort; even so, most of 

the time this is a significant investment, translating into several hundred thousand dollars. 

A simple spreadsheet workbook with worksheets for the plan and each time stamped snap 

shot of status to date can serve an effective EVM function for smaller projects

■ Companies that do establish a good EVM system realize better project management 

decision making, fewer cost and schedule overruns, and potentially greater repeat 

business. It is hard to measure those gains, but some experts have noted that the 

return on investment is reasonable. The smaller the company, the more difficult it is to 

implement because upfront costs are prohibitive

■ While an EVM system is expensive to implement, not having one may cost a company 

future work because of the inability to compete with others that have a system. The cost 

of not getting potential business is also expensive. Balancing must be done to implement 

what is required in a manner that is sensitive to the corporate bottom line
208 Chapter 18 GAO-09-3SP

Concern Basis for concern

2. EVM is not 

useful for 

short-term, 

small-dollar 

projects

■ A certain amount of judgment must be applied to determine the viability and utility of 

a full-blown EVM system for short-term or small-dollar projects. Because typical EVM 

reporting is monthly, a project of 6 months or less cannot use trends (at least three 

data points) effectively: it would be half way completed before any trending could be 

adequately used, and then corrective action would take another data point or two to 

realize. Weekly reporting would impose significantly higher resource demands and costs 

that might not be acceptable for small-dollar contracts

■ Even on shorter, less costly projects, a well-structured, planned, and executed project is 

desirable. Most projects do not trip a threshold of $20 million or $50 million, for example. 

In some cases, for every large and high visibility project there are between 10 and 20 

small projects. Failure to execute on time or within costs on these small projects is just 

as unacceptable as on large projects, even though the relative impact is smaller. Several 

small projects can add up to a substantial loss of money and unhappy customers and can 

result in the loss of larger projects or future awards if a pattern of overrunning is evident

■ EVM can be tailored and ingrained into the culture to ensure that project cost and 

schedule goals are met for smaller or shorter projects; smaller projects will benefit from 

having the work scope defined by a WBS and having a detailed plan and schedule for 

accomplishing the work. Small-dollar projects still need to have a baseline in place to 

manage changes and variances and require risk management plans to address issues 

■ On the corporate side, losing money is not an acceptable option, even if the project’s 

visibility is lower. Poor performance on a smaller project can damage a company’s 

reputation just as much as poor performance on a large, highly visible project. So even 

though a full EVM system is not required for small, short-term projects, the need to apply 

the fundamentals of EVM may still pertain. EVM is good, practical project management

3. EVM practices 

go above 

and beyond 

basic project 

management 

practices

■ Our experts noted project managers who claim that they have been successfully not 

using EVM to manage their projects for years; when pressed to say how they ensure 

that cost and schedule goals are met, and how they manage their baselines along with 

changes, however, they inevitably resort to EVM by other means 

■ The biggest difference for successful project managers is the formality and rigor of 

EVM. Our experts noted that project managers who do not use a formal EVM system 

generally do so because they are not required to. Those who are forced to use formal 

EVM practices often do so grudgingly but warm up to it over time. Those who have been 

using formal EVM for years often do not know how they got by without it in the past

■ A second difference between formal EVM practices and basic project management 

practices is the uniformity of data and formatting of information that makes it possible 

to draw comparisons against other like projects. Successful project managers who do 

not use a formal EVM system invariably have their “own system” that works for them 

and does much of the same things as a formal system. Unfortunately, it is very difficult 

to compare their systems to other projects, to do analysis, or to validate the data for 

good decision making. How much management visibility these systems have for timely 

decision making is debatable. Many companies, hindered by problem identification and 

corrective actions, have limited management insight into their projects

■ The rigor and discipline of a formal EVM system ensure a certain continuity and 

consistency that are useful, notwithstanding the availability and turnover of 

knowledgeable personnel. When staff leave the job for an extended time, the structure 

of the system makes it possible for another person to take over for those who left. The 

new staff may not have the personal knowledge of the specific project, schedule, or EVM 

data but may understand enough about EVM to know how to interpret the data and 

evaluate the processes because of this disciplined structure

■ Thus, EVM practices go beyond the basics, have greater rigor and formality; the benefit is 

that this ensures uniform practices that are auditable and consistent with other entities 

for relative comparison and benchmarking. Without this formality, it would be much 

more difficult to draw industry standard benchmarks and comparisons for improvement
GAO-09-3SP Chapter 18 209

Concern Basis for concern

4. EVM is merely 

a government 

reporting 

requirement 

■ It is, viewed only as a reporting requirement. But the benefit of a formal EVM system 

in government reporting is that the end-product occurs after organizing, planning, 

authorizing, executing, change management, analysis, and controlling are completed. 

The reports give management as well as government a view into the health of a project 

to make sure taxpayer money is being used judiciously

■ While it makes for project visibility to the government, it is primarily intended as a 

systematic approach to help in managing a project or program. Reports are only as good 

as the data and the processes that support them; EVM serves more as a set of mandated 

government project management tools with reporting as a by-product

5. Reports are a 

key product 

of EVM

■ Yes they are, but it would be short sighted to focus on reporting without recognizing the 

need for other subsets of an EVM system to provide reliable and auditable data. What 

comes out is a by-product of what goes in and how well it is maintained

■ EVM reporting is intended to provide reliable information for timely decision making to 

maximize the probability of successfully executing a project; it is a project management 

“process tool set” that helps make certain that proven management techniques are used 

to run projects

■ Where EVM is institutionalized, management uses reports to identify significant variances 

and drill down into areas of exception (management by exception) for corrective actions 

and decision making. When EVM is ingrained, reports are greatly anticipated and 

thoroughly discussed by senior management

6. EVM is a 

financial 

management 

tool

■ Yes, to some degree, but in reality, it is an enhancement to traditional financial 

management; EVM requirements came about largely to reduce the high percentage of 

cost and schedule overruns that still ended up delivering a product that was technically 

inferior to the government’s product. Trying to do forensic analysis of a failed project is 

tough enough without a reliable and rigorous system in place. If one can prevent having 

a failed project in the first place, forensics may not be necessary

■ EVM enhances the traditional financial management tool by adding visibility of actual 

performance for budgeted tasks; this dimension of information, coupled with the 

traditional planned budget vs. actual costs, allows for better forecasting of final costs, as 

well as early warning of performance variances for timely decision making and corrective 

actions

■ Because EVM is a more accurate mechanism for predicting costs than the traditional 

financial models, it is more reliable for determining funding requirements and use

7. EVM data are 

backward 

looking and 

too old to be 

useful

■ This is only partially true. Some metrics data an EVM system, produce are backward 

looking and show performance to date, both cumulative and by period; they can help 

identify past trends that can reliably be used to predict costs and schedule performance, 

along with the final cost of a project

■ Presenting standard graphics is a best practice for reporting EVM trends and status to 

senior management

■ Using EVM, management has the ability to make timely decisions and adjustments as 

needed to affect the final outcome of a project and maximize profitability

8. Variances 

EVM reveals 

are bad and 

should always 

be avoided

■ Variances are expected because programs are rarely performed to plan: neither good nor 

bad, they simply measure how much actual performance has varied from the plan

■ Variance thresholds try to quantify an acceptable range of deviation; those that do not 

exceed a threshold are not usually a concern while those that do are worthy of further 

inspection to determine the best course of action to minimize any negative impacts to 

cost and schedule objectives

■ Variances can indicate one or more of the following: how well the project was planned 

(statement of work definition, estimating and estimating assumptions, execution 

strategy, procurement strategy, risk management); how well changes to the baseline plan 

are being implemented; how much planned and unplanned change has occurred since 

inception; how well the project is being executed
210 Chapter 18 GAO-09-3SP

Concern Basis for concern

9. No one cares 

about EVM 

data 

■ False. That is like saying that the pilot of a jet aircraft does not care about what the 

navigation instrumentation says. EVM data are the navigation instrumentation that tells 

the project manager how well the flight plan is working

■ If line managers and the project manager ignore the EVM data, they may not arrive at 

cost and schedule goals; the data help them make the necessary midcourse adjustments 

so they can arrive at the planned destination on time

10. EVM does 

not help with 

managing a 

program

■ False: Refer to previous 9 items, especially 3, 7, 8, and 9, which apply to both projects and 

programs

■ When managing a program, it is very important to identify and manage resources to 

ensure that over- or underallocations do not exist; EVM helps identify these conditions 

■ It helps identify and manage program and project risks and program and project funding 

requirements to ensure that funding shortfalls do not surprise the program manager

Source: GAO.

Implementing EVM at the Program Level

Implementing EVM at the program rather than just the contract level is considered a best practice. 

Furthermore, it directly supports federal law requiring executive agency heads to approve or define the 

cost, performance, and schedule goals for major agency acquisition programs. Specifically, the Federal 

Acquisition Streamlining Act of 1994 established the congressional policy that the head of each executive 

agency should achieve, on average, 90 percent of the agency’s cost, performance, and schedule goals 

established for major acquisition programs.64 When it is necessary to implement this policy, agency heads 

are to determine whether there is a continuing need for programs that are significantly behind schedule, 

over budget, or not in compliance with the performance or capability requirements and identify suitable 

actions to be taken, including termination. Additionally, OMB Circular A-11, part 7, section 300, 

addresses the use of EVM as an important part of a program’s management and decision making.65 That 

policy requires the use of an integrated EVM system across the entire program to measure how well the 

government and its contractors are meeting a program’s approved cost, schedule, and performance goals. 

Integrating government and contractor cost, schedule, and performance status should result in better 

program execution through more effective management. In addition, integrated EVM data can be used to 

justify budget requests.

Requiring EVM at the program level also makes government functional area personnel accountable for 

their contributions to the program. Further, it requires government agencies to plan for a risk-adjusted 

program budget so that time and funds are available when needed to meet the program’s approved 

baseline objectives. Continuous planning through program-level EVM also helps government program 

64 41 U.S.C. § 263. A similar requirement in 10 U.S.C. § 2220 applied to the Department of Defense but was later amended to 

remove the 90 percent measure. DOD has its own major program performance oversight requirements in chapters 144 (Major 

Defense Acquisition Programs) and 144A (Major Automated Information System Programs) of title 10 of the U.S. Code, including 

the Nunn-McCurdy cost reporting process at 10 U.S.C. § 2433. Regarding information technology programs, 40 U.S.C. § 11317 

(formerly 40 U.S.C. § 1427) requires agencies to identify in their strategic information resources management plans any major 

information technology acquisition program, or phase or increment of that program, that has significantly deviated from cost, 

performance, or schedule goals established for the program. 

65OMB, Preparation, Submission, and Execution of the Budget, Circular A-11 (Washington, D.C.: Executive Office of the President, 

June 2006), part 7, Planning, Budgeting, Acquisition, and Management of Capital Assets, sec. 300. www.whitehouse.gov/omb/

circulars/index.html.
GAO-09-3SP Chapter 18 211

managers adequately plan for the receipt of material, like government furnished equipment, to ensure 

that the contractor can execute the program as planned. Finally, program-level EVM helps identify key 

decision points up front that should be integrated into both the contractor’s schedule and the overall 

program master schedule, so that significant events and delivery milestones are clearly established and 

known by all. IBRs should include all government and contractor organizations involved in performing 

the program, as well as those responsible for establishing requirements, performing tests, and monitoring 

performance.

Federal and Industry Guidelines for Implementing EVM

The benefits of using EVM are singularly dependent on the data from the EVM system. Organizations 

must be able to evaluate the quality of an EVM system in order to determine the extent to which the 

cost, schedule, and technical performance data can be relied on for program management purposes. In 

recognition of this, the American National Standards Institute (ANSI) and the Electronic Industries 

Alliance (EIA) have jointly established a national standard for EVM systems—ANSI/EIA-748-B. The 

National Defense Industrial Association (NDIA) is the subject matter expert for the standard.66

Soon after the standard was established, leading companies, including commercial businesses, began 

using it to manage their programs, even though they did not mandate EVM. They saw ANSI and 

EIA standards as best practices that provided a scaleable approach to using EVM for any contract type, 

contract size, and duration. 

DOD adopted the ANSI guidelines for managing government programs with the expectation that 

program managers would be responsible for ensuring that industry-developed standards were being met 

by ongoing process surveillance. Other agencies soon followed DOD’s example. Recently, OMB imposed 

the use of EVM for all major capital acquisitions in accordance with OMB Circular A-11, Part 7—OMB 

stated in its 2006 Capital Programming Guide that all major acquisitions with development effort are to 

require that contractors use an EVM system that meets the ANSI guidelines.67

The ANSI guidelines were originally written for companies, but the EIA-748-B version began introducing 

more generic terminology for government and other organizations. They consist of 32 guidelines in five 

basic categories: (1) organization; (2) planning, scheduling, and budgeting; (3) accounting considerations; 

(4) analysis and management reports; and (5) revisions and data maintenance (see table 31). In general, 

they define acceptable methods for organizations to define the contract or program scope of work using 

a WBS; identify the organizations responsible for performing the work; integrate internal management 

subsystems; schedule and budget authorized work; measure the progress of work based on objective 

indicators; collect the cost of labor and materials associated with the work performed; analyze variances 

from planned cost and schedules; forecast costs at contract completion; and control changes.

66 See, for example, ANSI/EIA 748 32 Industry Guidelines (American National Standards Institute (ANSI)/Electronic Industries 

Alliance (EIA) Standard, Earned Value Management Systems, ANSI/EIA-748-B-2007, approved July 9, 2007, at www.acq.osd.mil/

pm/historical/Timeline/EV%20Timeline.htm, and NDIA, National Defense Industrial Association (NDIA) Program Management 

Systems Committee (PMSC) ANSI/EIA-748-A Standard for Earned Value Management Systems Intent Guide (Arlington, Va.: 

January 2005).

67 See OMB, Capital Programming Guide, II.2.4, “Establishing an Earned Value Management System.” The OMB requirements are 

also reflected in the FAR at 48 C.F.R. subpart 34.2. 
212 Chapter 18 GAO-09-3SP

Table 31: ANSI Guidelines for EVM Systems

Guideline Category and statement

Organization

1 Define the authorized work elements for the program. A WBS, tailored for effective internal 

management control, is commonly used in this process

2 Identify the program organizational structure, including the major subcontractors responsible 

for accomplishing the authorized work, and define the organizational elements in which work 

will be planned and controlled

3 Provide for the integration of the planning, scheduling, budgeting, work authorization, and cost 

accumulation processes with one another and, as appropriate, the program WBS and program 

organizational structure

4 Identify the organization or function responsible for controlling overhead (indirect costs)

5 Provide for integration of the program WBS and the program organizational structure in a 

manner that permits cost and schedule performance measurement by elements of either 

structure or both, as needed

Planning, scheduling, and budgeting

6 Schedule the authorized work in a way that describes the sequence of work and identifies 

significant task interdependencies required to meet the program’s requirements

7 Identify physical products, milestones, technical performance goals, or other indicators that will 

be used to measure progress

8 Establish and maintain a time-phased budget baseline, at the control account level, against 

which program performance can be measured. Initial budgets established for performance 

measurement will be based on either internal management goals or the external customernegotiated target cost, including estimates for authorized but undefinitized work.a

 Budget for 

far-term efforts may be held in higher-level accounts until an appropriate time for allocation 

at the control account level. If an overtarget baseline is used for performance measurement 

reporting purposes, prior notification must be provided to the customer

9 Establish budgets for authorized work with identification of significant cost elements (labor, 

material) as needed for internal management and control of subcontractors

10 To the extent it is practical to identify the authorized work in discrete work packages, establish 

budgets for this work in terms of dollars, hours, or other measurable units. Where the entire 

control account is not subdivided into work packages, identify the far-term effort in larger 

planning packages for budget and scheduling purposes

11 Provide that the sum of all work package budgets and planning package budgets within a 

control account equals the control account budget

12 Identify and control level-of-effort activity by time-phased budgets established for this purpose. 

Only effort not measurable or for which measurement is impractical may be classified as level of 

effort

13 Establish overhead budgets for each significant organizational component for expenses that will 

become indirect costs. Reflect in the program budgets, at the appropriate level, the amounts in 

overhead pools that are planned to be allocated to the program as indirect costs

14 Identify management reserves and undistributed budget

15 Provide that the program target cost goal is reconciled with the sum of all internal program 

budgets and management reserves

Accounting considerations

16 Record direct costs in a manner consistent with the budgets in a formal system controlled by the 

general books of account

17 When a WBS is used, summarize direct costs from control accounts into the WBS without 

allocating a single control account to two or more WBS elements
GAO-09-3SP Chapter 18 213

Guideline Category and statement

18 Summarize direct costs from the control accounts into the organizational elements without 

allocating a single control account to two or more organizational elements

19 Record all indirect costs that will be allocated to the program consistent with the overhead 

budgets

20 Identify unit costs, equivalent unit costs, or lot costs when needed

21 For EVM system, the material accounting system will provide for (1) accurate cost accumulation 

and assignment of costs to control accounts in a manner consistent with the budgets using 

recognized, acceptable, costing techniques; (2) cost recorded for accomplishing work performed 

in the same period that earned value is measured and at the point in time most suitable for the 

category of material involved but no earlier than the time of actual receipt of material; (3) full 

accountability of all material purchased for the program, including the residual inventory

Analysis and management reports

22 At least monthly, generate the following information at the control account and other levels 

as necessary for management control, using actual cost data from, or reconcilable with, the 

accounting system: (1) comparison of the amount of planned budget and the amount of budget 

earned for work accomplished (this comparison provides the schedule variance); (2) comparison 

of the amount of the budget earned and the actual (applied where appropriate) direct costs for 

the same work (this comparison provides the cost variance)

23 Identify, at least monthly, the significant differences between both planned and actual schedule 

performance and planned and actual cost performance and provide the reasons for the 

variances in the detail needed by program management

24 Identify budgeted and applied (or actual) indirect costs at the level and frequency needed by 

management for effective control, along with the reasons for any significant variances

25 Summarize the data elements and associated variances through the program organization or 

WBS to support management needs and any customer reporting specified in the contract

26 Implement managerial actions taken as the result of earned value information

27 Develop revised estimates of cost at completion based on performance to date, commitment 

values for material, and estimates of future conditions. Compare this information with 

the performance measurement baseline to identify variances at completion important to 

management and any applicable customer reporting requirements, including statements of 

funding requirements

Revisions and data maintenance

28 Incorporate authorized changes in a timely manner, recording their effects in budgets and 

schedules. In the directed effort before negotiating a change, base such revisions on the amount 

estimated and budgeted to the program organizations

29 Reconcile current budgets to prior budgets in terms of changes to authorized work and internal 

replanning in the detail needed by management for effective control

30 Control retroactive changes to records pertaining to work performed that would change 

previously reported amounts for actual costs, earned value, or budgets. Adjustments should 

be made only for correcting errors, making adjustments for routine accounting or the effects of 

customer or management directed changes, or improving the baseline integrity and accuracy of 

performance measurement data

31 Prevent revisions to the program budget except for authorized changes

32 Document changes to the performance measurement baseline

Source: Excerpts from Earned Value Management Systems (ANSI/EIA 748-B), Copyright © (2007), Government Electronics and Information Technology Association. All Rights 

Reserved. Reprinted by Permission.

a

An undefinitized contract is one in which the contracting parties have not fully agreed on the terms and conditions.
214 Chapter 18 GAO-09-3SP

As noted earlier, OMB requires the use of EVM on all major acquisition programs for development. 

Further, it must be compliant with agencies’ implementation of the ANSI guidelines. Several other guides 

are available to help agencies implement EVM systems. We outlined these guides in table 3 and list them 

again here in table 32).

Table 32: EVM Implementation Guides

Guide

Applicable 

agency Description

DOD, The Program Manager’s Guide to 

the Integrated Baseline Review Process

(Washington, D.C.: OSD (AT&L), April 2003)

DOD Defines the IBR’s purpose, goals, and objectives; 

discusses how it leads to mutual understanding 

of risks inherent in contractors’ performance 

plans and management control systems; and 

explains the importance of formulating a plan 

to handle and mitigate these risks

NDIA, National Defense Industrial Association 

(NDIA) Program Management Systems 

Committee (PMSC) Surveillance Guide

(Arlington, Va.: October 2004).

All Defines a standard industry approach for 

monitoring whether an EVM system satisfies the 

processes and procedures outlined in the ANSI 

guidelines

NDIA, National Defense Industrial Association 

(NDIA) Program Management Systems 

Committee (PMSC) Earned Value Management 

Systems Intent Guide (Arlington, Va.: January 

2005).

All Defines in detail the management value and 

intent for all 32 ANSI guidelines. Contractors 

use it to assess initial compliance and perform 

implementation surveillance

Defense Contract Management Agency, 

Department of Defense Earned Value 

Management Implementation Guide

(Alexandria, Va.: October 2006)

DOD, FAA, 

NASA

Provides guidance on the framework to follow 

during implementation and surveillance of an 

EVM system

National Defense Industrial Association, 

Program Management Systems Committee, 

“NDIA PMSC ANSI/EIA 748 Earned Value 

Management System Acceptance Guide,” 

draft, working release for user comment 

(Arlington, Va.: November 2006)

All Defines an EVM system acceptance process 

that would apply to industry and government. 

NDIA has expanded this proposal to a draft EVM 

process implementation guide that will connect 

its guides with more specific information on 

how they relate to one another

National Defense Industrial Association, 

Program Management Systems Committee, 

“NDIA PMSC Earned Value Management 

Systems Application Guide,” draft, working 

release for use and comment (Arlington, Va.: 

March 2007)

All Defines a standard approach for all 

organizations implementing an EVM system 

through all phases of acquisition

Source: GAO.

The remainder of the Cost Guide assumes that readers understand basic EVM principles. Readers 

unfamiliar with EVM can also obtain such information from, for example, the Defense Acquisition 

University and the Project Management Institute (PMI).68

68 See, for example, DAU’s fundamental courses at www.dau.mil/schedules/schedule.asp and PMI’s literature at www.pmibookstore.

org/PMIBookStore/productDetails.aspx?itemID=372&varID=1.
GAO-09-3SP Chapter 18 215

The Thirteen Steps in the EVM Process

The EVM process has thirteen fundamental steps, outlined and described in this section: 

1. define the scope of effort using a WBS;

2. identify who in the organization will perform the work;

3. schedule the work;

4. estimate the labor and material required to perform the work and authorize the budgets, including 

management reserve;

5. determine objective measure of earned value;

6. develop the performance measurement baseline;

7. execute the work plan and record all costs;

8. analyze EVM performance data and record variances from the PMB plan;

9. forecast EACs using EVM;

10. conduct an integrated cost-schedule risk analysis;

11. compare EACs from EVM (step 9) with EAC from risk analysis (step 10);69

12. take management action to mitigate risks; and

13. update the performance measurement baseline as changes occur.

Define the Scope with a WBS

The WBS, a critical component of EVM that defines the work to be performed, should be the basis of the 

cost estimate and the project schedule. In the schedule, the WBS elements are linked to one another with 

logical relationships and lead to the end product or final delivery. The WBS progressively deconstructs the 

deliverables of the entire effort through lower-level WBS elements and control accounts.

Figure 23 shows how the overall program plan breaks down. The hierarchical WBS ensures that the 

entire statement of work accounts for the detailed technical tasks and, when completed, facilitates 

communication between the customer and supplier on cost, schedule, technical information, and the 

progress of the work. It is important that the WBS is comprehensive enough to represent the entire 

program to a level of detail sufficient to manage the size, complexity, and risk associated with the program. 

In addition, the WBS should be the basis of the program schedule. Furthermore, there should be only 

one WBS for each program, and it should match the WBS used for the cost estimate and schedule so 

that actual costs can be fed back into the estimate and there is a correlation between the cost estimate and 

schedule. Moreover, while costs are usually tracked at lower levels, what is reported in an EVM system is 

usually summarized at a higher level, perhaps matching the summary level of the schedule that is often 

used for a schedule risk analysis, facilitating the preparation of an integrated cost-schedule risk analysis. 

However, through the fluidity of the parent-child relationship, the WBS can be expanded to varying 

degrees of detail so that problems can be quickly identified and tracked.

69This step demonstrates the integration of EVM and risk management processes.
216 Chapter 18 GAO-09-3SP

Figure 23: WBS Integration of Cost, Schedule, and Technical Information

System specification

1000 Prime mission

1100 Subsystem A

1110 Component 1

Requirements

1000 Prime mission product

 1100 Subsystem A

 1110 Component 1

 o

 o

 o

 1189 Component n 

WBS elements

1000 Prime mission product

 1100 Subsystem A

 1110 Component 1

 o

 o

 o

$1,234,567

$456,890

$23,552

Program cost estimate

Program plan Events

1. Preliminary design review (PDR) PDR 1.a. Duty cycle defined

 b. Preliminary analysis complete

Accomplishment criteria

Program plan

Detailed tasks 20 XX 20 XY 20 XZ

Program events

1. Preliminary design complete, duty cycle defined

PDR CDR

Program schedule

Source: NDIA.

Subsystem (WBS 1100)

Design, develop, produce,

and verify complete

subsystem A, defined as

component 1, component 2,

and other elements

Technical description

Note: CDR = critical design review. 

Identify Who Will Do the Work

Once the WBS has been established, the next step is to assign someone to do the work. Typically, someone 

from the organization is assigned to perform a specific task identified in the WBS. To ensure that 

someone is accountable for every WBS element, it is useful to determine levels of accountability, or control 

accounts, at the points of intersection between the organizational breakdown structure and the WBS. The 

control account becomes the management focus of an EVM system and the focal point for performance 

measurement. 

It is at the control account level that actual costs are collected and variances from the baseline plan are 

reported in the EVM system. Figure 24 shows how control accounts are determined. The WBS is shown 

at the top, including program elements and contract reporting elements and detailed elements. To the left 

is the organizational breakdown structure. The control accounts lie in the center of the figure, where the 

WBS and organizational breakdown structure intersect. As the box at the far right of the figure indicates, 

each control account is further broken down into work packages and planning packages. Each of these has 

staff who are assigned responsibility for managing and completing the work.
GAO-09-3SP Chapter 18 217

Figure 24: Identifying Responsibility for Managing Work at the Control Account 

Develop

and integrate

aquarium

Program

management

and systems

engineering

Aquarium

development

program

Organizational

breakdown

structure

Program WBS

elements

Contract WBS

reporting

elements

Contract WBS

detailed

elements

Control

account

Control

account

Hydrobotanical

engineering

Hardware

engineering Operations

Control

account

Hydro

engineering

Biological

engineering Engineering Aquaticsdivision

Control

account

Control

account

Control

account

Tropical

engineering Marketing

Tropical

fish

Plant

life

Environmental

control

Integration

quality

control

Biological

integration

Structural

integration

Development

documentation

Material

integration

Material

acquisition

Work packages

Planning

packages

WBS data

summary

OBS data summary

Source: © 2003 SCEA, “Earned Value Management Systems.”

Control accounts represent the level by which actual costs are accumulated and compared to planned 

costs. A control account manager is responsible for managing, tracking, and reporting all earned value 

data defined within each control account. Thus, control accounts are the natural control point for EVM 

planning and management.

Work packages—detailed tasks typically 4 to 6 weeks long—require specific effort to meet control 

account objectives and are defined by who authorizes the effort and how the work will be measured and 

tracked. They reflect near-term effort. Planning packages are far-term work and usually planned at higher 

levels. Budgets for direct labor, overhead, and material are assigned to both work and planning packages so 

that total costs to complete the program are identified at the outset. As time passes, planning packages are 

broken down into detailed work packages. This conversion of work from a planning to a work package, 

commonly known as “rolling wave” planning, occurs for the entire life of the program until all work has 

been planned in detail. A best practice is to plan the rolling wave to a design review, test, or other major 

milestone rather than to an arbitrary period such as 6 months.

In planning the baseline, programs ought to consider the allocation of risk into the baseline up front—

especially when addressing the issue of rework and retesting. Experts have noted that to set up a realistic 

baseline, anticipated rework could be a separate work package so as to account for a reasonable amount 

of rework but still have a way to track variances. Using this approach, programs are not to exclude rework 
218 Chapter 18 GAO-09-3SP

from the budget baseline, because they acknowledge efforts that are bound to involve a lot of revision like 

design.

Schedule the Work to a Timeline

Developing a schedule provides a time sequence for the duration of the program’s activities and helps 

everyone understand both the dates for major milestones and the activities, often called “critical and nearcritical activities,” that drive the schedule. A program schedule also provides the vehicle for developing 

a time-phased budget baseline. The typical method of schedule analysis is the critical path method, 

implemented in standard scheduling software packages. 

Because some items such as labor, supervision, rented equipment and facilities, and escalation cost more if 

the program takes longer, a schedule can contribute to an understanding of the cost impact if the program 

does not finish on time. The program’s success also depends on the quality of its schedule. If it is well 

integrated, the schedule clearly shows the logical relationships between program activities, activity resource 

requirements and durations, and any constraints that affect their start or completion. The schedule shows 

when major events are expected as well as the completion dates for all activities leading up to them, which 

can help determine if the schedule is realistic and achievable. When fully laid out, a detailed schedule 

can be used to identify where problems are or could potentially be. Moreover, as changes occur within a 

program, a well-statused schedule will aid in analyzing how they affect the program. 

For these reasons, an integrated schedule is key in managing program performance and is necessary for 

determining what work remains and the expected cost to complete it. As program complexity increases, so 

must the schedule’s sophistication. To develop and maintain an integrated network schedule, 

■ all activities must be defined (using the WBS) at some level of detail; 

■ all activities must be sequenced and related using network logic. The schedule should be 

horizontally and vertically integrated;

■ the activities must be resource-loaded with labor, material, and overhead; 

■ the duration of each activity must be estimated, usually with reference to the resources to be 

applied and their productivity, along with any external factors affecting duration;

■ the program master schedule and critical path must be identified;

■ float—the amount of time a task can slip before affecting the critical path—for activities must be 

calculated; 

■ a schedule risk analysis must be run for larger, more complex, important, or risky programs; 

■ the schedule should be continuously updated using logic and durations to determine dates; and

■ the schedule should be analyzed continuously for variances and changes to the critical path and 

completion date.

We discuss each of these items next.

The schedule should reflect all activities (steps, events, outcomes), including activities the government 

and its contractors are to perform, and should be derived from the program’s work breakdown structure. 

The schedule’s activities should also be traceable to the program statement of work to ensure all effort 
GAO-09-3SP Chapter 18 219

is included. Steps 1 and 2 of the EVM process define the activities and provide input for loading the 

activities with labor costs. 

The schedule should line up all activities in the order that they are to be carried out. In particular, 

activities that must finish before the start of other activities (that is, predecessor activities) as well as 

activities that cannot begin until other activities are completed (successor activities) should be identified. 

In this way, dependencies among activities that lead to the accomplishment of events or milestones can be 

established and used as a basis for guiding work and measuring progress. When activities are sequenced, 

using dependencies between them that reflect the program’s execution plan, the result is a network of 

activity chains like those shown in figure 25. 

Figure 25: An Activity Network 

Activities in the critical path

Activities not in the critical path

Start

Start

Start

Start

Source: © 2005 MCR LLC, “Schedule Risk Analysis.”

A network diagram not only outlines the order of the activities and their dependencies; it also documents 

how the program measures progress toward certain milestones. By linking activities with finish-to-start 

logic, one can know which activities must finish before others (known as predecessor activities) begin and 

which activities may not begin until others (successor activities) have been completed. Other relationships 

such as start-to-start and finish-to-finish are used as well. Using this approach, a valid Critical Path 

Method (CPM) network of logically linked tasks and events begins to emerge, enabling the schedule 

network to calculate dates and to predict changes in future task performance. A valid CPM network 

should be the basis for any schedule so that it best represents the plan and can respond to changes. This 

information fosters communication between team members and better understanding of the program 

as a whole, identifies disconnects as well as hidden opportunities, and promotes efficiency and accuracy. 

Moreover, this also provides a method for controlling the program by comparing actual to planned 

progress.

Schedules should be integrated horizontally and vertically. Integrated horizontally, the schedule links the 

products and outcomes associated with already sequenced activities. These links are commonly referred 

to as hand offs and serve to verify that activities are arranged in the order that will achieve aggregated 

products or outcomes. Horizontal integration also demonstrates that the overall schedule is rational, 

planned in a logical sequence, accounts for interdependencies between work and planning packages, and 
220 Chapter 18 GAO-09-3SP

provides a way to evaluate current status. Being traceable horizontally, however, is not a simple matter 

of making sure that each activity has a successor. Activities need to have certain predecessor-successor 

relationships so the schedule gives the correct results when they are updated or when durations change. 

Two logic requirements have to be provided:

1. finish-to-start or start-to-start predecessors, so that if the activity is longer than scheduled it does 

not just start earlier automatically, and

2. finish-to-start or finish-to-finish successors that will be “pushed” if they take longer or finish later. 

These logical requirements are needed to prevent “dangling logic,” which happens when activities or tasks 

are created without predecessors or successors. Fundamentally, although a start-to-start successor is proper 

and sometimes useful, it is not sufficient to avoid danglers. With dangling logic, risk in activities will not 

cascade down to their successors automatically when schedules are updated. This is not only good critical 

path method scheduling but is also crucial during Monte Carlo simulation when activity durations are 

changed on purpose thousands of times. Without this logic, the simulation will not be able to identify the 

correct dates and critical paths when the durations change.

The schedule should also be integrated vertically, meaning that traceability exists among varying levels of 

activities and supporting tasks and subtasks. Such mapping or alignment within the layers of the schedule 

among levels—master, intermediate, detailed—enables different groups to work to the same master 

schedule. When schedules are vertically integrated, lower-level schedules are clearly traced to upper-tiered 

milestones, allowing for total schedule integrity and enabling different teams to work to the same schedule 

expectations.

More risky or more complex programs should have resource-loaded schedules—that is, schedules with 

resources of staff, facilities, and materials needed to complete the activities that use them. Resource loading 

can assist in two ways:

1. scarce resources can be defined and their limits noted, so that when they are added to the activities 

and “resource-leveled,” the resources in scarce supply will not be overscheduled in any time period, 

and 

2. all resources can be defined and have costs placed on them so that the program cost estimate can 

be developed within the scheduling package.

The next step is estimating how long each activity will take—who will do the work, whether the resources 

are available and their productivity, and whether any external factors might affect the duration (funding 

or time constraints). It is crucial at this point in schedule development to make realistic assumptions 

and specify realistic durations for the activities. In determining the duration of each activity, the same 

rationale, data, and assumptions used for cost estimating should be used for schedule estimating. Further, 

these durations should be as short as possible and they should have specific start and end dates. Excessively 

long periods needed to execute an activity should prompt further decomposition of the activity so that 

shorter execution durations will result.

Often the customer, management, or other stakeholder will ask to shorten the program schedule. 

Strategies may help. Some activities can be shortened by adding more people to do the work, although 
GAO-09-3SP Chapter 18 221

others will take a fixed amount of time no matter what resources are available. Other strategies often 

require “fast track” or “concurrent” scheduling that schedules successor activities or phases to finish before 

their logical predecessors have completed. In this case, activities or phases that would without the pressure 

for a shorter schedule, be scheduled in sequence are overlapped instead. This approach must be used with 

caution since shortening activity durations or overlapping activities may not be prudent or even possible. 

Further, schedules need to consider program calendars and special calendars that may be more appropriate 

for shared resources—test facilities may work 24/7; calendars recognize holidays and other vacations. If 

training is required, it should be provided in the schedule. Also, since it is sometimes unwise to assume 

100 percent productivity, many organizations routinely provide sick leave in their estimates. Procurement 

time for ordering and receiving material and equipment must be added so it is available when needed—

some material and equipment take time to obtain or produce and are often called long lead time items. 

Schedules need to recognize these items as critical, so they can be ordered before design is complete. 

It is useful to rely on historical data for scheduling information as much as possible when developing 

activity durations so they are as realistic as possible. Often parts of the program have no analogous 

estimates, so program participants will use expert judgment to estimate durations. Further , it is a best 

practice for schedule duration rationale to tie directly to the cost estimate documentation. Figure 26 shows 

the typical output of the activity duration estimate.

Figure 26: Activity Durations as a Gantt Chart 

2002, Qtr 2 NameID Start

Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

2002, Qtr 3 2002, Qtr 4 2003, Qtr 1

1 Event 1

2 Accomplishment 1.1

3 Criterion 1.1.1

4 Task 1.1.1.1

5 Criterion 1.1.2

6 Task 1.1.2.1

7 Accomplishment 1.2

8 Criterion 1.2.1

9 Task 1.2.1.1

10 Event 2

11 Accomplishment 2.1

12 Criterion 2.1.1

13 Task 2.1.1.1

14 Criterion 2.1.2

4/28/02

4/28/02

4/28/02

4/28/02

5/12/02

5/12/02

5/5/02

5/5/02

5/5/02

5/3/02

5/3/02

7/26/02

7/26/02

5/3/02

Source: DOD.

Historically, state-of-the-art technology development programs have taken longer than planned for the 

same reasons that costs often exceed the estimate: no point estimate for schedule duration is correct and 

risk is generally high in development programs. Instead, each estimate of activity duration has a range 

of possible outcomes, driven by various uncertainties such as lack of available technical capability, slow 
222 Chapter 18 GAO-09-3SP

software development, integration problems, and test failures. Even if staff work overtime, schedule 

overruns may still occur, since overworked staff are less efficient. Understanding how program risks 

may affect durations is often accomplished by using 3-point estimate durations (optimistic, most 

likely, and pessimistic) to check the reasonableness of the durations used. A standard way to use these 

values to improve the accuracy of the schedule durations is to average them. The resulting single-point 

or “deterministic” durations are usually more accurate than simply the most likely durations without 

considering other possible scenarios. 

After the activity durations have been estimated, scheduling software can be used to determine the 

program’s overall schedule and critical path, which represents the chain of dependent activities with the 

longest total duration.70 Along the critical path—the shaded boxes in figure 25—if any activity slips, the 

entire program will be delayed. Therefore, management must focus not only on problems in activities 

along the critical path (activities with zero total float) but also on near-critical activities (activities with 

low total float), because these activities typically have the least time to slip before they delay the total 

program. Management should also identify whether the problems are associated with items being tracked 

on the program’s risk management list. This helps management develop workarounds, shift resources 

from noncritical path activities to cover critical path problems, and implement risk management actions 

to address problem areas. In addition, the critical path in the overall schedule is invaluable in helping 

determine where management reserve and unfunded contingencies may exist. 

The schedule should identify how long a predecessor activity can slip before the delay affects successor 

activities, known as float. As a general rule, activities along the critical path have the least amount of float. 

Therefore, critical path tasks have the least schedule flexibility. 

Also called slack or total float or total slack, float is the time an activity can slip before it impacts the 

end date of the program. The time a predecessor activity can slip before the delay affects the successor 

activities is called free-float, or free-slack. It is a subset of the total float and is calculated, for a finish-tostart relationship, as the early start of the successor minus the early finish of the predecessor. For other 

relationships, this calculation is similar, going with the “flow” of their relationship. This concept of freefloat is important, because some resources of the affected activities may be available only during certain 

time periods, which could be detrimental to the completion of the subsequent activities and even the 

entire program. 

As the schedule is statused, float will change and can be positive or negative. Positive float indicates the 

amount of time the schedule can fluctuate before affecting the end date. Negative float indicates critical 

path effort and may require management action such as overtime, second or third shifts, or resequencing 

of work. As a result, float should be continuously assessed.

A schedule risk analysis should be performed using a good CPM schedule and data about project 

schedule risks, as well as Monte Carlo simulation techniques to predict the level of confidence, in 

meeting a program’s completion date, the contingency time needed for a level of confidence, and identify 

high-priority risks. This analysis focuses not only on critical path activities but also on other schedule 

70 Since the activity durations are estimates and may differ from those in the schedule, the actual critical path may differ from 

that computed by the scheduling software. This is one reason that a schedule risk analysis provides information on the schedule 

“criticality,” the probability that schedule activities will be on the final critical path.
GAO-09-3SP Chapter 18 223

paths that may become critical, since they can potentially affect program status. A schedule and cost 

risk assessment recognizes the interrelationship between schedule and cost and captures the risk that 

schedule durations and cost estimates may vary because of, among other things, limited data, optimistic 

estimating, technical challenges, lack of qualified personnel, unrealistic durations, poor or inadequate 

logic, overuse of constraints, several parallel paths, multiple merge points, material lead times, and 

external factors (weather, funding); it identifies activities that most affect the finish date. This helps 

management focus on important risk mitigation efforts. As a result, the baseline schedule should include 

a reserve of extra time for contingencies based on the results of a schedule risk analysis. This reserve 

should be held by the project manager and applied as needed to activities that take longer than scheduled 

because of the identified risks. 

To determine the full effect of risks on the schedule, a schedule risk analysis should be conducted to 

determine the level of uncertainty. A schedule risk analysis can help answer three questions that are 

difficult for deterministic critical path method scheduling to address:

1. How likely is it that the program will finish on or before the scheduled completion or baseline 

date?

2. How much schedule reserve time is needed to provide a date that satisfies the stakeholders’ desires 

for certainty?

3. Which activities or risks are the main drivers of schedule risk and the need for schedule reserve? 

This last type of information helps management mitigate schedule risk to improve the chances of finishing 

on time. In addition, an 11-point assessment should be conducted (more detail is in appendix X). 

Risk inherent in a schedule makes it prudent to add in schedule reserve for contingencies—a buffer for the 

schedule baseline. Typically, schedule reserve is calculated by conducting a schedule risk analysis, choosing 

a percentile that represents the organization’s tolerance for overrun risk, and selecting the date that 

provides that degree of certainty. As a general rule, the reserve should be held by the project manager and 

applied as needed to activities that take longer than scheduled because of the identified risks. Reserves of 

time should not be apportioned in advance to any specific activity, since the risks that will actually occur 

and the magnitude of their impact are not known in advance. 

Schedule reserve is a management tool for dealing with risk and should be identified separately in the 

schedule baseline. It is usually defined as an activity at the end of the schedule that has no specific scope 

assigned, since it is not known which risks may materialize. Best practices call for schedule reserve to be 

allocated, based on the results of the schedule risk analysis so that high-risk activities have first priority for 

schedule reserve. 

Once this analysis is done, the schedule should use logic and durations in order to reflect realistic start 

and completion dates for program activities. Maintaining the integrity of the schedule logic is not only 

necessary to reflect true status but is also required before conducting follow-on schedule risk analyses. The 

schedule should avoid logic overrides and artificial constraint dates that are chosen to create a certain result 

on paper. 
224 Chapter 18 GAO-09-3SP

To ensure that the schedule is properly updated, individuals trained in critical path method scheduling 

should be responsible for statusing the schedule. The schedule should be continually monitored to 

determine when forecasted completion dates differ from planned dates, which can be used to determine 

whether schedule variances will affect downstream work. In this analysis, the schedule should be 

monitored and progress reported regularly so that the current status of the activities, total float, and the 

resulting critical path can be determined. Variances between the baseline and current schedule should 

be examined and assessed for impact and significance. Changes to the program scope should also be 

incorporated with the appropriate logic. 

From the analysis, management can make decisions about how best to handle poor schedule performance. 

For example, management could decide to move resources to critical path activities to improve status or 

allocate schedule reserve to immediately address a risk that is turning into an issue. Thus, schedule analysis 

is necessary for monitoring the adequacy of schedule reserve and determining whether the program can 

finish on time. It is also important for identifying problems early, when there is still time to act. 

Estimate Resources and Authorize Budgets

Budgets should be authorized as part of the EVM process, and they must authorize the resources needed 

to do the work. They should not be limited to labor and material costs. All required resources should be 

accounted for, such as the costs for special laboratories, facilities, equipment, and tools. It is imperative that 

staff with the right skills have access to the necessary equipment, facilities, and laboratories. In step 3, we 

discussed how the schedule is resource loaded. This feeds directly into the EVM process and should tie 

back to the cost estimate methodology so it can be considered reasonable. 

Management reserve should be included in the budget to cover uncertainties such as unanticipated effort 

resulting from accidents, errors, technical redirections, or contractor-initiated studies. When a portion 

of the management reserve budget is allocated to one of these issues, it becomes part of the performance 

measurement baseline that is used to measure and control program cost and schedule performance. 

Management reserve provides management with flexibility to quickly allocate budget to mitigate problems 

and control programs. However, it cannot be used to offset or minimize existing cost variances; it can be 

applied only to in-scope work.

Programs with greater risk, such as development programs, usually require higher amounts of 

management reserve than programs with less risk, such as programs in production. The two issues 

associated are how much management reserve should be provided to the program and how will it be 

controlled? Regarding the first issue, research has found that programs typically set their contract value 

so they can set aside 5 to 10 percent as management reserve. This amount may not be sufficient for some 

programs and may be more than others need. The best way to calibrate the amount of management 

reserve needed is to conduct a risk analysis for schedule (to determine the schedule reserve needed) and for 

cost (to determine the management reserve for cost). 

The second issue is very important because if budgets are not spread according to the amount of 

anticipated risk, then control accounts that are overbudgeted will tend to consume all the budget rather 

than return it to management reserve—“budget allocated equals budget spent.” If reserve is not set 

aside for risks further downstream, it tends to get consumed by early development activities, leaving 

inadequacies for later complex activities like integration and testing. 
GAO-09-3SP Chapter 18 225

Experts agree that some form of integration of the program risk management system with the EVM 

system should exist. As a best practice, therefore, management reserve should be linked to a program’s 

risk analysis so that WBS cost elements with the most risk are identified for risk mitigation (figure 21). 

Prioritizing and quantifying total management reserve this way helps ensure that adequate budget is 

available to mitigate the biggest risks that typically occur later in a program. Typically held at a high level, 

the management reserve budget may be controlled directly by the program manager or distributed among 

functional directors or team leaders. In any case, it must be identified and accounted for at all times. 

In addition, the risks from the cost estimate uncertainty analysis should be compared against the 

management reserve allocation. This practice further ties the cost estimating risk analysis with EVM 

(as noted in figure 21). It can also help avoid using management reserve whenever a part of the program 

encounters a problem, ensuring that as more complicated tasks occur later in the program there will still 

be management reserve left if problems arise. When uncertainty analysis is used to specify the probability 

that cost of work will be performed within its budget, then the likelihood of meeting the budget can be 

increased by establishing a sufficient management reserve budget. Using this approach, the probability 

of achieving the budget as a whole can be understood up front. Moreover, using decision analysis tools, 

managers can use the overall probability of success as the basis for allocating budgets for each WBS 

element, increasing their ability to manage the entire program to successful completion. This method also 

allows allocating budget in a way that matches each control account’s expected cost distribution, which is 

imperative for minimizing cost overruns. 

Determine an Objective Measure for Earned Value

Performance measurement is key to earned value because performance represents the value of work 

accomplished. Before any work is started, the control account managers or teams should determine which 

performance measures will be used to objectively determine when work is completed. These measures 

are used to report progress in achieving milestones and should be integrated with technical performance 

measures. Examples of objective measures are requirements traced, reviews successfully completed, 

software units coded satisfactorily, and number of units fully integrated. Table 33 describes several 

acceptable, frequently used methods for determining earned value performance. 

Table 33: Typical Methods for Measuring Earned Value Performance

Method Description Types of tasks using this 

method

Advantages and 

disadvantages

0/100 No performance is taken until a 

task is finished

Take less than 1 month to 

complete

Objective; commonly used 

for quick turnaround as in 

procuring material or brief 

meetings or trips; no partial 

credit is given

50/50, 25/75, etc. Half the earned value is taken 

when the task starts, the other 

half when it is finished; other 

percentage combinations can 

be used

Usually completed within 

2 months

Objective; provides for some 

credit when the task starts
226 Chapter 18 GAO-09-3SP

Method Description Types of tasks using this 

method

Advantages and 

disadvantages

Apportioned 

effort

Effort that by itself is not readily 

divisible into short-span work 

packages but is related in direct 

proportion to measured effort

Historically depend on 

another task that can be 

measured discretely

Provides more objective 

status information than the 

level-of-effort method

Level of effort Performance always equals 

planned cost

Related to the passage 

of time with no physical 

products or defined 

deliverables, such as 

program management

Because performance 

always equals the scheduled 

amount, no schedule 

variances occur; cost 

variances may occur if actual 

costs are higher than planned

Milestone Objective monthly milestones 

are established and the 

assigned budget is divided 

by the value assigned each 

milestone; earned value 

is taken as milestones are 

completed

Work packages exceed 2 

months

Best for accurately and 

objectively measuring 

performance but not always 

practical or possible

Percent 

complete

Performance is equal to the 

percent a task is complete. 

Percent complete should 

be based on underlying, 

quantificable measures as 

much as possible (e.g., number 

of drawings completed) and 

can be measured by the 

statusing of the resourceloaded schedule

Do not have obvious 

interim milestones

If truly based on underlying 

quantifiable measures, 

this method is actually 

the most objective. If that 

is not possible, it can be 

too subjective and a more 

objective method should be 

utilized.

Weighted 

milestone 

Performance is taken as defined 

milestones are accomplished; 

objective milestones (weighted 

by importance) are established 

monthly and the budget is 

divided by milestone weights; 

as milestones are completed, 

value is earned 

Tasks that can be 

planned using interim 

milestones—and the like

Best method for work 

packages that exceed 2 

months; the most accurate 

and objective way to measure 

earned value.

Source: DOD, © 2003 SCEA “Earned Value Management Systems Tracking Cost and Schedule Performance on Projects.”

No one method for measuring earned value status is perfect for every program. Several WBS elements 

may use different methods. What is important is that the method be the most objective approach for 

measuring true progress. Therefore, level of effort should be used sparingly: programs that report using 

a high level of effort for measuring earned value are not providing objective data and the EVM system 

will not perform as expected. As a general rule, if more than 15 percent of a program’s budget is classified 

as level of effort, then the amount should be scrutinized. When level of effort is used excessively for 

measuring status, the program is not really implementing EVM as intended and will fall short of the 

benefits EVM can offer. While the 15 percent benchmark is widely accepted as a trigger point for analysis, 

no given percentage should be interpreted as a hard threshold, because the nature of work on some 

programs and contracts does not always lend itself to more objective measurement.
GAO-09-3SP Chapter 18 227

The other methods provide a more solid means for objectively reporting work status. As work is 

performed, it is earned using the same units as it was planned with, whether dollars, labor hours, or other 

quantifiable units. Therefore, the budget value of the completed work is credited as earned value, which is 

then compared to the actual cost and planned value to determine cost and schedule variances. Figure 27

shows how this works.

Figure 27: Earned Value, Using the Percent Complete Method, Compared to Planned Costs

Tasks J

Detailed schedule

Excavation

Masonry

Plumbing

Electrical

Carpentry

Roofing

Acceptance

Total project

$10

$100

$260

$130

$150

$200

$140

$10

$1,000

100%

100%

25%

0%

0%

0%

0%

0%

$10

$100

$65

$0

$0

$0

$0

$0

$175

1

2

3

4

5

6

7

8

9

10

Legend

Performance status

Milestone/

activity

Value

by task

Task %

complete

Earned

value

Scheduled Complete

F M A M J J

2005

A S O N D

Status

date

Planned value Earned value

(1) Listing of

all tasks

(2) Time

phasing

(3)Budgeted

 value

(4) Task %

 complete

(5) Total

 earned

Source: GAO and Quentin W. Fleming at http://www.quentinf.com.

Figure 27 displays how planned effort is compared with work accomplished. It also shows how earned 

value represents the budgeted value of the work completed and directly relates to the percentage complete 

of each activity. 

When earned value is compared to the planned value for the same work and to its actual cost, 

management has access to program status. This big picture provides management with a better view 

of program risks and better information for understanding what resources are needed to complete the 

program.

Develop the Performance Measurement Baseline 

The performance measurement baseline represents the cumulative value of the planned work over 

time. It takes into account that program activities occur in a sequenced order, based on finite resources, 

with budgets representing those resources spread over time. The performance measurement baseline is 

essentially the resource consumption plan for the program and forms the time-phased baseline against 

which performance is measured. Deviations from the baseline identify areas where management should 

focus attention. 

Figure 28 shows how it integrates cost, schedule, and technical effort into a single baseline.
228 Chapter 18 GAO-09-3SP

Figure 28: The Genesis of the Performance Measurement Baseline

1. Define the work

2. Schedule the work

3. Allocate budgets

40

100

60

15

25

10

30

•

•

Identify statement of work

Extend WBS to control

account/work package

$

Time

PMB

BCWS

BAC

•

•

Arrange work packages

in order

Sequence over time

•

•

•

Budget work packages

Classify the work and select

an earned value technique

Aggregate cumulative BCWS

Contract budget

Management reserve

Source: © 2005 MCR, LLC, “Using Earned Value Data.”

The performance measurement baseline includes all budgets for resources associated with completing the 

program, including direct and indirect labor costs, material costs, and other direct costs associated with 

the authorized work. It represents the formal baseline plan for accomplishing all work in a certain time 

and at a specific cost. It includes any undistributed budget, used as a short-term holding account for new 

work until it has been planned in detail and distributed to a particular control account. To help ensure 

timely performance measurement, it is important that undistributed budget be distributed to specific 

control accounts as soon as practicable. Some sources we reviewed stated that undistributed budget should 

be distributed within 60 to 90 days of acquiring the new funds or authorization.

The performance measurement baseline does not equal the program contract value, because it does not 

include management reserve or any fee. The budget for management reserve is accounted for outside 

the performance measurement baseline, since it cannot be associated with any particular effort until it is 

distributed to a particular control account when a risk occurs and leads to a recovery action. Together, the 

performance measurement baseline and the management reserve represent the contract budget base for the 
GAO-09-3SP Chapter 18 229

program, which in turn represents the total cost of the work. However, fee must be added to the contract 

budget base to reflect the total contract price.

Figure 29 depicts a typical time-phased cumulative performance measurement baseline that typically 

follows the shape of an S curve, portraying a gradual build-up of effort in the beginning, followed by 

stabilization in the middle, and finally a gradual reduction of effort near program completion. The 

management reserve and performance measurement baseline values together make up the contract budget 

base.

Figure 29: The Time-Phased Cumulative Performance Measurement Baseline

Source: © 2003 SCEA, “Earned Value Management Systems.”

Feb Apr Jun Aug Oct Dec Feb Apr

0

500

1,000

1,500

2,000

2,500

$3,000

BCWS

Management reserve

PMB

CBB

Note: BCWS = budgeted cost for work scheduled; CBB = contract budget base; PMB = performance measurement baseline.

Common problems in developing and managing the performance measurement baseline are, first, that it 

may be front-loaded—that is, a disproportionate share of budget has been allocated to early tasks. In this 

case, budget is typically insufficient to cover far-term work. Front-loading tends to hide problems until it 

is too late to correct them. The program can severely overrun in later phases, causing everyone involved to 

lose credibility and putting the program at risk of being canceled. 

Second, the performance measurement baseline can have a rubber baseline—that is, a continual shift of 

the baseline budget to match actual expenditures in order to mask cost variances. Both problems result 

in deceptive baselines by covering up variances early in the program, delaying insight until they are 

difficult if not impossible to mitigate. Third, the performance measurement baseline can become outdated 

if changes are not incorporated quickly. As a result, variances do not reflect reality, and this hampers 

management in realizing the benefits of EVM.
230 Chapter 18 GAO-09-3SP

Execute the Work Plan and Record All Costs

For this step, program personnel execute their tasks according to the performance measurement baseline 

and the underlying detailed work plans. Actual costs are recorded by the accounting system and are 

reconciled with the value of the work performed so that effective performance measurement can occur. 

A program cost-charging structure must be set up before the work begins, to ensure that actual costs 

can be compared with the associated budgets for each active control account. In particular, accounting 

for material costs should be consistent with how the budget was established, to keep variances due to 

accounting accrual issues to a minimum.

Analyze EVM Performance Data and Record Variances from the Performance 

Measurement Baseline Plan

Because programs all carry some degree of risk and uncertainty, cost and schedule variances are normal. 

Variances provide management with essential information on which to assess program performance and 

estimate cost and schedule outcomes. EVM guidelines provide for examining cost and schedule variances 

at the control account level at least monthly and for focusing management attention on variances with 

the most risk to the program. This means that for EVM data to be of any use, they must be regularly 

reviewed. In addition, management must identify solutions for problems early if there is any hope of 

averting degradation of program performance. 

Forecast Estimates at Completion Using EVM

As in step 8, managers should rely on EVM data to generate EACs at least monthly. EACs are 

derived from the cost of work completed along with an estimate of what it will cost to complete all 

unaccomplished work. A best practice is to continually reassess the EAC, obviating the need for periodic 

bottoms-up estimating. It should be noted, however, that DOD requires an annual comprehensive EAC.

Conduct an Integrated Cost-Schedule Risk Analysis

An integrated schedule can be used, in combination with risk analysis data (often including traditional 

3-point estimates of duration) and Monte Carlo simulation software, to estimate schedule risk and the 

EAC. Using the results of the schedule risk analysis, the cost elements that relate to time uncertainty 

(labor, management, rented facilities, escalation) can be linked directly to the uncertainty in the schedule. 

In this approach, the schedule risk analysis provides—in addition to an estimate of when the program may 

finish and the key risk drivers—uncertainty in the schedule activities or summary tasks that relate to timedependent cost elements. These results, which are probability distributions produced by the Monte Carlo 

simulation of the schedule, can be imported to a spreadsheet where cost models and estimates are often 

developed and stored. 

The cost risk analysis uses these schedule risks to link the uncertainty in cost to the uncertainty in 

schedule. This approach models the way labor cost will be determined and converts time to a cost estimate 

by using headcount and labor and overhead rates with any material costs added to the final result. 

(Appendix X has more details on performing a schedule risk analysis.)
GAO-09-3SP Chapter 18 231

Compare EACs from EVM with EAC from Risk Analysis

The integrated cost-schedule risk analysis produces a cumulative probability distribution for the program’s 

cost. This estimate can be compared to the estimate using EVM extrapolation techniques. The reasons to 

compare the two are that they use quite different approaches. 

EVM uses baseline and actual data from the program. The variances are used to estimate future 

performance. Risk analysis uses data that represent the probability that risks will occur and, usually, 

3-point estimates of the risks’ impact on the schedule and cost. These data are projections. Although 

historical data can be used, much of the risk analysis data is derived from interviews and workshops and 

represents expert judgment.

If two methods so different in their approach, models, software, and input data make forecasts of the EAC 

given the current plan, it makes sense to compare their results. If their results are in general agreement, 

their conclusions are probably sound. If not, one or the other method (or both) should be reviewed for 

changes and revisions.

Take Management Action to Mitigate Risk

Management should integrate the results of information from steps 8 through 11 with the program’s 

risk management plan to address and mitigate emerging and existing risks. Management should focus 

on corrective actions and identify ways to manage cost, schedule, and technical scope to meet program 

objectives. It should also keep track of all risks and analyze EVM data trends to identify future problems. 

(Chapter 19 discusses this step further.)

Update the PMB as Changes Occur

Because changes are normal, the ANSI guidelines allow for incorporating changes—unless it is a 

retroactive change to the performance data (with the exception of error correction). However, it is 

imperative that changes be incorporated into the EVM system as soon as possible to maintain the validity 

of the performance measurement baseline. When they occur, both budgets and schedules are reviewed 

and updated so that the EVM data stay current. Furthermore, the EVM system should outline procedures 

for maintaining a log of all changes and for incorporating them into the performance measurement 

baseline, and the log should be maintained so that changes can be tracked. 

Integrated Baseline Reviews

Just as EVM supports risk management by identifying problems when there is still time to act, so an 

IBR helps program managers fully understand the detailed plan to accomplish program objectives and 

identifies risks so they can be included in the risk register and closely monitored. The purposes of the IBR 

are to verify as early as possible whether the performance measurement baseline is realistic and to ensure 

that the contractor and government (or implementing agency) mutually understand program scope, 

schedule, and risks. To do this, the IBR assesses the following risks:

■ Is the technical scope of the work fully included and consistent with authorizing documents?

■ Are key schedule milestones identified and does the schedule reflect a logical flow?
232 Chapter 18 GAO-09-3SP

■ Are resources involving cost—budgets, facilities, skilled staff—adequate and available for 

performing assigned tasks?

■ Are tasks well planned and can they be measured objectively relative to technical progress?

■ Are management processes in place and in use?

OMB requires the government to conduct an IBR for all programs in which EVM is required. While 

agency procedures dictate when the IBR should be conducted, the FAR allows contracting officers the 

option of conducting an IBR before a contract is awarded—this is known as a preaward IBR. Preaward 

IBRs help ensure that cost, schedule, and performance goals have been thoroughly reviewed before the 

contractor is selected.71

Although not mandatory, preaward IBRs verify that a realistic and fully inclusive technical and cost 

baseline has been established. This helps facilitate proposal analysis and negotiation. The benefits from 

doing an IBR (and when appropriate, a pre-award IBR) are that it

■ ensures that both the government and offeror understand the statement of work as stated in the 

contract or request for proposals;

■ allows the government to determine if the offeror’s EVM system complies with agency 

implementation of the ANSI guidelines;

■ ensures that the offeror’s schedule process adequately maintains, tracks, and reports significant 

schedule conditions to the government;

■ assesses the offeror’s risk management plans for the program;

■ assesses the offeror’s business system’s adequacy to maintain program control and report program 

performance objectively; and

■ evaluates the adequacy of available and planned resources to support the program.

Preaward IBRs support confidence in proposal estimates. However, caution must be taken to safeguard 

competition-sensitive or source selection information if multiple offerors are engaged in the competition. 

To lessen the risk of inadvertent disclosure of sensitive information, additional steps such as firewalls may 

be necessary.

Although the pre- and postaward IBRs share the same overall goal, they are noticeably different in 

execution. On the contractor side, a preaward IBR requirement can involve the contract-pricers, marketers, 

and EVM specialists working together to develop the proposal. On the government side, EVM specialists 

and cost analysts become members of the technical evaluation team. However, unlike a traditional IBR, 

the government’s EVM evaluation is limited to the proposal evaluation. Consequently, the government 

EVM or cost analysts cannot conduct control account manager interviews; instead, they submit technical 

71According to OMB, if a preaward IBR is required, it must be included in the proposed evaluation process during the best value 

trade-off analysis. If a preaward IBR was not contemplated at the time of the solicitation, but the source selection team determines 

that the proposals received do not clearly demonstrate that the cost, schedule, and performance goals have a high probability of 

being met, an IBR may be conducted before the award is made.
GAO-09-3SP Chapter 18 233

evaluation questions to the contractors’ equivalent personnel about any issues found in a preaward IBR 

proposal. 

In the preaward IBR, the government reviews the adequacy of the proposed performance measurement 

baseline and how it relates to the integrated master schedule and Integrated Master Plan (IMP) milestones 

and deliverables. In addition, the government reviews the amount of management reserve in relation to the 

risk identified in the proposal. A preaward IBR can also be used to determine potential critical path issues 

by comparing proposed staff levels and costs to these events and associated risks. 

The benefits of conducting a preaward IBR are numerous. First, it provides a new tool to the acquisition 

community that can give insight into contractor performance and management disciplines before 

contract award. Second, it requires the government and contractor to work together to determine the 

reasonableness of a contractor’s proposed baseline. Third, it can allow a contractor to showcase how it 

plans to use EVM to manage the proposed solution. Finally, a preaward IBR forces competing contractors 

to establish high-level baselines that the government can assess for risks before contract award. This 

analysis can help in choosing viable contractors and reducing baseline setup time after contract award.

While there is a cost to both the government and contractor to perform a preaward IBR, the view on 

risks and proposed performance management is worth the extra effort. Subsequently, if a preaward IBR is 

performed, a less-detailed IBR will likely occur after award, resulting in a quicker postaward review. (The 

details of conducting IBRs are discussed in chapter 19.)

Award Fees

Contracts with provisions for award fees allow the government to adjust the fee based on how the 

contractor is performing. The purpose of award fee contracting is to provide motivation to the contractor 

for excellence in such areas as quality, timeliness, technical ingenuity, and cost effective management. 

Before issuing the solicitation, the government establishes the award fee criteria. It is important that 

the criteria be selected to properly motivate the contractor to perform well and encourage improved 

management processes during the award fee period.

It is bad management practice to use EVM measures, such as variances or indexes, as award fee criteria, 

because they put emphasis on the contractor’s meeting a predetermined number instead of achieving 

program outcomes. Award fees tied to reported EVM measures may encourage the contractor to behave in 

undesirable ways, such as overstating performance or changing the baseline budget to “make the number” 

and secure potential profit. These actions undermine the benefits to be gained from the EVM system 

and can result in a loss of program control. For example, contractors may front-load the performance 

measurement baseline or categorize discrete work as level of effort, with the result that variances are 

hidden until the last possible moment. Moreover, tying award fee criteria to specific dates for completing 

contract management milestones, such as the IBR, is also bad practice, because it may encourage the 

contractor to conduct the review before it is ready. 

Best practices indicate that award fee criteria should motivate the contractor to effectively manage its 

contract using EVM to deliver the best product possible. For example, criteria that reward the contractor 
234 Chapter 18 GAO-09-3SP

for integrating EVM with program management, establishing realistic budgets and schedules and 

estimates of costs at completion, providing meaningful variance analysis, performing adequate cost 

control, and providing accurate and timely data represent best practices. In addition, experts agree that 

award fee periods should be tied to specific contract events like preliminary design review rather than 

monthly cycles. (More detail on award fee best practices criteria for EVM is in appendix XIII.)

Progress and Performance-Based Payments Under Fixed-Price Contracts

The principles of EVM are best management practices that are applicable in the administration of certain 

fixed-price type contracts that typically involve non-commercial items. These contracts use performancebased payments or progress payments based on a percentage or stage of completion. Applying relevant 

EVM principles is particularly useful in setting up the progress or performance-based payment structure 

at contract inception and in administering progress payments during contract performance. The informal 

use of EVM principles here does not involve applying the comprehensive “ANSI-compliant EVM” that is 

often used in large cost-reimbursement type contracts where the government faces more risks.

The Federal Acquisition Regulation authorizes progress payments and performance-based payments in 

certain circumstances for fixed-price type contracts for non-commercial items.72 Progress payments are 

based on (1) costs incurred by the contractor as work progresses under the contract or (2) a percentage 

or stage of completion. Progress payments based on a percentage or stage of completion may be used 

as a payment method for work accomplished that meets the quality standards established under the 

contract. Performance-based payments are contract financing payments made on the basis of performance 

measured by objective and quantifiable methods, accomplishment of defined events, or other quantifiable 

measures or results.

The FAR addresses in detail the use of progress payments based on costs incurred. However, it is the 

category of progress payments based on percentage or stage of completion that provides the opportunity 

to apply EVM principles. Specifically, a schedule of values is established between the contractor and the 

government that divides the contract value into quantifiable scope elements. In some cases the contract 

requires that this schedule of values be generated as an output of the resource-loaded, critical path method 

(CPM) schedule, thereby reinforcing the EVM concept of cost and schedule integration. The percent 

complete method, based on either quantifiable units of measure or statused schedule activities can be 

utilized to assess partial progress prior to each scope element’s completion. For this reason, progress 

payments are usually preferred by contractors over milestone-based payments (discussed next), since 

progress they allow for a more favorable cash flow position throughout the project’s execution.

The performance-based payments arrangement also provides opportunities to apply EVM principles. 

Performance-based payments differ from the more traditional progress payments in that they are based 

on the 0/100 or milestone methods as shown in figure 30. Establishing the performance-based payments 

structure requires the government customer and contractor to agree on a set of milestones that will 

become the basis for the performance-based payments. Choosing the milestones usually results in selecting 

critical path activities that lead up to successfully achieving a significant event. This effort requires detailed 

planning to fully identify the work that needs to be accomplished and the relative dollar value of the 

milestones. After the parties have agreed on the performance plan, actual performance is monitored and 

72Federal Acquisition Regulation section 32.102 and subparts 32.5 and 32.10.
GAO-09-3SP Chapter 18 235

payments are made according to the actual achievement of the established milestones. When properly 

planned and implemented, the performance-based payments approach can result in less oversight costs 

for the government, compared to a progress payment arrangement, and enhanced technical and schedule 

focus for the contractor. However, as mentioned above, such an arrangement may not be preferred by the 

contractor because of the impact on cash flow. (See figure 30.)

Figure 30: A Performance-Based Payments Structured Contract

Tasks J

Detailed schedule

Excavation

Masonry

Plumbing

Electrical

Carpentry

Roofing

Acceptance

Total project

$10

$100

$260

$130

$150

$200

$140

$10

$1,000

100%

100%

0%

0%

0%

0%

0%

0%

$10

$100

$0

$0

$0

$0

$0

$0

$110

1

2

3

4

5

6

7

8

9

10

Legend

Performance status

Milestones

Tasks

Value

by M/S

M/S %

complete

Total

payment

ScheduledComplete

F MAM J J

2005

A S O N D

Status

date

(1) Listing of

 milestones

Performance assessment of $110,000 less a 10% withhold = Payment due of $99,000

(2) Time

 phasing

(3)Contract

 values

(4) Status

 complete

(5) Total

 earned

Source: GAO and Quentin W. Fleming at http://www.quentinf.com.

Note: M/S = Milestone.

In the example in figure 30, eight milestones will be used to determine payments. At this point in time, 

two milestones have been met at a cost of $110,000. However, under the performance-based payments 

arrangement, the government would pay the contractor only $99,000, since it will hold back the final 

10 percent until the work is complete (this method of withholding “retainage” is also common practice in 

progress-based payment systems). 

Both progress payment and performance-based payment methods require detailed planning to fully 

identify the work that needs to be accomplished and the relative dollar value of the scope elements or 

milestones. While both are a simplified form of EVM in that physical progress is the basis for payments, 

the government does not have visibility into actual costs borne by the contractor because of the fixedprice nature of the contract. Therefore, care must be taken to ensure that in either method the contractor 

has not “front-loaded” the schedule of values or the performance baseline to increase early payments. In 

addition, progress and milestone events should represent measurable performance in terms of quality 

and technical performance as well as cost and schedule. This is why government review and approval is 

required in both cases at contract inception—the government can thereby guard against paying too much 

for work as it is actually accomplished (which is important should the initial contractor need to be replaced 

and the remaining work re-solicited), while maintaining the protection against paying for any final cost 

overruns that a fixed-price type contract normally provides. By focusing on these issues, government 
236 Chapter 18 GAO-09-3SP

projects performed under fixed-price type contracts have reported improved ability to meet requirements, 

better focus on outcomes, and improved completion time.

Validating the EVM System

If EVM is to be used to manage a program, the contractor’s (and subcontractors’) EVM system should 

be validated to ensure that it complies with the agency’s implementation of the ANSI guidelines, provides 

reliable data for managing the program and reporting its status to the government, and is actively used 

to manage the program. This validation process is commonly referred to as system acceptance. The steps 

involved in the system acceptance process are shown in figure 31. Sometimes these steps may overlap 

rather than go in sequence because of resource or capability constraints between the EVM system owner, 

the government customer, or both. However, all steps leading up to actual acceptance must be addressed 

for an EVM system owner or agency program to implement an ANSI-compliant EVM system.73

Figure 31: The EVM System Acceptance Process

Establish

EVM

policy

Establish

EVM

system

Implement

EVM

program

Comment on

assessment

report

Prepare

assessment

report

Accept

EVM system

compliance

Issue EVMS

letter of

acceptance

Issue EVMS

advance

agreement

Conduct

compliance evaluation

review (CER)

Submit

compliance evaluation

review report

Develop surveillance

and system revision

procedures

Implement surveillance

and system revision

procedures

Process

steps

Design and

implementation Review Assessment Acceptance

OR Acceptance

authority

Compliance

evaluation

team

EVM

system

owner

Source: Copyright 2004–2005 National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC).

The system acceptance process has four phases. In system design and implementation, establishing 

the EVM policy (which includes documented processes and procedures) is followed by developing 

and implementing an EVM system. Once complete, the compliance evaluation review can begin. The 

purpose of this review is to verify that the EVM system meets the ANSI guidelines and has been fully 

73More information on EVM system acceptance is in NDIA, Program Management Systems Committee, “NDIA PMSC ANSI/

EIA 748 Earned Value Management System Acceptance Guide,” draft, working release for user comment (Arlington, Va.: 

November 2006).
GAO-09-3SP Chapter 18 237

implemented on selected contracts, projects, or programs. Data traces are necessary for verifying that 

lower-level reporting aligns with higher levels and that the data provide accurate management information. 

Interviews verify that the EVM system is fully implemented and actively used to manage the program. 

Additionally, the compliance review process and its results should be documented. 

The compliance evaluation review is an independent review conducted by an individual or organization 

that

■ has no stake in the EVM system, project, or contract being reviewed;74

■ has the knowledge, skills, and abilities to fairly evaluate the fitness of the EVM system’s 

implementation or surveillance; and

■ relies on the NDIA EVMS intent guide to determine whether the EVM system is compliant with 

the ANSI guidelines.

Upon successful completion of EVM system acceptance, an acceptance recognition document should 

be prepared and released. When cross-agency acceptance occurs, this is best accomplished by mutual 

agreements between agencies and organizations to recognize EVM system ANSI compliant acceptance or 

recognition documents. 

An agency can accept another organization’s EVMS acceptance with the understanding that it will need 

to instill a rigorous surveillance process (see chapter 20) to ensure that the written system description 

meets the intent of the 32 guidelines and is actively being followed. An alternative acceptance procedure 

is for a partner agency (or cross-agency) to review the documentation from the EVM system owner’s 

compliance evaluation review. 

When no independent entity exists to perform EVM acceptance, the assessment may be performed 

by a qualified source that is independent from the program’s development, implementation, and direct 

supervision—for example, an agency’s inspector general. Moreover, civilian agencies may negotiate an 

interagency agreement to conduct acceptance reviews to satisfy the criteria for independence. For this 

arrangement to succeed, staff trained in EVM system reviews are required, and these resources are scarce 

in the government. 

Best practices call for centers of excellence that include staff who are experienced in EVM system design, 

implementation, and validation and have a strong knowledge of ANSI guidelines. In addition, these 

staff should have good evaluation skills, including the ability to review and understand EVM data and 

processes and the ability to interview personnel responsible for the EVM system implementation to 

determine how well they understand their own system description and processes. 

Case studies 44 and 45 highlight what can happen to a program when an EVM system has not been 

validated as being compliant with the ANSI guidelines.

74This criterion does not apply when the EVM system owner conducts a self-evaluation review.
238 Chapter 18 GAO-09-3SP

Case Study 44: Validating the EVM System, from Cooperative Threat 

Reduction, GAO-06-692

In September 2004, DOD modified its contract with Parsons Global Services, allocating 

about $6.7 million and requiring the company to apply EVM to the Shchuch’ye project. 

Parsons was expected to have a validated EVM system by March 2005, but by April 2006, 

it had not yet developed an EVM system that provided useful and accurate data to the 

chemical weapons destruction facility’s program managers. In addition, GAO found that 

the project’s EVM data were unreliable and inaccurate: in numerous instances, data had 

not been added properly for scheduled work. Parsons’ EVM reports, therefore, did not 

accurately capture data that project management needed to make informed decisions 

about the Shchuch’ye facility. 

For example, Parsons’ EVM reports from September 2005 through January 2006 contained 

errors in addition that did not capture almost $29 million in actual project costs. Such 

omissions and other errors may have caused DOD and Parsons project officials to 

overestimate the available project funding. GAO also found several instances in which 

the accounting data were not allocated to the correct cost accounts, causing large cost 

over- and underruns. Accounting data had been placed in the wrong account or Parsons’ 

accounting system was unable to track costs at all levels of detail within EVM.

GAO concluded that until Parsons fixed its accounting system, manual adjustments would 

have to be made monthly to ensure that costs were properly aligned with the correct 

budget. Such adjustments meant that the system would consistently reflect inaccurate 

project status for Parsons and DOD managers. Parsons’ outdated accounting system had 

difficulty capturing actual costs for the Shchuch’ye project and placing them in appropriate 

cost categories. Parsons management should have discovered such accounting errors 

before the EVM report was released to DOD. 

The Defense Contract Audit Agency therefore questioned whether Parsons could generate 

correct accounting data and recommended that it update its accounting system. DOD 

expected Parsons to use EVM to estimate cost and schedule impacts and their causes and, 

most importantly, to help eliminate or mitigate identified risks. GAO recommended that 

DOD ensure that Parsons’ EVM system contained valid, reliable data and that the system 

reflect actual cost and schedule conditions. GAO also recommended that DOD withhold a 

portion of Parsons’ award fee until the EVM system produced reliable data.

However, before GAO issued its report, Parsons had begun to improve its EVM processes 

and procedures. It had established a new functional lead position to focus on cost 

management requirements in support of government contracts. In addition, Parsons 

installed a new EVM focal point to address the lack of progress made in achieving 

validation of the EVM system for the Shchuch’ye project. 

Immediately after GAO’s report was issued, Parsons’ new EVM focal point was able to 

identify and correct the system problems that had led to the unreliable and inaccurate 

EVM data. The new focal point also found that the data integrity problems GAO had 

identified were not directly related to a need to update Parsons’ accounting system. 

First, the project’s work breakdown structure had not been developed to the level of 

detail required to support a validated EVM system before Parsons received the contract 

modification to implement the system, and the project’s original cost management
GAO-09-3SP Chapter 18 239

practices, policies, and procedures had not been robust enough to effectively prevent the 

historical miscoding of actual costs against the existing WBS. Second, the more recent data 

quality issues GAO cited resulted from the lack of a reconcilable means of downloading 

actual cost information from Parsons’ accounting system into a cost processor that had not 

yet been optimized. 

Parsons’ accounting system was deemed adequate in an August 2006 Defense Contract 

Audit Agency audit report. DOD chose not to withhold Parsons’ award fee, given the 

progress being made toward improving the data integrity issues GAO had identified. The 

Shchuch’ye project’s EVM system was formally validated in a May 2007 Defense Contract 

Management Agency letter. 

GAO, Cooperative Threat Reduction: DOD Needs More Reliable Data to Better 

Estimate the Cost and Schedule of the Shchuch’ye Facility, GAO-06-692

(Washington, D.C.: May 31, 2006).

Case Study 45: Validating the EVM System, from DOD Systems 

Modernization, GAO-06-215

The Naval Tactical Command Support System (NTCSS) elected to use EVM, but Navy and 

DOD oversight authorities did not have access to the reliable and timely information they 

needed to make informed decisions. The EVM system that NTCSS implemented to measure 

program performance did not provide data for effectively identifying and mitigating 

risks. According to the NTCSS central design agency’s self-assessment of its EVM system, 

17 of industry’s 32 best practices criteria were not being met. GAO also found 29 of the 32 

criteria were not satisfied.

Two NTCSS projects for which EVM activities were reportedly being performed were 2004 

Optimized Organizational Maintenance Activity (OOMA) software development and 2004 

NTCSS hardware installation and integration. GAO found several examples of ineffective 

EVM implementation on both projects.

The estimate at completion for the 2004 OOMA software project—a forecast value 

expressed in dollars representing final projected costs when all work was completed—

showed a negative cost for the 6 months November 2003 to April 2004. If EVM had been 

properly implemented, this amount, which is always a positive number, should have 

included all work completed.

The cost performance index for the OOMA software project—which was to reflect 

the critical relationship between the actual work performed and the money spent to 

accomplish the work—showed program performance during a time when the program 

office stated that no work was being performed. 

The estimate at completion for the OOMA hardware installation project showed that 

almost $1 million in installation costs had been removed from the total sunk costs, but no 

reason for doing so was given in the cost performance report.

The cost and schedule indexes for the OOMA hardware installation project showed 

improbably high program performance when the installation schedules and installation 

budget had been drastically cut because OOMA software failed operational testing.
240 Chapter 18 GAO-09-3SP

GAO concluded that because EVM was ineffectively implemented in these two projects, 

NTCSS program officials did not have access to reliable and timely information about 

program status or a sound basis for making informed program decisions. Therefore, GAO 

recommended that the NTCSS program implement effective program management 

activities, including EVM.

GAO, DOD Systems Modernization: Planned Investment in the Naval 

Tactical Command Support System Needs to Be Reassessed, GAO-06-215

(Washington, D.C.: Dec. 5, 2005).

15. Best Practices Checklist: Managing Program Costs: Planning

 A cost estimate was used to measure performance against the original plan, 

using EVM. 

 EVM and risk management were tightly integrated to ensure better program 

outcomes.

3 Strong leadership demands EVM be used to manage programs. 

3 Stakeholders make it clear that EVM matters and hold staff accountable 

for results.

3 Management is willing to hear the truth about programs and relies on 

EVM data to make decisions on how to mitigate risk.

3 Policy outlines clear expectations for EVM as a disciplined management 

tool and requires pertinent staff to be continuously trained in cost 

estimating, scheduling, EVM, and risk and uncertainty analysis.

 EVM is implemented at the program level so that both government and 

contractor know what is expected and are held accountable.

3 EVM relied on the cost of completed work to determine true program 

status.

3 EVM planned all work to an appropriate level of detail from the 

beginning. 

3 It measured the performance of completed work with objective 

techniques. 

3 It used past performance to predict future outcomes. 

3 It integrated cost, schedule, and performance with a single 

management control system.

3 It directed management to the most critical problems, reducing 

information overload.

3 It fostered accountability between workers and management. 

 The EVM system complied with the agency’s implementation of ANSI’s 32 

guidelines.
GAO-09-3SP Chapter 18 241

 The following steps in the EVM process were taken: 

3 The work’s scope was defined with a WBS, and effort was broken into 

work and planning packages.

3 The WBS and organizational breakdown structure were cross-walked to 

identify control accounts that show who will do the work. 

3 An acceptable technique was used to schedule work to resource load 

activities.

ù All activities were identified and sequenced, logically networked, 

clearly showing horizontal and vertical integration. 

ù Activities were resource loaded with labor, material, and overhead 

and durations were estimated with historical data when available, 

and float was identified.

ù Program master schedule and critical path were identified.

ù A schedule risk analysis was performed based on an 11-point 

schedule assessment.

ù Schedule reserve was chosen and prioritized for high-risk activities.

ù The schedule was updated using logic and durations to determine 

dates and reflects accomplishments and is continuously analyzed 

for variances and changes to the critical path and completion date. 

3 Resources were adequate to complete each activity and were estimated 

to do the work, authorize budgets, and identify management reserve 

for high-risk efforts.

3 Objective methods for determining earned value were used.

3 The performance measurement baseline was developed for assessing 

program performance; EVM performance data were analyzed and 

variances from the baseline plan were recorded; the performance 

measurement baseline was updated.

3 EACs were forecast using EVM.

3 An integrated cost-schedule risk analysis was conducted.

3 EACs from EVM were compared with an EAC from risk analysis.

3 Management took action to mitigate risk.

3 A preaward IBR was performed where provided for to verify the 

performance measurement baseline’s realism and compliance with ANSI 

guidelines.

3 Award fee criteria were developed to motivate the contractor to 

manage its contract with EVM to deliver the best possible product, were 

tied to specific contract events, and did not predetermine specific EVM 

measures.

3 A performance based payment contract was considered for fixed-price 

contracts where technical effort and risk are low.

3 The EVM system implemented was validated for compliance with the 

ANSI guidelines by independent and qualified staff and therefore can be 

considered to provide reliable and valid data from which to manage the 

program.
242 Chapter 18 GAO-09-3SP
GAO-09-3SP Chapter 19 243

Managing Program Costs: Execution

Chapter 19

Studies of more than 700 defense programs have shown limited opportunity for getting a wayward 

program back on track once it is more than 15 percent to 20 percent complete.75 EVM data allow 

wayward management to quickly track deviations from a program’s plan for prompt understanding of 

problems. Proactive management results in better focus and increases the chance that a program will 

achieve its goals on time and within the expected cost. 

To rely on EVM data, an IBR must be conducted to ensure that the performance measurement baseline 

accurately captures all the work to be accomplished. Data from the CPR can then be used to assess 

program status—typically, monthly. Cost and schedule variances are examined and various estimates at 

completion are developed and compared to available funding. The results are shared with management for 

evaluating contractor performance. Finally, because EVM requires detailed planning for near-term work, 

as time progresses, planning packages are converted into detailed work packages. This cycle continues 

until all work has been planned and the program is complete. 

Validating the Performance Measurement Baseline with an IBR

An IBR is an evaluation of the performance measurement baseline to determine whether all program 

requirements have been addressed, risks identified, and mitigation plans put in place and all available and 

planned resources are sufficient to complete the work. Too often, programs overrun because estimates 

fail to account for the full technical definition, unexpected changes, and risks. Using poor estimates 

to develop the performance measurement baseline will result is an unrealistic baseline for performance 

measurement. 

The IBR concept to ensure comprehensive baselines for managing programs was developed in 1993 as a 

best practice after numerous DOD programs experienced significant cost and schedule overruns because 

their baselines were too optimistic. An IBR’s goal is to verify that the technical baseline’s budget and 

schedule are adequate for performing the work. Key benefits are that 

■ it lays a solid foundation for successfully executing the program,

■ it gives the program manager and contractor mutual understanding of the risks,

■ the program manager knows what to expect at the outset of the program,

■ planning assumptions and resource constraints are understood,

■ errors or omissions in the baseline plan can be corrected early in the program,

75The source of this statement is © 2003, Society of Cost Estimating and Analysis, “Earned Value Management Systems (EVMS) 

Tracking Cost and Schedule Performance on Projects,” p. 7.
244 Chapter 19 GAO-09-3SP

■ developing variances can be discovered sooner, and

■ resources for specific challenges and risks can be identified.

Conducting an IBR increases everyone’s confidence that the performance measurement baseline provides 

reliable cost and schedule data for managing the program and that it projects accurate estimated costs at 

completion. OMB has endorsed the IBR as a critical process for risk management on major investments 

and requires agencies to conduct IBRs for all contracts that require EVM. 

The IBR is the crucial link between cost estimating and EVM because it verifies that the cost estimate has 

been converted into an executable program plan. While the cost estimate provides an expectation of what 

could be, based on a technical description and assumptions, the baseline converts that expectation into a 

specific plan for achieving the desired outcome. Once the baseline is established, the IBR assesses whether 

its estimates are reasonable and risks have been clearly identified.

OMB directs agencies to conduct IBRs in accordance with DOD’s Program Manager’s Guide to the 

Integrated Baseline Review Process, which outlines four activities to be jointly executed by the program 

manager and contractor staff: performance measurement baseline development, IBR preparation, IBR 

execution, and management processes.76

Experts agree that it a best practice for the government and prime contractor to partner in conducting an 

IBR on every major subcontractor in conjunction with the prime contractor IBR. This practice cannot be 

emphasized enough, especially given that many major systems acquisitions are systems of systems with the 

prime contractor acting as the main integrator. The expert community has seen up to 60 to 70 percent of 

work being subcontracted out. Pair this risk with the lack of focus on systems engineering, and many risks 

may go unnoticed until they are realized. Furthermore, the increasing roles and responsibilities assumed 

by subcontractors in these contracts make the accuracy of subcontractor EVM data that much more 

important.

Performance Measurement Baseline Development 

As the principal element of EVM, the performance measurement baseline represents the time-phased 

budget plan against which program performance is measured for the life of the program. This plan 

comes from the total roll-up of work that has been planned in detail through control accounts, summary 

planning packages, and work packages with their schedules and budgets. 

Performance measurement baseline development examines whether the control accounts encompass 

all contract requirements and are reasonable, given the risks. To accomplish this, the government and 

contractor management teams meet to understand whether the program plan reflects reality. They ask,

■ Have all tasks in the statement of work been accounted for in the baseline?

■ Are adequate staff and materials available to complete the work?

■ Have all tasks been integrated, using a well-defined schedule?

76 See DOD, The Program Manager’s Guide to the Integrated Baseline Review Process (Washington, D.C.: Office of the Secretary of 

Defense (AT&L), April 2003).
GAO-09-3SP Chapter 19 245

Since it is not always feasible for the IBR team to review every control account, the team often samples 

control accounts to review. To ensure a comprehensive and value-added review, teams can consider

■ medium to high technical risk control accounts,

■ moderate to high dollar value control accounts,

■ critical path activities,

■ elements identified in the program risk management plan, and

■ significant material subcontracts and non-firm-fixed-price subcontracts.

The IBR team should ask the contractor for a list of all performance budgets in the contract. The 

contractor can typically provide a matrix of all control accounts, their managers, and approved budget 

amounts. Often called a dollarized responsibility assignment matrix, it is a valuable tool in selecting 

control accounts that represent the most risk.

At the end of the IBR, the team’s findings inform the program’s risk management plan and should give 

confidence in the quality of the contractor’s performance reports. If no IBR is conducted, confidence is 

less that monthly EVM reporting will be meaningful or accurate.

IBR Preparation

An IBR is most effective if the focus is on areas of greatest risk to the program. Government and 

contractor program managers should try for mutual understanding of risks and formulate a plan to 

mitigate and track them through the EVM and risk management processes. In addition, developing 

cooperation promotes communication and increases the chance for effectively managing and containing 

program risks.

Depending on the program, the time and effort in preparing for the IBR varies. Specific activities include

■ identifying program scope to review, including appropriate control accounts, and associated 

documentation needs;

■ identifying the size, responsibilities, and experience of the IBR team;

■ program management planning, such as providing training, obtaining required technical expertise, 

and scheduling review dates;

■ classifying risks by severity and developing risk evaluation criteria; and 

■ developing an approach for conveying and summarizing findings.

Program managers should develop a plan for conducting the review by first defining the areas of the 

program scope the team will review. To do this, they should be familiar with the contract statement of 

work and request the appropriate documents, including the LCCE and program risk assessment, to decide 

areas that have the most risk. They should also have a clear understanding of management processes that 

will be used to support the program, including how subcontractors will be managed.

Each IBR requires participation from specific program, technical, and schedule experts. Staff from a 

variety of disciplines—program management, systems engineering, software engineering, manufacturing, 

integration and testing, logistics support—should assist in the review. In addition, experts in functional 
246 Chapter 19 GAO-09-3SP

areas like cost estimating, schedule analysis, EVM, and contracting should be members of the team. In 

particular, EVM specialists and contract management personnel should be active participants. The IBR 

team may at times also include subcontractor personnel. The team’s size should be determined by the 

program’s complexity and the risk associated with achieving its objectives. 

While IBRs have traditionally been conducted by government program offices and their contractors, 

OMB guidance anticipates that EVM will be applied at the program level. Therefore, program-level IBR 

teams should include participants from other stakeholder organizations, such as the program’s business 

unit, the agency’s EVM staff, and others, as appropriate.

Team members must have appropriate training before the IBR is conducted to ensure that they can 

correctly identify and assess program risks. Team members should be trained so they understand the cost, 

schedule, and technical aspects of the performance measurement baseline and the processes that will be 

used to manage them.

As we stated earlier, identifying potential program risk is the main goal of an IBR. Risks are generally 

categorized as cost, management process, resource, schedule, and technical (table 34).

Table 34: Integrated Baseline Review Risk Categories

Category Definition

Cost Evaluates whether the program can succeed within budget, resource, and schedule 

constraints as depicted in the performance measurement baseline; cost risk is driven by the 

quality and reasonableness of the cost and schedule estimates, accuracy of assumptions, use 

of historical data, and whether the baseline covers all efforts outlined in the statement of 

work

Management 

process

Evaluates how well management processes provide effective and integrated technical, 

schedule, cost planning, and baseline change control; it examines whether management 

processes are being implemented in accordance with the EVM system description. 

Management process risk is driven by the need for early view into risks, which can be 

hampered by inability to establish and maintain valid, accurate, and timely performance data, 

including subcontractors’ data 

Resource Represents risk associated with the availability of personnel, facilities, and equipment needed 

to perform program-specific tasks; includes staff lacking because of other company priorities, 

unexpected downtime precluding or limiting the use of specific equipment or facilities when 

needed

Schedule Addresses whether all work scope has been captured in the schedule and time allocated to 

lower-level tasks meets the program schedule; schedule risk is driven by the interdependency 

of scheduled activities and logic and the ability to identify and maintain the critical path

Technical Represents the reasonableness of the technical plan for achieving the program’s objectives 

and requirements; deals with issues such as the availability of technology, capability of the 

software development team, technology, and design maturity

Source: Adapted from DOD, The Program Manager’s Guide to the Integrated Baseline Review Process (Washington, D.C.: Office of the Secretary of Defense (AT&L), April 2003).

Program managers should also outline the criteria for evaluating risks in table 34 and develop a method 

for tracking them within the risk management process. In addition, they should monitor the progress of 

all risks identified in the IBR and develop action plans for resolving them.
GAO-09-3SP Chapter 19 247

IBR Execution

Because an IBR provides a mutual understanding of the performance measurement baseline and its 

associated risk, identifying potential problems early allows for developing a plan for resolving and 

mitigating them. Thus, the IBR should be initiated as early as possible—before award, when appropriate, 

and no later than 6 months after. To be most effective, maturity indicators should be assessed to ensure 

that a value-added assessment of the performance measurement baseline can be accomplished:

1. Work definition: 

■ a WBS should be developed;

■ specifications should flow down to subcontractors;

■ internal statement of work for work package definitions should be defined.

2. Integrated schedule: 

■ lowest and master level should be vertically integrated;

■ tasks should be horizontally integrated;

■ product handoffs should be identified;

■ subcontractor schedules should be integrated with the prime master schedule.

3. Resources, labor, and material should be fully planned and scheduled;

■ constrained resources should be identified or rescheduled; 

■ staffing resources should be leveled off;

■ subcontractor baselines should be integrated with the prime baseline;

■ schedule and budget baselines should be integrated;

■ work package earned value measures should be defined;

■ the baseline should be validated at the lowest levels and approved by management.

The absence of maturity indicators is itself an indication of risk. An IBR should not be postponed 

indefinitely; it should begin, with a small team, as soon as possible to clarify plans for program execution. 

In executing the IBR, the team assesses the adequacy, realism, and risks of the baseline by examining if

■ the technical scope of work is fully included (an allowance for rework and retesting is considered),

■ key schedule milestones are identified,

■ supporting schedules reflect a logical flow to accomplish tasks,

■ the duration of each task is realistic and the network schedule logic is accurate,

■ the program’s critical path is identified,

■ resources—budgets, facilities, personnel, skills—are available and sufficient for accomplishing tasks,

■ tasks are planned to be objectively measured for technical progress,

■ the rationale supporting performance measurement baseline control accounts is reasonable, and

■ managers have appropriately implemented required management processes.
248 Chapter 19 GAO-09-3SP

After it has been determined that the program is defined at an appropriate level, interviewing control account 

managers is the next key IBR objective. Interviews should focus on areas of significant risk and management 

processes that may affect the ability to monitor risks. Discussions should take place among a small group of 

people, addressing how the baseline was developed and the supporting documentation. If the contractor has 

reasonably developed an integrated baseline, preparing for the IBR should require minimal time. 

During the interview process, the IBR team meets with specific control account managers to understand 

how well they use EVM to manage their work and whether they have expertise in their area of discipline. 

Typical discussion questions involve how the control account managers receive work authorization, how 

they ensure that the technical content of their effort is covered, and how they use the schedule to plan and 

manage their work. In addition, interviews are an excellent way to determine whether a control account 

manager needs additional training in EVM or lacks appropriate resources. A template gives interviewers a 

consistent guide to make sure they cover all aspects of the IBR objectives. Figure 32 is a sample template.

Figure 32: IBR Control Account Manager Discussion Template

Step 5 Document. Complete control account risk evaluation sheet, reach concurrence on risk and action items. 10 minutes

Step 4 Evaluate baseline for each work package 90 minutes

Step 3 Describe control account or work packages, briefly describe performance to date 5 minutes

Step 1 Introductions 5 minutes

Baseline discussion starter

Step 2 Overview of control accounts

 General description, work content

5 minutes

No. Title Budget at completion % complete BCWP method Discuss?

Work scope

Documents to review

All work included?

Clear work description?

Risk mitigation?

Technical risk?

Trace from scope of work to

WBS to control account or

work package descriptions

Statement of work, contractor

WBS dictionary, work package

descriptions, risk plans

BCWP method

Documents to review

Objective measures of work?

Level of effort minimized?

Subcontractor performance?

Milestones defined?

Method for calculating

percentage complete?

Control account plan, back-up

worksheets for BCWP,

subcontractor reports

Budget

Documents to review

Basis for estimate?

Management challenges?

Realistic budget? (focus on

hours)

Phasing?

Developing

cost variance?

Variance at complete?

Budget risk?

Control account plan, basis of

estimate, variance reports,

purchase order for material

Schedule

Documents to review

Realistic? Complete?

Subcontractors?

Task durations? Network logic?

Handoffs? Vertical and

horizontal integration? Critical

path? Concurrence?

Developing schedule variance?

Completion variance from

schedule? Budget risk?

IMS, work package schedules,

staffing plans

Source: DCMA.
GAO-09-3SP Chapter 19 249

After completing the IBR, the program managers assess whether they have achieved its purpose—they 

report on their understanding of the performance measurement baseline and their plan of action for 

handling risks. They should develop a closure plan that assigns staff responsibility for each risk identified 

in the IBR. Significant risks should then be included in the program’s risk management plan, while lowerlevel risks are monitored by responsible individuals. An overall program risk summary should list each 

risk by category and severity in order to determine a final risk rating for the program. This risk assessment 

should be presented to senior management—government and contractors—to promote awareness.

The IBR team should document how earned value will be assessed and whether the measurements are 

objective and reasonable. It should discuss whether management reserve will cover new risks identified in 

the IBR. Finally, if the team found deficiencies in the EVM system, it should record them in a corrective 

action request and ask the EVM specialist to monitor their status.

Although a formal IBR report is not usually required, a memorandum for the record describing the 

findings with all backup documentation should be retained in the official program management files. 

And, while the IBR is not marked with an official pass or fail, a determination should be made about 

whether the performance measurement baseline is reliable and accurate for measuring true performance.

Management Processes

When the IBR is complete, the focus should be on the ongoing ability of management processes to reveal 

actual program performance and detect program risks. The IBR risk matrix and risk management plan 

should give management a better understanding of risks facing the program, allowing them to manage 

and control cost and schedule impacts. The following management process should continue after the IBR 

is finished: 

■ the baseline maintenance process should continue to ensure that the performance measurement 

baseline reflects a current depiction of the plan to complete remaining work and follows a 

disciplined process for incorporating changes, and

■ the risk management process should continue to document and classify risks according to the 

probability that they will occur, their consequences, and their handling.

Other typical business processes that should continue to support the management of the program involve 

activities like scheduling, developing estimates to complete, and EVM analysis so that risks may be 

monitored and detected throughout the life of the program. (Appendix XIV has a case study example on 

IBRs.)

Contract Performance Reports

The IBR completed and the PMB validated, now EVM data can be used to assess performance and 

project costs at completion. EVM data are typically summarized in a standard CPR. This report becomes 

the primary source for program cost and schedule status and provides the information needed for 

effective program control. The CPR provides cost and schedule variances, based on actual performance 

against the plan, which can be further examined to understand the causes of any differences. 

Management can rely on these data to make decisions regarding next steps. For example, if a variance 

stems from an incorrect assumption in the program cost estimate, management may decide to obtain 

more funding or reduce the scope. 
250 Chapter 19 GAO-09-3SP

Reviewing CPR data regularly helps track program progress, risks, and plans for activities. When variances 

are discovered, CPR data identify where the problems are and the degree of their impact on the program. 

Therefore, the ANSI guidelines specify that, at least monthly, cost and schedule variance data should be 

generated by the EVM system to give a view into causes and allow action. Since management may not be 

able to review every control account, relying on CPR data enables management to quickly assess problems 

and focus on the most important issues.

CPR data come from monthly assessment of and reports on control accounts. Control account managers 

summarize the data to answer the following questions: 

■ How much work should have been completed by now—or what is the planned value or BCWS?

■ How much work has been done—or what is the earned value or BCWP?

■ How much has the completed work cost—or what is the actual cost or ACWP?

■ What is the planned total program cost—or what is the BAC?

■ What is the program expected to cost, given what has been accomplished—or what is the EAC?

Figure 33 is an example of this type of monthly assessment. It shows that the performance measurement 

baseline is calculated by summarizing the individual planned costs (BCWS) for all control accounts 

scheduled to occur each month. Earned value (BCWP) is represented by the amount of work completed 

for each active control account. Finally, actual costs (ACWP) represent what was spent to accomplish the 

completed work.

Figure 33: Monthly Program Assessment Using Earned Value

Task description J

Concrete

Framing

Roofing

Electrical

Plumbing

Interior

Monthly budget

Cum budget (PMB)

Earned value (BCWP)

Actual cost (ACWP)

$10,000

20,000

15,000

40,000

35,000

35,000

100%

60

30

$10,000

12,000

5,000

$27,000

Budgeted

%

F M A M J J A S O N D Complete Earned

$3,000 $10,000 $13,000 $13,000 $16,000 $15,000 $21,000 $12,000 $12,000 $13,000 $12,000 $15,000

3,000 13,000 26,000 39,000 55,000 70,000 91,000 103,000 115,000 128,000 140,000 155,000

1,000 5,000 15,000 27,000

2,000 7,000 19,000 33,000

3,000 5,000 2,000

5,000 10,000 5,000

1,000 8,000 6,000

10,000 15,000 15,000

6,000 12,000 12,000 5,000

8,000 12,000 15,000

Source: Naval Air Systems Command (NAVAIR).

According to the data in figure 33, by the end of April the control account for concrete has been 

completed, while the framing and roofing control accounts are only partially done—60 percent and 

30 percent complete, respectively. Examining what was expected to be done by the end of April—$39,000 

worth of work—with what was actually accomplished—$27,000 worth of work—one can determine 

that $12,000 worth of work is behind schedule. Likewise, by assessing what was accomplished—$27,000 

worth of work—with what was spent—$33,000—one can see that the completed work cost $6,000 more 

than planned. These data can also be graphed to quickly obtain an overall program view, as in figure 34.
GAO-09-3SP Chapter 19 251

Figure 34: Overall Program View of EVM Data 

Feb Apr Jun Aug Oct

Time

now

Dec Feb Apr Jun Aug

0

500

1,000

1,500

2,000

2,500

$3,000

Schedule

variance

BCWS

PMB

CBB Variance at

completion

BAC

EAC

Schedule

delay

Cost

variance

Management reserve

ACWP

BCWP

Source: © 2003 SCEA, “Earned Value Management Systems.”

Note: ACWP = actual cost of work performed; BAC = budget at completion; BCWP = budgeted cost for work performed; BCWS = 

budgeted cost for work scheduled; CBB = contract budget baseline; EAC = estimate at completion; PMB = performance measurement 

baseline.

Figure 34 shows that in October, the program is both behind schedule and overrunning cost. The EAC 

shows projected performance and expected costs at completion. Cost variance is calculated by taking the 

difference between completed work (BCWP) and its cost (ACWP), while schedule variance is calculated 

by taking the difference between completed work (BCWP) and planned work (BCWS). Positive variances 

indicate that the program is either underrunning cost or performing more work than planned. Conversely, 

negative variances indicate that the program is either overrunning cost or performing less work than 

planned. 

It is important to understand that variances are neither good nor bad. They are merely measures that 

indicate that work is not being performed according to plan and that it must be assessed further to 

understand why. From this performance information, various estimates at completion can be calculated. 

The difference between the EAC and the budget at completion (BAC) is the variance at completion, 

which represents either a final cost overrun or an underrun.

Management should use the EVM data captured by the CPR data to (1) integrate cost and schedule 

performance data with technical performance measures, (2) identify the magnitude and impact of actual 

and potential problem areas causing significant cost and schedule variances, and (3) provide valid and 

timely program status to higher management. As a management report, the CPR provides timely, reliable 

summary EVM data with which to assess current and projected contract performance. 
252 Chapter 19 GAO-09-3SP

The primary value of the report is its ability to reflect current contract status and reasonably project future 

program performance. When the data are reliable, the report can facilitate informed, timely decisions by a 

variety of program staff—engineers, cost estimators, and financial management personnel, among others. 

CPR data are also used to confirm, quantify, and track known or emerging problems and to communicate 

with the contractor. As long as the CPR data accurately reflect how work is being planned, performed, 

and measured, they can be relied on for analyzing actual program status. The five formats within a CPR 

are outlined in figure 35. 

Figure 35: A Contract Performance Report’s Five Formats

Staff loading

Format 4

Work breakdown

structure

•

•

Format 1

WBS level 2

WBS level 3

Explanation of

variances

Identify:

•

•

•

•

•

•

•

•

Format 5

Nature of the problem

Reason for cost or

schedule variance

Impact on total program

Corrective action taken

Amounts attibuted to

rate changes

Undistributed budget

application

Management reserve

application

Baseline changes

Baseline

•

•

•

Format 3

Changes

Undistributed budget

Management reserve

Functional categories

Format 2

Source: Naval Air Systems Command (NAVAIR).

All five formats in a CPR should be tailored to ensure that only information essential to management on 

cost and schedule is required from contractors. Format 1 provides cost and schedule data for each element 

in the program’s product-oriented WBS—typically, hardware, software, and other services necessary for 

completing the program. Data in this format are usually reported to level three of the WBS, but high-cost 

or high-risk elements may be reported at lower levels to give management an appropriate view of problems. 

Format 2 provides the same cost and schedule data as format 1 but breaks them out functionally, using the 

contractor’s organizational breakdown structure. Format 2 may be optional for agencies other than DOD. 

It need not be obtained, for example, when a contractor does not manage along functional lines. 

Format 3 shows the budget baseline plan, against which performance is measured, as well as any changes 

that have occurred. It also displays cumulative, current, and forecasted data, usually in detail for the next 6 

months and in larger increments beyond 6 months. This format forecasts the time-phased budget baseline 

cost to the end of the program—in other words, the reported data primarily look forward—and should be 

correlated with the cost estimate. 

Format 4 forecasts the staffing levels by functional category required to complete the contract and is an 

essential component to evaluating the EAC. This format—also forward looking—allows the analyst to 

correlate the forecast staffing levels with contract budgets and cost and schedule estimates. 

Format 5 is a detailed, narrative report explaining significant cost and schedule variances and other 

contract problems and topics.
GAO-09-3SP Chapter 19 253

The majority of EVM analysis comes from the CPR’s format 1—that is, from examining lower-level 

control account status to determine lower-level variances—and format 5—that is, from explanations for 

what is causing the variances in format 1. Table 35 describes some of the major data elements in format 1. 

Table 35: Contract Performance Report Data Elements: Format 1 

Data element Description

Contract data

Contract budget 

base

Includes the negotiated contract cost plus the estimated cost of any authorized, 

unpriced work

Negotiated cost Includes the dollar value (excluding fee or profit) of the contractually agreed-to program 

cost, typically the definitized contract target cost for an incentive-type contract;a

excludes costs for changes that have not been priced and incorporated into the contract 

through a modification or supplemental agreement

Estimated cost 

of authorized, 

unpriced work 

Excludes fee or profit; represents work that has been authorized but the contract 

price for it has not been definitized by either a contract change order or supplemental 

agreementa

Budget at 

completion (BAC)

The sum of all estimated budgets, representing at the program level the cumulative 

value of BCWS over the life of the program; at lower-levels, such as a control account or 

WBS element, it represents a roll-up of total estimated cost for the individual element 

(within a contract, the summary BAC is, in effect, the official spend plan for the contract)

Estimated cost at 

completion (EAC)

Represents a range of estimated costs at completion so that management has flexibility 

to analyze possible outcomes; it should be as accurate as possible, consider known 

or anticipated risks, and be reported without regard to the contract ceiling cost; it is 

derived by adding to actual costs the forecasted cost of work remaining (budgeted cost 

for work remaining), using an appropriate forecasting method

Variance at 

completion 

Representing the entire program overrun or underrun, it is calculated by taking the 

difference between the BAC and EAC 

Performance data

Budgeted cost for 

work scheduled 

(BCWS)

Representing the amount of work set aside for a specific effort over a stated period of 

time, it specifically describes the detailed work that was planned to be accomplished 

according to the program schedule; it is the sum of the budgets for all the work 

packages, planning packages, etc., scheduled to be accomplished within a given time 

period; it is the monthly spread of the BAC at the performance measurement level

Budgeted cost for 

work performed 

(BCWP)

Representing the earned value for the work accomplished, it is the prime schedule item 

in the CPR; as earned value, it is the sum of the budgets for completed work packages 

and completed portions of open work packages, plus the applicable portion of the 

budgets for apportioned effort and level of effort; BCWP represents that portion of 

BCWS earned

Actual cost of work 

performed (ACWP)

Represents actual or accrued costs of the work performed

Cost variance The difference between BCWP and ACWP represents the cost position—a positive 

number means that work cost less than planned, a negative number that it cost more 
254 Chapter 19 GAO-09-3SP

Data element Description

Schedule variance The difference between BCWP and BCWS represents the schedule status—a positive 

number means that planned work was completed ahead of schedule, a negative 

number that it was not completed as planned. Although it is expressed in dollars and 

not time, one needs to consider that work takes time to complete and requires resources 

such as money; therefore, schedule variance is reported as a dollar amount to reflect 

the fact that scheduled work has a budget; it does not always translate into an overall 

program schedule time delay; if it is caused by activities on the critical path, then it may 

cause a time delay in the program

Budgeted cost for 

work remaining 

Represents the planned work that still needs to be done; its value is determined by 

subtracting budgeted cost for work performed from budget at completion

Source: DOD and SCEA.

a

Definitized cost or price = contract cost or contract price that has been negotiated. 

Using the measures in format 1 at the control account level, management can easily detect problems. The 

sooner a problem is detected, the easier it will be to reduce its effects or avoid it in future. However, it is 

not enough just to know there is a problem. It is also critical to know what is causing it. The purpose of 

format 5 of the CPR is to provide necessary insight into problems. This format focuses on how the control 

account manager will make corrections to avoid future cost overruns and schedule delays or change cost 

and schedule forecasts when corrective action is not possible. In addition, format 5 reports on what is 

driving past variances and what risks and challenges lie ahead. It is an option, though, to focus the format 

5 analyses on the top problems of the program instead of looking at each significant variance found in 

format 1 or 2. Thus, to be useful for providing good insight into problems, the format 5 variance report 

should discuss

■ changes in management reserve;

■ differences in various EACs;

■ performance measurement milestones that are inconsistent with contractual dates, perhaps 

indicating an overtarget schedule;

■ formal reprogramming or overtarget baseline;

■ significant staffing estimate changes; and

■ a summary analysis of the program. 

It should also discuss in detail significant problems for each cost or schedule variance, including their 

nature and reason, the effect on immediate tasks and the total program, corrective actions taken or 

planned, the WBS number of the variance, and whether the variance is driven primarily by labor or 

material.

That is, the format 5 variance report should provide enough information for management to understand 

the reasons for variances and the contractor’s plan for fixing them. Good information on what is causing 

variances is critical if EVM data are to have any value. If the format 5 is not prepared in this manner, then 

the EVM data will not be meaningful or useful as a management tool, as case study 46 illustrates.
GAO-09-3SP Chapter 19 255

Case Study 46: Cost Performance Reports, from Defense Acquisitions, 

GAO-05-183

The quality of the Navy’s cost performance reports, whether submitted monthly or 

quarterly, was inadequate in some cases—especially with regard to the variance analysis 

section describing the shipbuilder’s actions on problems. The Virginia class submarine 

and the Nimitz class aircraft carrier variance analysis reports discussed the root causes of 

cost growth and schedule slippage and described how the variances were affecting the 

shipbuilders’ projected final costs. However, the remaining ship programs tended to report 

only high-level reasons for cost and schedule variances, giving little to no detail regarding 

root cause analysis or mitigation efforts. For example, one shipbuilder did not provide 

written documentation on the reasons for variances, making it difficult for managers to 

identify risk and take corrective action.

Variance analysis reporting was required and being conducted by the shipbuilders, but the 

quality of the reports differed greatly. DOD rightly observed that the reports were one of 

many tools the shipbuilders and DOD used to track performance. To be useful, however, 

the reports should have contained detailed analyses of the root causes and impacts of cost 

and schedule variances. CPRs that consistently provided a thorough analysis of the causes 

of variances, their associated cost impacts, and mitigation efforts would have allowed the 

Navy to more effectively manage, and ultimately reduce, cost growth. 

Therefore, to improve management of shipbuilding programs and promote early 

recognition of cost issues, GAO recommended that the Navy require shipbuilders to 

prepare variance analysis reports that identified root causes of reported variances, 

associated mitigation efforts, and estimated future cost impacts.

GAO, Defense Acquisitions: Improved Management Practices Could Help 

Minimize Cost Growth in Navy Shipbuilding Programs, GAO-05-183

(Washington, D.C.: Feb. 28, 2005).

The level of detail for format 5 is normally determined by specific variance analysis thresholds, which, if 

exceeded, require problem analysis and narrative explanations. Therefore, each program has its own level 

of detail to report. Thresholds should be periodically reviewed and adjusted to ensure that they continue 

to provide management with the necessary view on current and potential problems. In addition, because 

the CPR should be the primary means of documenting ongoing communication between program 

manager and contractor, it should be detailed enough that cost and schedule trends and their likely effect 

on program performance are transparent. 

Monthly EVM Analysis

EVM data should be analyzed and reviewed at least monthly so that problems can be addressed as soon 

as they occur and cost and schedule overruns can be avoided or at least their effect can be lessened. Some 

labor intensive programs review the data weekly, using labor hours as the measurement unit, to spot and 

proactively address specific problems before they get out of control. 

Using data from the CPR, a program manager can assess cost and schedule performance trends. This 

information is useful because trends can be difficult to reverse. Studies have shown that once programs are 

15 percent complete, performance indicators can predict the final outcome. For example, a CPR showing 

an early negative trend for schedule status would mean that work is not being accomplished and the 
256 Chapter 19 GAO-09-3SP

program is probably behind schedule. By analyzing the CPR and the schedule, one could determine the 

cause of the schedule problem, such as delayed flight tests, changes in requirements, or test problems. A 

negative schedule variance can be a predictor of later cost problems, because additional spending is often 

necessary to resolve problems. CPR data also provide the basis for independent assessments of a program’s 

cost and schedule status and can be used to project final costs at completion, in addition to determining 

when a program should be completed.

Analyzing past performance provides great insight into how a program will continue to perform and 

can offer important lessons learned. Effective analysis involves communicating to all managers and 

stakeholders what is causing significant variances and developing trends and what corrective action plans 

are in place so informed decisions can be made. Analysis of the EVM data should be a team effort that 

is fully integrated into the program management process so results are visible to everyone. Finally, while 

the analysis focuses on the past and what can be learned from variances, it also projects into the future by 

relying on historical performance to predict where a program is heading. The principal steps for analyzing 

EVM data are

1. Analyze performance:

■ check data to see if they are valid,

■ determine what variances exist,

■ probe schedule variances to see if activities are on the critical path,

■ develop historical performance data indexes,

■ graph the data to identify any trends, and

■ review the format 5 variance analysis for explanations and corrective actions.

2. Project future performance:

■ identify the work that remains,

■ calculate a range of EACs and compare the results to available funding,

■ determine if the contractor’s EAC is feasible, and

■ calculate an independent date for program completion.

3. Formulate a plan of action and provide analysis to management.

These steps should be taken in sequence, since each step builds on findings from the previous one. 

Skipping the analysis steps to start off with projecting independent EACs would be dangerous if the 

EVM data have not been checked to see if they are valid. In addition, it is important to understand what 

is causing problems before making projections about final program status. For example, if a program is 

experiencing a negative schedule variance, it may not affect the final completion date if the variance is not 

associated with an activity on the critical path or if the schedule baseline represents an early “challenge” 

date. Therefore, it is a best practice to follow the analysis steps in the right order so that all information is 

known before making independent projections of costs at completion.
GAO-09-3SP Chapter 19 257

Analyze Performance

Check to See If the Data Are Valid

It is important to make sure that the CPR data make sense and do not contain anomalies that would 

make them invalid. If errors are not detected, then the data will be skewed, resulting in bad decisionmaking. To determine if the data are valid, they should be checked at all levels of the WBS, focusing on 

whether there are errors or data anomalies such as

■ negative values for ACWP, BAC, BCWP, BCWS, or EAC; 

■ unusually large performance swings (BCWP) from month to month; 

■ BCWP and BCWS data with no corresponding ACWP;

■ BCWP with no BCWS; 

■ BCWP with no ACWP;

■ ACWP with no BCWP;

■ ACWP that is way above or below the planned value;

■ inconsistency between EAC and BAC—for example, no BAC but an EAC or a BAC with no 

EAC;

■ ACWP exceeds EAC; 

■ BCWP or BCWS exceed BAC.

If the CPR data contain anomalies, the performance measurement data will be distorted. For example, a 

CPR reporting actual costs (ACWP) with no corresponding earned value (BCWP) could indicate that 

unbudgeted work is being performed but not captured in the CPR. When this happens, the performance 

measurement data will not reflect true status.

In addition to checking the data for anomalies, the EVM analyst should check whether the CPR data are 

consistent. For instance, the analyst should review whether the data reported at the bottom line in format 

1 match the total in format 2. The analyst should also assess whether program cost is consistent with the 

authorized budget. 

Determine What Variances Exist

Cost and schedule deviations from the baseline plan give management at all levels information about 

where corrective actions are needed to bring the program back on track or to update completion dates and 

EACs. While variances are often perceived as something bad, they provide valuable insight into program 

risk and its causes. Variances empower management to make decisions about how best to handle risks. For 

example, management may decide to allocate additional resources or hire technical experts, depending on 

the nature of the variance.

Because negative cost variances are predictive of a final cost overrun if performance does not change, 

management needs to focus on containing them as soon as possible. A negative schedule variance, 

however, does not automatically mean program delay; it means that planned work was not completed. 
258 Chapter 19 GAO-09-3SP

To know whether the variance will affect the program’s completion date, the EVM analyst also needs 

to analyze the time-based schedule, especially the critical path. Because EVM data cannot provide this 

information, data from the contractor’s scheduling system are needed. Therefore, EVM data alone cannot 

provide the full picture of program status. Other program management tools and information are also 

needed to better understand variances.

Probe Schedule Variances for Activities on the Critical Path

Schedule variances should be investigated to see if the effort is on the critical path. If it is, then the whole 

program will be delayed. And, as we mentioned before, any delay in the program will result in additional 

cost unless other measures are taken. The following methods are often used to mitigate schedule problems:

■ consuming schedule reserve if it is available,

■ diverting staff to work on other tasks while dealing with unforeseen delays,

■ preparing for follow-on activities early so that transition time can be reduced,

■ consulting with experts to see if process improvements can reduce task time,

■ adding more people to speed up the effort, and

■ working overtime.

Caution should be taken with adding more people or working overtime, since these options cost money. 

In addition, when too many people work on the same thing, communication tends to break down. 

Similarly, working excessive overtime can make staff less efficient. Therefore, careful analysis should 

precede adding staff or instituting overtime.

A good network schedule that is kept current is a critical tool for monitoring program performance. 

Carefully monitoring the contractor’s network schedule will allow for quickly determining when 

forecasted completion dates differ from the planned dates. Tasks may be resequenced or resources 

realigned to reduce the schedule condition. It is also important to determine whether schedule variances 

are affecting downstream work. For example, a schedule variance may compress remaining activities’ 

duration times or cause “stacking” of activities toward the end of the program, to the point at which it 

is no longer realistic to predict success. If this happens, then an overtarget schedule may be necessary 

(discussed in chapter 20).

Various schedule measures should be analyzed to better understand the impact of schedule variances. For 

example, the amount of float, as well as the number of tasks with lags, constraints, or lack of progress, 

should be examined each month. Excess float usually indicates that the schedule logic is flawed, broken, or 

absent. Large float values should be checked to determine if they are real or a consequence of incomplete 

scheduling. Similarly, a large number of tasks with constraints (such as limitations on when an activity 

can start or finish), typically are substitutes for logic and can mean that the schedule is not well planned. 

Lags are often reserved for time that is unchanging, does not require resources, and cannot be avoided 

(as in waiting for concrete to cure), but lags are often inappropriately used instead of logic to put activities 

on a specified date. Similarly, if open work packages are not being statused regularly, it may be that the 

schedule and EVM are not really being used to manage the program. Analyzing these issues can help 

assess the schedule’s progress.
GAO-09-3SP Chapter 19 259

In addition to monitoring tasks on the critical path, close attention should be paid to near-critical tasks 

and near-term critical path effort, as these may alert management to potential schedule problems. If a task 

is not on the critical path but is experiencing a large schedule variance, the task may be turning critical. 

Therefore, schedule variances should be examined for their causes. For instance, if material is arriving late 

and the variance will disappear once the material is delivered, its effect is minimal. But if the late material 

is causing tasks to slip, then its effect is much more significant.

Remember that while a negative schedule variance eventually disappears when the full scope of work is 

ultimately completed, a negative cost variance is not corrected unless work that has been overrunning 

begins to underrun—a highly unlikely occurrence. Schedule variances are usually followed by cost 

variances; as schedule increases, costs such as labor, rented tools, and facilities increase. Additionally, 

management tends to respond to schedule delays by adding more resources or authorizing overtime.

Develop Historical Performance Data Indexes

Performance indexes are necessary for understanding the effect a cost or schedule variance has on a 

program. For example, a $1 million cost variance in a $500 million program is not as significant as it is in 

a $10 million program. Because performance indexes are ratios, they provide a level of program efficiency 

that easily shows how a program is performing.

The cost performance index (CPI) and schedule performance index (SPI) in particular can be used 

independently or together to forecast a range of statistical cost estimates at completion. They also give 

managers early warning of potential problems that need correcting to avoid adverse results. Table 36

explains what the values of three performance indexes indicate about program status.

Table 36: EVM Performance Indexes 

Index Formula Indicator

Cost performance index (CPI), 

the ratio of work performed (or 

earned value) to actual costs for 

work performed

CPI = 

BCWP / ACWP

Like a negative cost variance, a CPI less than 1 is 

unfavorable, because work is being performed less 

efficiently than planned; a CPI greater than 1 is favorable, 

implying that work is being performed more efficiently 

than planned. CPI can be expressed in dollars: 0.9 means 

that for every dollar spent, the program has received 90 

cents worth of completed work

Schedule performance index 

(SPI), the ratio of work performed 

(or earned value) to the initial 

planned schedule

SPI = 

BCWP / BCWS

Like a negative schedule variance, an SPI less than 1 

indicates that work is not being completed as planned 

and the program may be behind schedule if the 

incomplete work is on the critical path; an SPI greater 

than 1 means work has been completed ahead of the 

plan. An SPI can be thought of as describing work 

efficiency: 0.9 means that for every dollar planned, the 

program is accomplishing 90 cents worth of work

To complete performance index 

(TCPI), cost performance to be 

achieved if remaining work is to 

meet contractor EAC

TCPI = BCWR /

(EAC – ACWP))a

CPI takes into account what the contractor has 

done and can be compared to TCPI to test the EAC’s 

reasonableness; if TCPI is higher than CPI, the contractor 

expects productivity to improve, which may not be 

feasible given past performance

Source: DOD and SCEA. 

a

BCWR = budgeted cost for work remaining.
260 Chapter 19 GAO-09-3SP

Just like variances, performance indexes should be investigated. An unfavorable CPI—one less than 

1.0—may indicate that work is being performed less efficiently or that material is costing more than 

planned. Or it could mean that more expensive labor is being employed, unanticipated travel was 

necessary, or technical problems were encountered. Similarly, a mistake in how earned value was taken 

or improper accounting could cause performance to appear to be less efficient. The bottom line: more 

analysis is needed to know what is causing an unfavorable condition. Likewise, favorable cost or schedule 

performance may stem from errors in the EVM system, not necessarily from work’s taking less time than 

planned or overrunning its budget. Thus, not assessing the full meaning behind the indexes runs the risk 

of basing estimates at completion on unreliable data.

Further, when using the CPI as a sanity check against the TCPI, if the TCPI is much greater than the 

current or cumulative CPI, then the analyst should discover whether this gain in productivity is even 

possible. If not, then the contractor is most likely being optimistic. A rule of thumb is that if the TCPI is 

more than 5 percent higher than the CPI, it is too optimistic. In addition, a CPI less than 1 is a cause for 

concern, because without exception, the cumulative CPI tends not to improve but, rather, declines after a 

program is 15 percent complete. 

An SPI different from 1.0 warrants more investigation to determine what effort is behind or ahead of 

schedule. To do this, one needs to examine the WBS to identify issues at the activity level associated with 

completing the work. Using this information, management could decide to reallocate resources, where 

possible, from activities that might be ahead of schedule (SPI greater than 1.10) to help activities that are 

struggling (SPI less than 0.90) to get back on track. There also should be a discussion on analyzing the 

free-float of activities that are slipping to see if proactive actions should take place so resources are not lost 

in future activities.

Performance reported early in a program tends to be a good predictor of how the program will perform 

later, because early control account budgets tend to have a greater probability of being achieved than those 

scheduled to be executed later. DOD’s contract analysis experience suggests that all contracts are frontloaded to some degree, simply because more is known about near-term work than far-term. To the extent 

possible, the IBR should check for this condition.

In addition to the performance indexes, three other simple and useful calculations for assessing program 

performance are

■ % planned = BCWS/BAC,

■ % complete = BCWP/BAC, and

■ % spent = ACWP/BAC.

Examining these formulas, one can see quickly whether a program is doing well or is in trouble. For 

example, if percent planned is much greater than percent complete, the project is significantly behind 

schedule. Similarly, if percent spent is much greater than percent complete, the project is significantly 

overrunning its budget. Moreover, if the percent of management reserve consumed is much higher than 

percent complete, the program is likely not to have sufficient budget to mitigate all risks. For example, if a 

program is 25 percent complete but has spent more than 50 percent of its management reserve, there may 

not be enough management reserve budget to cover remaining risks because, this early in the program, it 

is being consumed at twice the rate at which work is being accomplished.
GAO-09-3SP Chapter 19 261

Graph the Data to Discover Trends

For reasons we discussed in chapter 10, EVM data should be analyzed graphically to see what trends are 

apparent. Performance trends provide valuable information about how a program has been doing in terms 

of cost and schedule. They also help in understanding performance, important for accurately predicting 

costs at completion. Knowing what has caused problems in the past can help determine whether they will 

continue in the future. 

Trend analysis should plot current and cumulative EVM data and track the use of management reserve 

for a complete view of program status and an indication of where problems exist. Typical EVM data trend 

plots that can help managers know what is happening in their programs are

■ BAC and contractor EAC over the life of the contract;

■ historical, cumulative, and current cost and schedule variance trends;

■ CPI and SPI (cumulative and current), monthly burn rate, or current ACWP;

■ TCPI versus CPI (cumulative and current), format 3 baseline data; 

■ projected versus actual staffing levels from format 4; and

■ management reserve allocations and burn rate.

Plotting the BAC over the life of the contract will quickly show any contract rebaselines or major contract 

modifications. BACs that follow a stairstep trend mean that the program is experiencing changes or 

major overruns. Both should be investigated to see if the EVM data are still reliable. For example, if the 

contract has been modified, then an IBR may be necessary to ensure that the changes were incorporated 

and flowed down to the right control accounts. In figure 36, BAC for an airborne laser program has been 

plotted over time to show the effect of major contract modifications and program rebaselines.

Figure 36: Understanding Program Cost Growth by Plotting Budget at Completion Trends 

Feb 97

Jun 97

Oct 97

Feb 98

Jun 98

Oct 98

Feb 99

Jun 99

Oct 99

Feb 00

Jun 00

Oct 00

Feb 01

Jun 01

Oct 01

Feb 02

Jun 02

Oct 02

Feb 03

Oct 03

Feb 04

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

x

x

x

x

1,400,000

1,600,000

1,800,000

2,000,000

Dollars in thousands

Rebaselining and

restructuring efforts

indicated by X

//

Source: GAO.

Note: The trend examples in figures 36–38, shown for learning purposes, are drawn from GAO, Uncertainties Remain Concerning the 

Airborne Laser’s Cost and Military Utility, GAO-04-643R (Washington, D.C.: May 17, 2004), pp. 17–20.
262 Chapter 19 GAO-09-3SP

The figure reveals a number of contract modifications, program restructurings, and rebaselines in the 

airborne laser program over the 7 years 1997 to 2004. Looking at the plot line, one can quickly see that 

the program more than doubled in cost. The trend data also show instances of major change, making it 

easy to pinpoint exactly which CPRs should be examined to best understand the circumstances. 

In this example, cost growth occurred when the program team encountered major problems 

with manufacturing and integrating advanced optics and laser components. Initial cost estimates 

underestimated the complexity in developing these critical technologies, and funding was insufficient 

to cover these risks. To make matters worse, the team was relying on rapid prototyping to develop these 

technologies faster, and it performed limited subcomponent testing. These shortcuts resulted in substantial 

rework when parts failed during integration.

Besides examining BAC trends, it is helpful to plot cumulative and current cost and schedule variances 

for a high-level view of how a program is performing. If downward trends are apparent, the next step is 

to isolate where these problems are in the WBS. Figure 37 shows trends of increasing cost and schedule 

variance associated with the airborne laser program. 

Figure 37: Understanding Program Performance by Plotting Cost and Schedule Variances

Oct 02

Nov 02

Dec 02

Jan 03

Feb 03

Mar 03

Apr 03

May 03

Jun 03

Jul 03

Aug 03

Sep 03

0

(250,000)

(25,000)

(50,000)

(75,000)

(100,000)

(125,000)

(150,000)

(175,000)

(200,000)

(225,000)

(275,000)

(300,000)

Dollars in thousands

Cost

performance

Schedule

performance

Source: GAO.

Note: The trend examples in figures 36–38, shown for learning purposes, are drawn from GAO, Uncertainties Remain Concerning the 

Airborne Laser’s Cost and Military Utility, GAO-04-643R (Washington, D.C.: May 17, 2004), pp. 17–20.

In figure 37, cost variance steadily declined over fiscal year 2003, from an unfavorable $50 million to an 

almost $300 million overrun. At the same time, schedule variance also declined, but during the first half 

of the year it leveled off, after the program hired additional staff in March to meet schedule objectives. 

While the additional staff helped regain the schedule, they also caused the cost variance to worsen. 

Plotting both cost and schedule variances makes a wealth of information visible. Management can rely on 

this information to discover where attention is needed most.
GAO-09-3SP Chapter 19 263

Plotting various EACs along with the contractor’s estimate at completion is a very good way to see if 

the contractor’s estimate is reasonable. Figure 38, for example, shows expected cost overruns at contract 

completion for the airborne laser program.

Figure 38: Understanding Expected Cost Overruns by Plotting Estimate at Completion

Oct 02

Nov 02

Dec 02

Jan 03

Feb 03

Mar 03

Apr 03

May 03

Jun 03

Jul 03

Aug 03

Sep 03

0

(200,000)

(400,000)

(600,000)

(800,000)

(1,000,000)

Dollars in thousands

GAO best case

GAO most likely

GAO worst case

Contractor variance

at completion

Source: GAO.

Note: The trend examples in figures 37 and 38, shown for learning purposes, are drawn from GAO, Uncertainties Remain Concerning 

the Airborne Laser’s Cost and Military Utility, GAO-04-643R (Washington, D.C.: May 17, 2004), pp. 17–20.

Figure 38 plots various EACs that GAO generated from the contractor’s EVM data. GAO’s independent 

EACs showed that an overrun of between $400 million and almost $1 billion could be expected from 

recent program performance. The contractor, in contrast, was predicting no overrun at completion—

despite the fact that the program had already incurred a cost overrun of almost $300 million, seen in 

figure 37. 

That the program was facing huge technology development problems made it highly unlikely that the 

contractor could finish the program with no additional cost variances. In fact, there was no evidence that 

the contractor could improve its performance enough to erase the almost $300 million cumulative cost 

variance. Knowing this, the reasonable conclusion was that the contractor’s estimate at completion was 

not realistic, given that it was adding more personnel to the contract and still facing increasing amounts of 

uncompleted work from prior years. 

Another way to check the reasonableness of a contractor’s estimate at completion is to compare the CPI, 

current and cumulative, with the TCPI to see if historical trends support the contractor’s EAC.

Other trends that can offer insight into program performance include plotting the monthly burn rate, or 

ACWP. If the plotting shows a rate of increase, the analyst needs to determine whether the growth stems 

from the work’s becoming more complex as the program progresses or from overtime’s being initiated 

to make up for schedule delays. Reviewing monthly ACWP and BCWP trends can also help determine 

what is being accomplished for the amount spent. In the data in figures 37 and 38, for example, it was 

evident that the program was paying a large staff to make a technological breakthrough rather than 

paying its staff overtime just to meet schedule goals. It is important to know the reasons for variances, 
264 Chapter 19 GAO-09-3SP

so management can make decisions about the best course of action. For the program illustrated in the 

figures, we recognized that since the airborne laser program was in a period of technology discovery that 

could not be forced to a specific schedule, any cost estimate would be highly uncertain. Therefore, we 

recommended that the agency develop a new cost estimate for completing technology development and 

perform an uncertainty analysis to quantify its level of confidence in that estimate.

Other trend analyses include plotting CPR format 3 data over time to show whether budget is being 

moved to reshape the baseline. Comparing planned to actual staffing levels—using a waterfall chart 

to analyze month-to-month profiles—can help determine whether work is behind schedule for lack of 

available staff.77 This type of trend analysis can also be used to determine whether projected staffing levels 

shown in CPR format 4 represent an unrealistic expectation of growth in labor resources. 

Finally, plotting the allocation and burn rate of management reserve is helpful for tracking and analyzing 

risk. Since management reserve is a budget tool to help manage risks, analyzing its rate of allocation is 

important because when management reserve is consumed, any further risk that is realized can only be 

manifested as unfavorable cost variance. Accordingly, risks from the cost estimate uncertainty analysis 

should be compared against the management reserve allocation to understand where in the WBS risks 

are turning into issues. This analysis is a best practice because it further ties the cost estimating risk 

analysis with EVM. It can also prevent the handing out of budget whenever a program encounters a 

problem, ensuring that as more complicated tasks occur later in the program, management reserve will be 

available to mitigate any problems. Therefore, to meet this best practice, risks in the cost estimate should 

be identified up front and conveyed to the EVM analysts, so they can keep a look out for risks in specific 

WBS elements. Thus, it is absolutely necessary to integrate cost estimating and EVM in order to have the 

right information to make good judgments about when to allocate management reserve. 

Review the Format 5 Variance Analysis

After determining which WBS elements are causing cost or schedule variances, examining the format 5 

variance analysis can help determine the technical reasons for variances, what corrective action plans are 

in place, and whether or not the variances are recoverable. Corrective action plans for cost and schedule 

variances should be tracked through the risk mitigation process. In addition, favorable cost variances 

should be evaluated to see if they are positive as a result of performance without actual cost having been 

recorded. This can happen when accounting accruals lag behind invoice payments. Finally, the variance 

analysis report should discuss any contract rebaselines and whether any authorized unpriced work exists 

and what it covers.

Examining where management reserve has been allocated within the WBS is another way to identify 

potential issues early on. An alarming situation arises if the CPR shows that management reserves are 

being used faster than the program is progressing toward completion. For example, management should 

be concerned if a program has used 80 percent of its management reserves but has completed only 40 

percent of its work. EVM experts agree that a program’s management reserves should be sufficient to 

mitigate identified program risk so that budget will always be available to cover unexpected problems. 

77A waterfall chart is made up of floating columns that typically show how an initial value increases and decreases by a series 

of intermediate values leading to a final value; an invisible column keeps the increases and decreases linked to the heights of the 

previous columns. Waterfall charts can be created by applying widely available add-in tools to Microsoft Excel.
GAO-09-3SP Chapter 19 265

This is especially important toward the latter half of a program, when adequate management reserve is 

needed to cover problems during testing and evaluation. When management reserve is gone, any work 

that could have been budgeted from it can only manifest as additional cost overrun. And, when it is gone, 

the analyst should be alert to contractor requests to increase the contract value to avoid variances.

Project Future Performance

Identify the Work That Remains

Two things are needed to project future performance: the actual costs spent on completed work and 

the cost of remaining work. Actual costs spent on completed work are easy to determine because they 

are captured by the ACWP. The remaining work is determined by subtracting BCWP from BAC to 

derive the budgeted cost of work remaining. However, to be accurate, the EAC should take into account 

performance to date when estimating the cost of the remaining work.

Calculate a Range of EACs and Compare to Available Funding

It is a best practice to develop more than one EAC, but determining an accurate EAC is difficult because 

EVM data can be used to develop a multitude of EACs. Picking the right EAC is challenging since the 

perception is that bad news about a contract’s performance could put a program and its management in 

jeopardy. By calculating a range of EACs, management can know a likely range of costs for completing 

the program and take action in response to the results.

While plenty of EACs can be generated from the EVM data, each EAC is calculated with a generic indexbased formula, similar to

EAC = ACWP (cumulative) + (BAC – BCWP (cumulative)) / efficiency index

The difference in EACs is driven by the efficiency index that is used to adjust the remaining work 

according to the program’s past cost and schedule performance. The idea in using the efficiency index is 

that how a program has performed in the past will indicate how it will perform in the future. The typical 

performance indexes include the CPI and SPI, but these could represent cumulative, current, or average 

values over time. In addition, the indexes could be combined to form a schedule cost index—as in CPI x 

SPI—which can be weighted to emphasize either cost or schedule impact. Further, EACs can be generated 

with various regression analyses in which the dependent variable is ACWP and the independent value is 

BCWP, a performance index, or time. Thus, many combinations of efficiency indexes can be applied to 

adjust the cost of remaining work. 

Table 37 summarizes findings from studies in which EACs make the best predictors, depending on where 

the program is in relation to its completion. The findings are based on extensive research that compared 

efficiency factors that appeared to best predict program costs. The conclusion was that no one factor 

was superior. Instead, the best EAC efficiency factor changes by the stage of the program. For example, 

the research found that assigning a greater weight to SPI is appropriate for predicting costs in the early 

stage of a program but not appropriate later on. SPI loses its predictive value as a program progresses 

and eventually returns to 1.0 when the program is complete. The research also found that averaging 

performance over a shorter period of time—3 months, for example—was more accurate for predicting 

costs than longer periods of time—such as 6 to12 months—especially in the middle of a program, when 

costs are being spent at a greater rate. 
266 Chapter 19 GAO-09-3SP

Table 37: Best Predictive EAC Efficiency Factors by Program Completion Status

EAC efficiency factor

Percent complete

Comment

Early:

0%–40%

Middle:

20%–80%

Late:

60%–100%

CPI Cumulative x x x Assumes the contractor will 

operate at the same efficiency for 

remainder of program; typically 

forecasts the lowest possible EAC

3-month 

average

x x x Weights current performance 

more heavily than cumulative 

past performance 6-month 

average

x x

12-month 

average

x x

CPI x SPI Cumulative x x Usually produces the highest EAC

6-month 

average

x x A variation of this formula (CPI6

 x 

SPI), also proven accuratea

SPI Cumulative x Assumes schedule will affect cost 

also but is more accurate early in 

the program than later

Regression x Using CPI that decreases within 

10% of its stable value can be a 

good predictor of final costs and 

should be studied further

Weighted x x Weights cost and schedule based 

on .x(CPI) + .x(SPI); statistically the 

most accurate, especially when 

using 50% CPI x 50% SPI a

Source: Industry.

a

According to DOD comments based on the work of David S. Christensen. 

Other methods, such as the Rayleigh model, rely on patterns of manpower build-up and phase-out to 

predict final cumulative cost. This model uses a nonlinear regression analysis of ACWP against time 

to predict final cumulative cost and duration and has been known to be a high-end EAC forecast. 

One benefit of using this model is that as long as actual costs are available, they can be used to forecast 

cumulative cost at completion and to assess overall cost and schedule risk.

Relying on the CPI and SPI performance factors usually results in higher EACs if their values are less 

than 1.0. How much the cost will increase depends on the specific index and how many months are 

included in determining the factor. Research has also shown that once a program is 20 percent complete, 

the cumulative CPI does not vary much from its value (less than 10 percent) and most often tends to get 

worse as completion grows nearer. Therefore, projecting an EAC by using the cumulative CPI efficiency 

factor tends to generate a best case EAC. 

In contrast, the schedule cost index—some form of CPI x SPI—takes the schedule into account to forecast 

future costs. This index produces an even higher EAC by compounding the effect of the program’s being 

behind schedule and over cost. The theory behind this index is that to get back on schedule will require 
GAO-09-3SP Chapter 19 267

more money because the contractor will either have to hire more labor or pay for overtime. As a result, the 

schedule cost index forecast is often referred to as a worst case predictor.

A more sophisticated EAC method relies on using actual costs to date plus the remaining work with a 

cost growth factor applied plus a cost impact for probable schedule delays. Using this method takes into 

account cost, schedule, and technical risks that can result from test failures or other external factors that 

have occurred in other past programs and relies on simulation to determine the probability effect. Finally, 

an integrated schedule can be used, in combination with risk analysis data and Monte Carlo simulation 

software, to estimate schedule risk and the EAC (chapter 18, step 10, has more details). 

EACs should be created not only at the program level but also at lower levels of the WBS. By doing this, 

areas that are performing poorly will not be masked by other areas doing well. If the areas performing 

worse represent a large part of the BAC, then this method will generate a higher and more realistic EAC. 

Once a range of EACs has been developed, the results should be analyzed to see if additional funding is 

required. Independent EACs provide a credible rationale for requesting additional funds to complete the 

program, if necessary. Their information is critical for better program planning and avoiding a situation 

in which work must be stopped because funds have been exhausted. Early warning of impending funding 

issues enables management to take corrective action to avoid any surprises.

Determine Whether the Contractor’s EAC Is Feasible

While EVM data are useful for predicting independent EACs, the contractor should also look at other 

information to develop its EAC. In particular, the contractor should

■ evaluate its performance on completed work and compare it to the remaining budget,

■ assess commitment values for material needed to complete remaining work, and

■ estimate future conditions to generate the most accurate EAC,

Further, the contractor should periodically develop a comprehensive EAC, using all information available 

to develop the best estimate possible. This estimate should also take into account an assessment of risk 

based on technical input from the team. Once the EAC is developed, it can be compared for realism 

against other independent EACs and historical performance indexes.

A case in point is the Navy’s A-12 program, cancelled in January 1991 by the Secretary of Defense because 

estimates based on EVM of the cost to complete it showed substantial overruns. Many estimates had 

been developed for the program. The program manager had relied on the lower EAC, even though higher 

EACs had been calculated. The inquiry into the A-12 program cancellation concluded that management 

tended to suppress bad news and that this was not a unique problem but common within DOD. 

Since a contractor typically uses methods outside EVM to develop an EAC, EVM and risk analysis 

results can be used to assess the EAC’s reliability. While the contractor’s EAC tends to account for special 

situations and circumstances that cannot be accurately captured by looking only at statistics, it also tends 

to include optimistic views of the future. One way to assess the validity of the EAC is to compare the 

TCPI to the CPI. Because the TCPI represents the ratio of remaining work to remaining funding and 

indicates the level of performance the contractor must achieve and maintain to stay within funding goals, 

it can be a good benchmark for assessing whether the EAC is reasonable. Therefore, if the TCPI is greater 
268 Chapter 19 GAO-09-3SP

than the CPI, this means that the contractor expects productivity to be higher in the future. To determine 

whether this is a reasonable assumption, analysts should look for supporting evidence that backs up this 

claim. 

A typical rule of thumb is that if the CPI and TCPI differ by more than 5 percent to 10 percent, and 

the program is more than 20 percent complete, the contractor’s EAC is too optimistic. For example, if 

a program’s TCPI is 1.2 and the cumulative CPI is 0.9, it is not statistically expected that the contractor 

can improve its performance that much through the remainder of the program. To meet the EAC cost, 

the contractor must produce $1.20 worth of work for every $1.00 spent. Given the contractor’s historical 

performance of $0.90 worth of work for every $1.00 spent, it is highly unlikely that it can improve 

its performance that much. One could conclude that the contractor’s EAC is unrealistic and that it 

underestimates the final cost. 

Another finding from more than 500 studies is that once a contract is more than 15 percent complete, 

the overrun at completion will usually be more than the overrun already incurred.78 Looking again at the 

example of the airborne laser program discussed around figures 37–38, we see that while the contractor 

predicted no overrun at completion, there was a cumulative unfavorable cost variance of almost $300 

million. According to this research statement, one could conclude that the program would overrun by 

$300 million or more. Using EVM data from the program, we predicted that the final overrun could be 

anywhere between $400 million and almost $1 billion by the time the program was done. 

Calculate an Independent Date for Program Completion

Dollars can be reallocated to future control accounts by management, but time cannot. If a cost underrun 

occurs in one cost account, the excess budget can be transferred to a future account. But if a control 

account is 3 months ahead and another is 3 months behind, time cannot be shifted from the one account 

to the other to fix the schedule variance. Given this dynamic, the schedule variance should be examined 

in terms of the network schedule’s critical and near-critical paths to determine what specific activities are 

behind schedule, and a schedule risk analysis should determine which activities may cause the schedule to 

extend in the future.

In the simplest terms, the schedule variance describes what was or was not accomplished but does not 

provide an accurate assessment of schedule progress. To project when a program will finish, management 

must know whether the activities that are contributing to a schedule variance are on the critical path or 

may ultimately be on the path, if mitigation is not pursued. If they may, then any slip in the critical path 

activities will result in a slip in the program, and sufficient slippage in near-critical paths may ultimately 

have the same result. Therefore, the program manager should analyze the activities undergoing delay 

to see if they may ultimately delay the program. If they are, then the program may be in danger of not 

finishing on schedule and an analysis, generally a schedule risk analysis, should be conducted to determine 

the most likely completion date. In addition, a schedule risk analysis (described in appendix X) should be 

made periodically to assess changes to the critical path and explain schedule reserve erosion and mitigation 

strategies for keeping the program on schedule.

78David S. Christensen, Determining an Accurate Estimate at Completion (Cedar City: Southern Utah University, 1993), p. 7.
GAO-09-3SP Chapter 19 269

Provide Analysis to Management

The ability to act quickly to resolve program problems depends on having an early view of what is causing 

them. Management’s having accurate progress assessments makes for a better picture of program status 

and leads to better decisions. When problems are identified, they should be captured within the program’s 

risk management process so that someone can be assigned responsibility for tracking and correcting them. 

In addition, using information from the independent EACs and the contractor’s EAC, management 

should decide whether additional program funding should be requested and, if so, make a convincing case 

for more funds. Management should also be sure to link program outcomes to award-fee objectives. For 

example, management can look back to earlier CPRs to see if they objectively depicted contract status and 

predicted certain problems. This approach supports performance-based reporting and rewards contractors 

for managing effectively and reporting actual conditions, reducing the need for additional oversight.

Continue EVM until the Program Is Complete

EVM detail planning is never ending and continues until the program is complete. Converting planning 

packages into detailed work packages so that near-term effort is always detailed is called “rolling wave” 

planning. This approach gives the contractor flexibility for planning and incorporating lessons learned.

Rolling-wave planning that is based solely on calendar dates is an arbitrary practice that may result in 

insufficient detail. When this approach is used, work is planned in 6-month increments; all effort beyond 

a 6-month unit is held in a planning package. Each month, near-term planning packages are converted to 

detailed work packages to ensure that 6 months of detailed planning are always available to management. 

This continues until all work has been planned in detail and the program is complete. A better method 

is to plan in detail a significant technical event, such as the preliminary design review. By using technical 

milestones rather than calendar dates, better cost, schedule, and technical performance integration can be 

achieved, as depicted in figure 39. 

Figure 39: Rolling Wave Planning

Dec 00

Time

Mar 01 Jun 01 Sept 01 Dec 01 Mar 02 Jun 02 Sept 02

0

100

200

300

400

Resources Preliminary

design review

Critical

design review

Project

end

Initial planning

detail

Future work in planning packages

Detail planning based on technical objectives

Facilitates event-based reporting incentives

Better integration of technical, schedule, and

cost performance and risk management

•

•

•

•

Source: Abba Consulting.
270 Chapter 19 GAO-09-3SP

The unwritten rule that 1 month of detailed planning should be added to previously detailed planning is 

related more to creating than managing to a baseline, which is the heart of EVM. Therefore, managing to 

a technical event is the best practice and yields the best EVM benefits.

Continually planning the work supports an EVM system that will help management complete the 

program within the planned cost and proposed schedule. This is important, since EVM data are essential 

to effective program management and can be used to answer basic program management questions such 

as those in table 38.

Table 38: Basic Program Management Questions That EVM Data Help Answer

Question Answer

How much progress has the program made so far? Percent complete

What are the significant deviations from the plan? ■ Cost variance 

■ Schedule variance 

■ Variance at completion 

How efficiently is the program meeting cost and 

schedule objectives?

■ Cost performance index (CPI)

■ Schedule performance index (SPI)

Are cost and schedule trends getting better or worse? Plotting cost and schedule variance, CPI, SPI, etc.

Will the program be completed within the budget? To complete performance index (TCPI) for the 

budget at completion (BAC)

Is the contractor’s estimate at completion (EAC) 

reasonable?

TCPI for the contractor’s EAC

What other estimates are reasonable for completing the 

authorized scope of work?

Independent EACs using statistical forecasting 

techniques based on various efficiency factors 

What action will bring the program back on track? Acting on format 5 variance analysis information

Source: © 2003, Society of Cost Estimating and Analysis (SCEA), “Earned Value Management Systems” 

From questions such as those in table 38, reliable EVM data can help inform the most basic program 

management needs. The questions also provide an objective way of measuring progress so that accurate 

independent assessments of EACs can be developed and presented to stakeholders.

16. Best Practices Checklist: Managing Program Costs: Execution

 An IBR verified that the baseline budget and schedule captured the entire 

scope of work, risks were understood, and available and planned resources 

were adequate. 

3 Separate IBRs were conducted at the prime contractor and all major 

subcontractors. 

3 A performance measurement baseline assessment made a 

comprehensive and value-added review of control accounts.

ù Before award, or not more than 6 months after, an IBR categorized 

risks by severity and provided team training.

ù Work definition (including provisions for rework and retesting), 

schedule integration, resource identification, earned value 

measures, and baseline validation were matured and reviewed.
GAO-09-3SP Chapter 19 271

ù Interviewers used a template in discussions with control account 

managers and identified where additional training was needed. 

ù An action plan for assigning responsibility for handling risks was 

developed, and a final program risk rating was based on a summary 

of all identified risks.

ù Management reserve was set aside that covered identified risks and 

care was taken to include risks identified during the IBR in the risk 

management plan.

ù An EVM analyst monitored corrective action requests for closure. 

ù A memorandum for the record described the IBR findings. 

 A contract performance report summarized EVM data.

3 The data were reviewed monthly to track program progress, risks, and 

plans.

3 Management used the data to

ù integrate cost and schedule performance data with technical 

measures;

ù identify the magnitude and effect of problems causing significant 

variances;

ù inform higher management of valid and timely program status and 

project future performance.

3 Format 1 of the CPR reported data to at least level 3 of the WBS, and 

format 5 explained variances and the contractor’s plans for fixing them.

 Program managers analyzed EVM data monthly and sequentially for 

variances and EACs.

3 The EVM data were checked for validity and anomalies. 

3 Performance indexes were analyzed and plotted for trends and 

variances.

3 Schedule variances were analyzed against the most recently statused 

schedule to see if problems were occurring on or near the critical path.

3 Management reserve allocations in the WBS were examined and 

compared against risks identified in the cost estimate. 

3 A range of EACs was developed, using a generic index-based formula or 

relying on probable cost growth factors on remaining work, combined 

with an integrated cost schedule risk analysis. 

3 An independent date for program completion was determined, using 

schedule risk analysis that identifies which activities need to be closely 

monitored.

3 Senior management used EVM data to answer basic program questions.
272 Chapter 19 GAO-09-3SP
GAO-09-3SP Chapter 20 273

Managing Program Costs: Updating

Chapter 20

Programs should be monitored continuously for their cost effectiveness by comparing planned and actual 

performance against the approved program baseline. In addition, the cost estimate should be updated 

with actual costs so that it is always relevant and current. The continual updating of the cost estimate 

as the program matures not only results in a higher-quality estimate but also gives opportunity to 

incorporate lessons learned. Future estimates can benefit from the new knowledge. For example, cost or 

schedule variances resulting from incorrect assumptions should always be thoroughly documented so as 

not to repeat history. Finally, actual cost and technical and historic schedule data should be archived in a 

database for use in supporting future estimates.

Most programs, especially those in development, do not remain static; they tend to change in the natural 

evolution of a program. Developing a cost estimate should be not a one-time event but, rather, a recurrent 

process. Before changes are approved, however, they should be examined for their advantages and effects 

on the program cost. If changes are deemed worthy, they should be managed and controlled so that the 

cost estimate baseline continuously represents the new reality. Effective program and cost control requires 

ongoing revisions to the cost estimate, budget, and projected estimates at completion. 

Incorporating Authorized Changes into the Performance Measurement

Baseline

While the 32 ANSI guidelines are for the overarching goal of maintaining the integrity of the baseline 

and resulting performance measurement data, changes are likely throughout the life of the program, so 

that the performance measurement baseline should be updated to always reflect current requirements 

or changes in scope. Some changes may be simple, such as modifying performance data to correct for 

accounting errors or other issues that can affect the accuracy of the EVM data. Other changes can be 

significant, as when major events or external factors beyond the program manager’s control result in 

changes that will greatly affect the performance measurement baseline. Key triggers for change include

■ contract modifications, including engineering change proposals;

■ shifting funding streams;

■ restricting funding levels;

■ major rate changes, including overhead rates;

■ changes to program scope or schedule;

■ revisions to the acquisition plan or strategy; and

■ executive management decisions.
274 Chapter 20 GAO-09-3SP

Since the performance measurement baseline should always reflect the most current plan for 

accomplishing authorized work, incorporating changes accurately and in a timely manner is especially 

important for maintaining the effectiveness of the EVM system. Table 39 describes the ANSI guidelines 

with regard to correctly revising the performance measurement baseline.

Table 39: ANSI Guidelines Related to Incorporating Changes in an EVM System

Guideline Description

Incorporate authorized changes in 

a timely manner, recording their 

effects in budgets and schedules; 

in the directed effort before 

negotiating a change, base the 

changes on the amount estimated 

and budgeted to the program 

organizations

Incorporating authorized changes quickly maintains the performance 

measurement baseline’s effectiveness for managing and controlling 

the program; therefore, authorized changes in the baseline should 

be incorporated in a documented, disciplined, and timely manner so 

that budget, schedule, and work remain coupled for true performance 

measurement. The contractor will develop its best estimate for 

planning and budgeting into changes not yet negotiated; when 

changes are incorporated, existing cost and schedule variances should 

not be arbitrarily eliminated, but economic price and rate adjustments 

may be made as appropriate

Reconcile current budgets to prior 

budgets in terms of changes to 

the authorized work and plan 

the effort in the detail needed by 

management for effective control

When budget revisions can be reconciled, the integrity of the 

performance measurement baseline can be verified; budget changes 

should be controlled and understood in terms of scope, resources, and 

schedule so the baseline reflects current levels of authorized work. 

Budget revisions should also be traceable to authorized control account 

budgets; if additional in-scope work has been identified, management 

reserve can augment existing control account budgets

Control retroactive changes 

to records pertaining to work 

performed that would change 

previously reported amounts 

for actual costs, earned value, or 

budgets

To avoid masking historic variance trends needed to project estimates 

at completion, retroactive changes need to be controlled; retroactive 

adjustments to costs should happen only as a result of routine 

accounting adjustments—e.g., change orders that have not been 

priced, rate changes, and economic price adjustments—customerdirected changes, or data entry corrections. Limiting retroactive 

changes to these conditions ensures baseline integrity and accurate 

performance measurement data 

Prevent revisions to the program 

budget except for authorized 

changes

If changes are not made within a controlled process, the integrity of 

performance trend data may be compromised and understanding of 

overall program status will be delayed; to maintain baseline integrity, 

unauthorized revisions to the performance measurement baseline 

should be prevented. All changes must be approved and implemented 

following a well-defined baseline management control process; 

this avoids implementing a budget baseline that is greater than the 

program budget. Only in the situation of an overtarget baseline should 

the performance budget or schedule objectives exceed the program 

plan 

Document changes to the 

performance measurement baseline

Properly maintaining the performance measurement baseline enables 

control account managers to accurately measure performance; it 

should always reflect the most current plan for accomplishing the work. 

All authorized changes should be quickly incorporated; before any new 

work begins, all planning documents should be updated to maintain 

the EVM system’s integrity

Source: Adapted from National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC), ANSI/EIA-748-A Standard for Earned Value 

Management Systems Intent Guide (Arlington, Va.: January 2005). 
GAO-09-3SP Chapter 20 275

It is also important to note that a detailed record of the changes made to the performance measurement 

baseline should be established and maintained. Doing so makes it easy to trace all changes to the program 

and lessens the burden on program personnel when compiling this information for internal and external 

program audits, EVM system surveillance reviews, and updates to the program cost estimate. If changes 

are not recorded and maintained, the program’s performance measurement baseline will not reflect reality. 

The performance measurement baseline will become outdated and the data from the EVM system will 

not be meaningful. Case study 47 highlights a program in which this occurred.

Case Study 47: Maintaining Performance Measurement Baseline 

Data, from National Airspace System, GAO-03-343

The Federal Aviation Administration (FAA) obtained monthly cost performance reports 

from the contractor on the Standard Terminal Automation Replacement System (STARS). 

The agency should have been able to use the reports for overseeing the contractor’s 

performance and estimating the program’s remaining development costs. FAA did not 

use these reports, however, because they were not current. Their central component, the 

performance measurement baseline—which established performance, cost, and schedule 

milestones for the contract—had not been updated since May 2000 and therefore did not 

incorporate the effects of later contract modifications.

For example, the September 2002 cost performance report did not reflect FAA’s March 

2002 reduction in STARS’ scope from 188 systems to 74 systems, and it did not include 

the cost of new work that FAA authorized between May 2000 and September 2002. 

Consequently, the report indicated that STARS was on schedule and within 1 percent of 

budget, even though—compared to the program envisioned in May 2000—FAA was now 

under contract to modernize fewer than half as many facilities at more than twice the cost 

per facility, 

FAA had not maintained and controlled the baseline because, according to program 

officials, the program was “schedule driven.” Without a current, valid performance 

management baseline, FAA could not compare what the contractor had done with what 

the contractor had agreed to do. And, because the baseline had not been maintained 

and was not aligned with the program’s current status, the reports were not useful 

for evaluating the contractor’s performance or for projecting the contract’s remaining 

costs. Therefore, FAA lacked accurate, valid, current data on the STARS program’s costs 

and progress. Without such data, FAA was limited in its ability to effectively oversee the 

contractor’s performance and reliably estimate future costs,

GAO, National Airspace System: Better Cost Data Could Improve FAA’s 

Management of the Standard Terminal Automation Replacement System, 

GAO-03-343 (Jan. 31, 2003).

The performance measurement baseline should be the official record of the current program plan. If it is 

updated in a timely manner to reflect inevitable changes, it can provide valuable management information 

that yields all the benefits discussed in chapter 18. 
276 Chapter 20 GAO-09-3SP

Using EVM System Surveillance to Keep the Performance Measurement

Baseline Current

Surveillance is reviewing a contractor’s EVM system as it is applied to one or more programs. Its purpose 

is to focus on how well a contractor is using its EVM system to manage cost, schedule, and technical 

performance. For instance, surveillance checks whether the contractor’s EVM system

■ summarizes timely and reliable cost, schedule, and technical performance information directly 

from its internal management system;

■ complies with the contractor’s implementation of ANSI guidelines;

■ provides timely indications of actual or potential problems by performing spot checks, sample data 

traces, and random interviews;

■ maintains baseline integrity;

■ gives information that depicts actual conditions and trends;

■ provides comprehensive variance analyses at the appropriate levels, including corrections for cost, 

schedule, technical, and other problem areas;

■ ensures the integrity of subcontractors’ EVM systems; 

■ verifies progress in implementing corrective action plans to mitigate EVM system deficiencies; and 

■ discusses actions taken to mitigate risk and manage cost and schedule performance.

Effective surveillance ensures that the key elements of the EVM process are maintained over time and 

on subsequent applications. The two goals associated with EVM system surveillance ensure that the 

contractor is following its own corporate processes and procedures and confirm that the contractor’s 

processes and procedures continue to satisfy the ANSI guidelines.

OMB has endorsed the NDIA surveillance guide we listed in tables 3 and 32 to assist federal agencies in 

developing and implementing EVM system surveillance practices, which include 

■ establishing a surveillance organization, 

■ developing an annual corporate-level surveillance plan,

■ developing a program-level surveillance plan, 

■ executing the program surveillance plan, and

■ managing system surveillance based on program results.79

Establishing a Surveillance Organization

An organization must have designated authority and accountability for EVM system surveillance to assess 

how well a contractor applies its EVM system relative to the ANSI guidelines. Surveillance organizations 

should be independent of the programs they assess and should have sufficient experience in EVM. These 

requirements apply to all surveillance organizations, whether internal or external to the agency, such as 

consultants. Table 40 further describes the elements of an effective surveillance organization. 

79NDIA, National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC) Surveillance Guide

(Arlington, Va.: October 2004).
GAO-09-3SP Chapter 20 277

Table 40: Elements of an Effective Surveillance Organization

Element Description

Independent organizational 

level 

The surveillance organization reports to a management structure different 

from the programs it surveys; it is independent to ensure that its findings are 

objective and that it will identify systemic issues on multiple programs; it has 

sufficient authority to resolve issues and typically rests at an agency’s higher 

levels

Organizational charter The organization’s charter is defined through agency policy, outlining its role, 

responsibilities, resolution process, and membership; responsibilities include 

developing annual surveillance plans, appointing surveillance review team 

leaders, assigning resources for reviews, communicating surveillance findings, 

tracking findings to closure, developing and maintaining databases of 

surveillance measures, and recommending EVM system process and training 

to fix systemic findings

Membership consistent with 

chartered responsibilities

The organization’s staff are consistent with its chartered responsibilities; their 

key attributes include multidisciplinary knowledge of the agency and its 

programs, practical experience in using EVM, good relationships with external 

and internal customers, and strong support of EVM systems compliance

Source: Adapted from National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC), Program Management Systems Committee

Surveillance Guide (Arlington, Va.: October 2004). 

OMB states that full implementation of EVM includes performing periodic system surveillance reviews 

to ensure that the EVM system continues to meet the ANSI guidelines. Periodic surveillance therefore 

subjects contractors’ EVM systems to ongoing government oversight. 

DCMA, a DOD support agency that provides a range of acquisition management services, monitors 

contractor performance through data tracking and analysis, onsite surveillance, and tailored support to 

program managers. DCMA also leads EVM system validation reviews before contract award, supports 

programs with monthly predictive EVM analysis, and participates in IBRs as requested. 

Unlike DOD, however, nonmilitary agencies do not have the equivalent of a DCMA, and since DCMA 

does not have enough staff to cover all DOD demands, it is not possible for all nonmilitary agencies to 

ask DCMA to provide their surveillance. Therefore, they often hire outside organizations or establish an 

independent surveillance function, such as an inspector general. Without an independent surveillance 

function, agencies’ abilities to use EVM as intended may be hampered, since surveillance monitors 

problems with the performance measurement baseline and EVM data. If these kinds of problems go 

undetected, EVM data may be distorted and may not be meaningful for decision making.

Developing a Corporate Surveillance Plan

An annual corporate-level surveillance plan should contain a list of programs for review. The plan’s 

objective is to address, over the course of the year, the question of whether the contractor is applying the 

full content of its EVM system relative to the 32 ANSI guidelines. The surveillance organization therefore 

should have the utmost flexibility to schedule its reviews so as not interfere with major program events. 

Surveillance findings may also rely on the results of other related reviews, such as reviews by DCMA or 

DCAA or other external organizations. Table 41 lists the key processes for each of the 32 ANSI guidelines. 
278 Chapter 20 GAO-09-3SP

Table 41: Key EVM Processes across ANSI Guidelines for Surveillance

Process Applicable ANSI guideline

Organizing 1, 2, 3, 5

Scheduling 6, 7

Work and budget authorization 8, 9, 10, 11, 12, 14, 15

Accounting 16, 17, 18, 20, 22, 30

Indirect management 4, 8, 13, 19, 24, 27

Managerial analysis 22, 23, 25, 26, 27

Change incorporation 28, 29, 30, 31, 32

Material management 21 (9, 10, 12, 22, 23, 27)

Subcontract management (2, 9, 10, 12, 16, 22, 23, 27)

Source: DCMA.

Note: Guidelines in parentheses are cross process guidelines.

In addition to addressing the 32 ANSI guidelines, senior management may ask the surveillance 

organization to focus its review on specific procedures arising from government program office concerns, 

interest in a particular process application, or risks associated with remaining work. This enables the 

surveillance organization to concentrate on processes that are the most relevant to the program phase. For 

example,

■ a surveillance review of the change incorporation process would be more appropriate for a program 

in which a new baseline had recently been implemented than for a program that had just started 

and had not undergone any changes (reviewing the work authorization process would be more 

beneficial);

■ a surveillance review of the EAC process would yield better insight to a development program 

in which technological maturation was the force behind growing EAC trends than it would to a 

production program that had stable EAC trends; 

■ although the goal is to review all 32 ANSI guidelines each year, if a program were almost complete, 

it would not make sense to focus on work authorization, since this process would not then be 

relevant. 

In line with the approach for selecting EVM processes to concentrate on, the surveillance organization 

should select candidate programs by the risk associated with completing the remaining work, so that 

surveillance can be value-added. To facilitate selection, it is important to evaluate the risks associated with 

each program. Table 42 outlines some risk factors that may warrant program surveillance.

Table 42: Risk Factors That Warrant EVM Surveillance

Risk factor Description

Baseline resets Programs experiencing frequent baseline resets need additional monitoring, 

since they often result from poor planning or a change in work approach that 

is causing significant schedule or technical challenges; surveillance of change 

control and EAC benefits such programs by ensuring that changes are 

correctly implemented and EVM data are reliable for making EAC projections
GAO-09-3SP Chapter 20 279

Risk factor Description

Contract phase and type Development contracts tend to be higher-risk and are therefore often good 

candidates for surveillance; production or follow-on contracts are usually 

lower-risk and therefore benefit less from surveillance

Contract value The higher the contract dollar value, the more appropriate the program for 

frequent EVM surveillance

Significant cost or schedule 

variance 

Programs with significant, unfavorable cost or schedule variances should 

be reviewed often; surveillance can help identify problems with baseline 

planning that may give insight into how to take effective corrective action

Nature of remaining work The technical content of remaining work should be reviewed to ensure 

that the most value-added EVM processes and guidelines are selected for 

surveillance

Volume or amount of 

remaining work

New efforts tend to benefit more from surveillance than those that are near 

completion

Program office experience Program office experience in implementing and using EVM processes 

may influence its selection of programs to survey; program offices lacking 

experience may implement the processes incorrectly, increasing the risk of 

generating unreliable program data

Time since last review If it has been a long time since the last surveillance review, the program 

should be selected for surveillance

Findings or concerns from 

prior reviews

Results from prior surveillance reviews may justify additional monitoring

Effectiveness of suppliers’ and 

subcontractors’ surveillance 

process

How well a program’s supplier or subcontractor implements its EVM process 

may influence the selection of programs to review.

Source: © 2004 National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC), Program Management Systems Committee Surveillance 

Guide (October 2004).

Using an algorithm that assigns relative weights and scales to each risk area and classifies risk as low, 

medium, or high can help determine which programs would most benefit from surveillance. Table 43

shows how an algorithm can be used to evaluate a candidate program.

Table 43: A Program Surveillance Selection Matrix

Risk factor Weight

Risk level Risk 

High = 3 Medium = 2 Low = 1 score

Contract value 0.05 More than 20% of 

business base

5%–20% Less than 5% 3

Nature of work 0.05 High-risk, many 

unknowns

Low-risk content 3

Program office 

experience

0.05 Inexperienced staff Very experienced staff 1

Program type 0.05 Development Production Operations and 

maintenance

3

Baseline resets 0.10 Many per year Once a year Less than one a year 3

Historic trends 0.10 Worsening Trends are improving 3

Previous findings 0.10 Many unresolved Few or easily closed 1
280 Chapter 20 GAO-09-3SP

Risk factor Weight

Risk level Risk 

High = 3 Medium = 2 Low = 1 score

Variance percent 0.10 Worse than –10% –5% to –10% Better than –5% 3

Management 

interest

0.40 High visibility Low visibility 3

Total 2.6

Source: © 2004 National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC), Program Management Systems Committee Surveillance 

Guide (October 2004).

For the sample program assessed in the algorithm in table 43, we can quickly determine that it is a highrisk program because it received a risk score of 2.6 of a possible 3.0. This risk is determined by the fact that 

the program has high contract value, the work is high-risk, and high variances had led to several baseline 

resets. Once a risk score has been calculated for all candidate programs, the scores can be used to decide 

which programs should be reviewed more often. The number of programs that can be reviewed each year, 

however, depends on available resources.

Developing a Program Surveillance Plan

The surveillance team designated to perform program reviews should consist of a few experienced staff 

who fully understand the contractor’s EVM system and the processes being reviewed. The surveillance 

organization should appoint the team leader and ensure that all surveillance team members are 

independent. This means that they should not be responsible for any part of the programs they assess.

Key activities on the surveillance team’s agenda include reviewing documents, addressing government 

program office concerns, and discussing prior surveillance findings and any open issues. Sufficient time 

should be allocated to all these activities to complete them. The documents for review should give the 

team an overview of the program’s implementation of the EVM process. Recommended documents 

include 

■ at least 2 months of program EVM system reports;

■ EVM variance analyses and corrective actions;

■ program schedules;

■ risk management plan and database;

■ program-specific instructions or guidance on implementing the EVM system;

■ WBS with corresponding dictionary;

■ organizational breakdown structure;

■ EAC and supporting documentation;

■ correspondence related to the EVM system;

■ contract budget baseline, management reserve, and undistributed budget log;

■ responsibility assignment matrix identifying control account managers;

■ work authorization documentation;

■ staffing plans;
GAO-09-3SP Chapter 20 281

■ rate applications used; and

■ findings from prior reviews and status.

Additionally, it is recommended that if there are any concerns regarding the validity of the performance 

data, the government program office be notified. Finally, inconsistencies identified in prior reviews should 

be discussed to ensure that the contractor has rectified them and continues to comply with its EVM 

system guidelines. 

Executing the Program Surveillance Plan

Surveillance should be approached in terms of mentoring or coaching the contractor on where there are 

deficiencies or weaknesses in its EVM process and offering possible solutions. The contractor can then 

view the surveillance team as a valuable and experienced asset to determine whether it can demonstrate 

that it is continuing to use the accepted EVM system to manage the program. 

Successful surveillance is predicated on access to objective information that verifies that the program team 

is using EVM effectively to manage the contract and complies with company EVM procedures. Objective 

information includes program documentation created in the normal conduct of business. Besides collecting 

documentation, the surveillance team should interview control account managers and other program staff 

to see if they can describe how they comply with EVM policies, procedures, or processes. During interviews, 

the surveillance team should ask them to verify their responses with objective program documentation 

such as work authorizations, cost and schedule status data, variance analysis reports, and back-up data for 

any estimates at completion. Finally, to ensure a common exposure to the program’s content and quicker 

consolidation of findings, the surveillance team should stay together as much as possible.

The interview is a key review effort because it enables the surveillance team to gauge the EVM knowledge 

of the program staff. This is especially important because control account managers are the source of 

much of the information on the program’s EVM system. Interviews also enable the surveillance team to 

monitor program personnel’s awareness of and practice in complying with EVM guidelines. In particular, 

interviews help the surveillance team determine whether the control account managers see EVM as an 

effective management tool. The following subjects should be covered in an interview: 

■ work authorization;

■ organization;

■ EVM methodologies, knowledge of the EVM process, use of EVM information, and EVM system 

program training;

■ scheduling and budgeting, cost and schedule integration, and cost accumulation;

■ EACs;

■ change control process;

■ variance analysis;

■ material management;

■ subcontract management and data integration; and

■ risk assessment and mitigation.
282 Chapter 20 GAO-09-3SP

When all the documentation has been reviewed and interviews have been conducted, the surveillance 

team should provide appropriate feedback to the program team. Specifically, surveillance team members 

and program personnel should clarify any questions, data requests, and responses to be sure everything 

is well understood. The surveillance team leader should present all findings and recommendations to the 

program staff so that any misunderstandings can be clarified and corrected. In addition, a preliminary 

report should be prepared, once program personnel have provided their preliminary feedback, that 

addresses findings and recommendations: 

Findings fall into two broad categories: compliance with the accepted EVM system description and 

consistency with EVM system guidelines. Local practices may be compliant with the system description, 

while others may fall short of the intent of an EVM guideline because of discrepancies in the system 

description. If findings cannot be resolved, confidence in program management’s ability to effectively 

use the EVM system will be lowered, putting the program at risk of not meeting its goals and objectives. 

Open findings may also result in withdrawing advance agreements and acceptance of the company’s 

EVM system.

Team members may recommend EVM implementation enhancements, such as sharing successful 

practices or tools. Unlike findings, however, recommendations need not be tracked to closure.

In addition to findings and recommendations, the final team report should outline an action plan 

that includes measurable results and follow-up verification, to resolve findings quickly. It should 

present the team’s consensus on the follow-up and verification required to address findings resulting 

from the surveillance review. An effective corrective action plan must address how program personnel 

should respond to each finding and it must set realistic dates for implementing corrective actions. The 

surveillance review is complete when the leader confirms that all findings have been addressed and closed.

Managing System Surveillance Based on Program Results

After a program’s surveillance is complete, the results are collected and tracked in a multiprogram 

database. This information is transformed into specific measures for assessing the overall health of a 

contractor’s EVM system process. They should be designed to capture whether the EVM data are readily 

available, accurate, meaningful, and focused on desirable corrective action. The types of measure may 

vary from contractor to contractor, but each one should be well defined, easily understood, and focused on 

improving the EVM process and surveillance capability. They should have the following characteristics:

■ surveillance results measures identify where there are deviations from documented EVM 

application processes and

■ system surveillance measures are EVM system process measures that indicate whether the 

surveillance plan is working by resolving systemic issues.

To develop consistent measures, individual program results can be summarized by a standard rating 

system that uses color categories to identify findings. Table 44 shows a standard color-category rating 

system.
GAO-09-3SP Chapter 20 283

Table 44: A Color-Category Rating System for Summarizing Program Findings 

Related 

to

EVM system rating

Low = green Moderate = yellow High = red

Organization 

1 One WBS used and authorized 

for the program

One WBS used for the program More than one WBS used 

for the program

2 WBS dictionary available and 

traceable to the contract WBS 

and statement of work 

WBS dictionary available but 

cannot be traced to the contract 

WBS and is inconsistent with the 

statement of work 

WBS dictionary not 

developed

3 Organizational breakdown 

system, including major 

subcontractors, defined

More than one organizational 

breakdown system used; not all 

are identified or some contain 

errors or omissions

Organizational breakdown 

system not defined 

4 Program WBS and organizational 

breakdown system integrated 

and identified by the 

responsibility assignment matrix 

Program WBS and organizational 

breakdown system identified but 

the responsibility assignment 

matrix is incomplete or outdated

Responsibility assignment 

matrix process is not 

implemented

Budget 

1 Budgets for authorized work 

identified

Budgets for authorized work 

have omissions

Budgets for authorized 

work not developed

2 Sum of work package budgets 

equals control account budgets; 

appropriate EVM techniques 

deployed

Sum of work package budgets 

equals control account budgets, 

but appropriate EVM techniques 

not applied

Sum of work package 

budgets does not equal 

control account budgets

3 Management reserve and 

undistributed budget identified; 

management reserve not used 

for cost growth or contract 

changes

Management reserve and 

undistributed budget identified 

but do not adequately cover 

existing program scope and risk

Management reserve used 

for cost growth or contract 

changes 

4 Time-phased budget established, 

against which performance can 

be measured

Not applicable Baseline cannot be used 

for accurate performance 

measurement

5 Authorized work identified in 

measurable units 

Authorized work identified 

in measurable units but has 

omissions

Authorized work not 

identified in measurable 

units 

Source: © 2004 National Defense Industrial Association (NDIA) Program Management Systems Committee (PMSC), Program Management Systems Committee Surveillance 

Guide (October 2004).

Summarizing individual program findings by a standard measure can help pinpoint systemic problems 

in a contractor’s EVM system and can therefore be useful for highlighting areas for correction. This 

may result in more training or changing the EVM system description to address a given weakness by 

improving a process. Without the benefit of standard measures, it would be difficult to diagnose systemic 

problems; therefore, it is a best practice to gather them and review them often.
284 Chapter 20 GAO-09-3SP

Overtarget Baselines and Schedules

At times, an organization may conclude that the remaining budget and schedule targets for completing 

a program are significantly insufficient and that the current baseline is no longer valid for realistic 

performance measurement. The purpose of an overtarget baseline or overtarget schedule is to restore 

management’s control of the remaining effort by providing a meaningful basis for performance 

management. Working to an unrealistic baseline could make an unfavorable cost or schedule condition 

worse. 

For example, if variances become too big, they may obscure management’s ability to discover newer 

problems that could still be mitigated. To quickly identify new variances, an overtarget baseline normally 

eliminates historic variances and adds budget for future work. The contractor then prepares and submits a 

request to implement a recovery plan—in the form of an overtarget baseline or overtarget schedule—that 

reflects the needed changes to the baseline. 

The Rebaseline Rationale

The focus during a rebaseline is ensuring that the estimated cost of work to complete is valid, remaining 

risks are identified and tracked, management reserve is identified, and the new baseline is adequate and 

meaningful for future performance measurement. 

An overtarget baseline is established by formally reprogramming the performance measurement baseline 

to include additional budget that is above and beyond the contract’s negotiated cost.80 This additional 

budget is believed necessary to finish work that is in process and becomes part of the recovery plan for 

setting new objectives that are achievable. 

An overtarget baseline does not always affect all remaining work in the baseline; sometimes only a portion 

of the WBS needs more budget. Similarly, an overtarget baseline may or may not reset cost and schedule 

variances, although in most cases the variances are eliminated. 

An overtarget baseline or overtarget schedule should be rare. Therefore, if a program is experiencing 

recurrent overtarget baselines, it may be that the scope is not well understood or simply that program 

management lacks effective EVM discipline and is unable to develop realistic estimates. 

Moreover, a program that frequently changes its baseline can appear to be trying to “get well” by 

management’s hiding its real performance, leading to distorted EVM data reporting. When this happens, 

decision makers tend to lose confidence in the program, as evidenced in case study 48.

80This action is not to be confused with reprogramming agency appropriations. In that context, reprogramming is a shifting 

of funds within an appropriation or fund account to use them for purposes other than those contemplated at the time of the 

appropriation. (See GAO, A Glossary of Terms Used in the Federal Budget Process, GAO-05-734SP (Washington, D.C.: Sept. 1, 

2005), p. 85.) The overtarget baseline action should also not be confused with replanning—that is, the replanning of actions for 

remaining work scope, a normal program control process accomplished within the scope, schedule, and cost objectives of the 

program.
GAO-09-3SP Chapter 20 285

Case Study 48: Maintaining Realistic Baselines, from Uncertainties 

Remain, GAO-04-643R

From the contract’s award in 1996 to 2003, the cost of the Airborne Laser’s (ABL) research 

and development contract increased from about $1 billion to about $2 billion. In fiscal 

year 2003 alone, work the contractor completed cost about $242 million more. Besides 

these cost overruns, the contractor was unable to complete $28 million worth of work 

planned for the fiscal year. GAO estimated from the contractor’s 2003 cost and schedule 

performance that the prime contract would overrun by $431 million to $943 million. 

The program had undergone several major restructurings and contract rebaselines from 

1996 on, primarily because of unforeseen complexity in manufacturing and integrating 

critical technology. According to program officials, rapid prototyping resulted in limited 

subcomponent testing, causing rework and changing requirements. At the time of 

GAO’s review, the program faced massively increasing amounts of incomplete work from 

previous years, even though the prime contractor had increased the number of people 

devoted to the program and had added shifts to bring the work back on schedule. In 

addition, unanticipated difficulties in software coding and integration, as well as difficulty 

in manufacturing advanced optics and laser components, caused cost growth.

Good investment decisions depend on understanding the total funds needed to obtain 

an expected benefit, but the Missile Defense Agency (MDA) had been unable to assure 

decision makers that its cost projections to complete technology development could 

be relied on. Decision makers would have been able to make more informed decisions 

about further program investments if they had understood the likelihood and confidence 

associated with MDA’s cost projections. Therefore, GAO recommended that MDA complete 

an uncertainty analysis of the contractor’s new cost estimate.

GAO, Uncertainties Remain Concerning the Airborne Laser’s Cost and Military 

Utility, GAO-04-643R (Washington, D.C.: Mar. 17, 2004).

The end result of an overtarget baseline is that its final budget always exceeds the contract budget 

base, which includes the negotiated contract cost plus any authorized, unpriced work. In EVM system 

terminology, the sum of all budgets (performance measurement baseline, undistributed budget, and 

management reserve) that exceed the contract budget base is known as total allocated budget, and 

the difference between the total allocated budget and contract budget base is the overtarget baseline. 

Figure 40 illustrates the effect an overtarget baseline has on a contract.

Figure 40: The Effect on a Contract of Implementing an Overtarget Budget

Total allocated budget

Contract budget base

Performance measurement baseline Management reserve

After overrun

Total allocated budget

Contract budget base Overtarget budget

Performance measurement baseline Management reserve

Before overrun

Source: DCMA.
286 Chapter 20 GAO-09-3SP

Like an overtarget budget, an overtarget schedule occurs when the schedule and its associated budgets 

are spread over time and work ends up being scheduled beyond the contract completion date. The new 

schedule becomes the basis for performance measurement. Typically, an overtarget schedule precipitates 

the need for an overtarget budget, because most increases in schedule also require additional budget.

As mentioned above, the contractor submits an overtarget budget and overtarget schedule request to the 

government program office for evaluation. It should contain the following key elements:

■ an explanation of why the current plan is no longer feasible, identifying the problems that led to 

the need to make a new plan of the remaining work and discussing measures in place to prevent 

recurrence;

■ a bottoms-up estimate of remaining costs and schedule that accounts for risk and includes 

management reserve;

■ a realistic schedule for remaining work that has been validated and spread over time to the new plan; 

■ a report on the overtarget budget in the CPR—the government program office needs to come 

to an agreement with the contractor on how it is to be reported in the CPR, how decisions are to 

be made on handling existing cost and schedule variances, and how perspectives on new budget 

allocations will be reported (whether variances are to be retained or eliminated or both); 

■ the overtarget budget’s implementation schedule, to be accomplished as soon as possible once 

approved; usually, it is established in one to two full accounting periods, with reporting continuing 

against the existing baseline in the meantime.

In determining whether implementing an overtarget budget and overtarget schedule is appropriate, 

the program office should consider the program’s health and status and should decide whether the 

benefits outweigh the costs. An overtarget budget should be planned with the same rigor as planning 

for the original program estimate and performance measurement baseline. While overtarget budget and 

overtarget schedule can restore program confidence and control by establishing an achievable baseline, 

with meaningful performance metrics, the time and expense required must be carefully considered. 

Contract type is a key factor to consider when rebaselining a program, because each contract has its own 

funding implications when an overtarget budget is implemented. Table 45 describes two common types of 

contracts and considerations for overtarget budget implementation.

Table 45: Overtarget Budget Funding Implications by Contract Type

Contract type Description Considerations

Fixed price 

incentive

Negotiated target cost plus 

estimated cost of authorized 

unpriced work equals the cost 

of the contract budget base; 

government program office 

liability is established up to a 

specified ceiling price

■ Although additional performance budget is allocated 

to the performance measurement baseline, the 

overtarget budget does not change the customer’s 

funding liability or any contract terms; the contractor 

has liability for a portion of costs above target and all 

actual costs over the ceiling price, because the work’s 

scope has not changed and the contract has not been 

modified

■ An overtarget budget is established on a fixed-price 

incentive contract without regard to profit, cost 

sharing, or ceiling implications
GAO-09-3SP Chapter 20 287

Contract type Description Considerations

Cost 

reimbursement

Provides for payment of 

allowable incurred costs to 

the contractor to the extent 

provided in the contract 

and, where included, for 

contractor’s fee or profit; the 

new contract budget base is 

based on the updated cost 

target 

■ The customer must be notified of the need for an 

overtarget budget, having agreed to pay for actual 

costs incurred to the extent provided in the contract; 

he may have to commit or seek additional funds to 

address the changing program condition and must 

therefore be aware of and involved in the overtarget 

budget implementation

■ While the government normally has full cost 

responsibility if this is a cost plus incentive fee contract, 

the contractor may lose the fee

■ A cost growth contract modification results in 

obligating additional funds to cover in-scope 

effort; this involves real dollars, so the performance 

measurement budget does not increase and the cost 

growth variance continues to be reported in the CPR; 

when a contract modification includes a new scope, 

the modification should clearly state the portion of 

the new estimated cost that is for new scope and the 

portion that is to provide funds for an acknowledged 

cost overrun

Source: GAO and Ivan Bembers and others, Over Target Baseline and Over Schedule Handbook (n.p., n.p.: May 7, 2003), p. 7. 

The program office and the contractor should also consider whether losing valuable historic performance 

variances and trends is worth the effort and time to reset the baseline. Table 46 identifies common 

problems and indicators that may be warning signs that a program may need an overtarget budget or 

schedule. 

Table 46: Common Indicators of Poor Program Performance

Indicator Description

Cost ■ Significant difference between Estimated cost to complete and budget for remaining work 

differ significantly

■ Significant difference between cumulative CPI and TCPI

■ Significant lack of confidence in the EAC

■ Frequent allocation of management reserve to the performance measurement baseline for 

newly identified in-scope effort

■ Inadequate control account budgets for remaining work

■ Work packages with no budget left

■ No reasonable basis for achieving the EAC

■ EACs that are too optimistic and do not adequately account for risks

Schedule ■ A great many concurrent activities in the integrated schedule 

■ Significant negative float in the integrated schedule’s critical path

■ Unrealistic activity durations 

■ Unrealistic logic and relationships between tasks 

■ Significant number of activities with constrained start or finish dates 

■ No horizontal or vertical integration in the schedule 

■ No basis for schedule reserve reductions except to absorb the effect of schedule delays 

Project 

execution risk

■ Risk management analysis that shows significant changes in risk levels 

■ Lack of correlation between budget phases and baseline schedule 

■ No correlation between estimate to complete time periods and current program schedule 

■ Program management’s reliance on ineffective performance data
288 Chapter 20 GAO-09-3SP

Indicator Description

Data accuracy ■ Frequent or significant current or retroactive changes 

■ Actual costs exceeding the EAC 

■ Work scope transferred without associated budget

■ An apparently front-loaded performance measurement baseline 

■ Inadequate planning for corrective action 

■ Repetitive reasons for variances 

■ No reflection of progress in earned value 

■ Late booking of actual costs that cause lagging variances

■ Frequent data errors 

Source: Ivan Bembers and others, Over Target Baseline and Over Schedule Handbook (n.p., n.p.: May 7, 2003).

Establishing a revised performance measurement baseline to incorporate significant variances should be 

a major wake-up call for program management, sending a serious message about the amount of risk a 

program is undertaking. Therefore, in conjunction with evaluating the indicators in table 46, program 

management should consider other aspects before deciding to implement an overtarget budget and 

schedule.

Work Completion Percentage

The contract should typically be 20 percent to 85 percent complete. A contract that is less than 20 percent 

complete may not be mature enough yet to benefit from the time and expense of implementing overtarget 

budget and schedule. A contract that is more than 85 percent complete gives management limited time to 

significantly change the program’s final cost.

Projected Growth

A projected growth of more than 15 percent may warrant an overtarget budget and schedule. The 

projection is made by comparing the estimated time of completion with the budget allocated for the 

remaining work. An overtarget budget’s most important criterion is whether it is necessary to restore 

meaningful performance measurement.

Remaining Schedule

If less than a year is required to complete the remaining work, the benefit of overtarget budget and 

schedule will most likely be negligible because of the time it typically takes to implement the new baseline.

Benefit Analysis

A benefit analysis should determine whether the ultimate goal of implementing overtarget budget and 

overtarget schedule gives management better control and information. With this analysis, the government 

program office and contractor should ensure that the benefits will outweigh the cost in both time and 

resources. If better management information is expected and the program team is committed to managing 

within the new baseline, then it should be implemented.

Rebaselining History

Several overtarget budget requests have suggested severe underlying management problems. These should 

be investigated before implementing a new budget.
GAO-09-3SP Chapter 20 289

Key Steps of the Overtarget Budget–Overtarget Schedule Process

While it is the primary responsibility of the contractor to ensure that a meaningful performance 

measurement baseline is established, every control account manager must develop new work plans that 

can be reasonably executed. The program manager and supporting business staff must have open lines of 

communication and a clear review process to ensure that the baseline is reasonable and accurate, reflecting 

known risks and opportunities.

Thus, overtarget budget–overtarget schedule implementation involves multiple steps and processes toward 

establishing a new performance measurement baseline, illustrated in figure 41.

Figure 41: Steps Typically Associated with Implementing an Overtarget Budget 

Consult with

customer

Consult with

customer

Consult with

customer

Statement of need

for OTB

Develop revised

integrated master

schedule

Consensus on

remaining scope

Schedule review

and concurrence

Revise detail

schedules and prepare

estimates to complete

Issue guidance to

revise cost

account plans

Input estimate to

complete into

EVM system

Make final OTB

cost and schedule

Control account

manager reviews and

estimate to complete

“scrubbing”

Senior management

review cost

and schedule

Establish new PMB

Source: Ivan Bembers and others, Over Target Baseline and Over Schedule Handbook (n.p., n.p., 2003).

The key steps we describe here include (1) planning the approach, (2) developing the new schedule and 

making new cost account plans, and (3) senior management’s reviewing the costs and schedule. Each step 

assumes early involvement and frequent interaction between the contractor and government program 

office. 

Planning the Overtarget Budget–Overtarget Schedule Approach

When developing a plan for an overtarget budget, certain factors should be considered:

■ What issues or problems resulted in the need for one? How will the new plan address them?

■ Can the overtarget budget be accomplished within the existing schedule? If not, then an overtarget 

schedule must also be performed. Conversely, does an overtarget schedule require an overtarget 

budget or can the schedule be managed within the existing budget? 
290 Chapter 20 GAO-09-3SP

■ How realistic is the estimate to complete? Does it need to be updated?

■ Are cost and schedule variances being eliminated or retained? Will future reporting include 

historical data or begin again when the new plan is implemented?

■ What is the basis for the overtarget budget management reserve account? Is it adequate for the 

remaining work?

■ To what extent are major subcontractors affected by the overtarget budget? How will it affect their 

target cost and schedule dates?

■ Were any EVM system discipline issues associated with the need for an overtarget budget? If so, 

how were they resolved?

If the new baseline is to provide management with better program status, a decision about whether to 

eliminate variances will have to be made. A single point adjustment—that is, eliminating cumulative 

performance variances, replanning the remaining work, and reallocating the remaining budget to establish 

a new performance measurement baseline—results in a new performance measurement baseline that 

reflects the plan of the remaining work and budget. Since existing variances can significantly distort 

progress toward the new baseline, a single point adjustment is a common and justifiable adjunct to an 

overtarget budget. Table 47 describes options for treating historical cost and schedule variances when 

performing a single point adjustment. 

Table 47: Options for Treating Variances in Performing a Single Point Adjustment

Variance option Description

Eliminate

All variances Eliminate cost and schedule variances for all WBS elements by setting BCWS and 

BCWP equal to ACWP; the most common type of variance adjustment, this normally 

generates an increase in BCWP and sometimes results in an adjustment to BCWS

Schedule variance only Cost variance is considered a valid performance measurement; the new performance 

measurement baseline retains the cost variance history but eliminates schedule 

variance by setting BCWS equal to BCWP, allowing revised planning for the 

remaining work and budgets

Cost variance only When, infrequently, cost variance impels an overtarget budget but schedule 

information is valid, variance is eliminated by setting BCWP equal to ACWP; the 

cumulative BCWP value is adjusted to match the cumulative cost variance. To 

preserve the existing schedule variance, the cumulative BCWS should be changed by 

the same amount as the BCWP; the CPR will reflect positive adjustments to both in 

the current period following the overtarget budget

Selected variances If one WBS element or a subcontractor shows performance out of line with the 

baseline, management may implement an overtarget budget for only that portion of 

the contract; all other variances remain intact

Retain

All variances A contractor may have been performing fairly well to the baseline plan with no 

significant variances, but additional budget is necessary to complete the work; or the 

contractor has large variances warranting an overtarget budget, but management 

wants to retain them. In both situations, cost and schedule variances are left alone 

but budget is added to cover future work in the overtarget budget process

Source: Ivan Bembers and others, Over Target Baseline and Over Schedule Handbook (n.p., n.p.: May 7, 2003).
GAO-09-3SP Chapter 20 291

It is important to understand that while cost and schedule variances can be adjusted in various ways, 

under no circumstances should the value of ACWP be changed in the overtarget budget process. The 

value of ACWP should always be reconcilable to the amount shown in the contractor’s accounting records. 

In addition, management reserve to be included in the final overtarget budget should be addressed in the 

overtarget budget planning step: The amount will depend on how much work and risk remain. Historic 

management reserve consumption before the overtarget budget may offer important insights into the 

amount to set aside. The bottom line is that a realistic management reserve budget should be identified 

and available for mitigating future risks. These two issues—keeping ACWP integrity and setting aside 

adequate management reserve—must be considered in making the new plan, regardless of whether the 

single point adjustment option is used. Figure 42 shows how a single point adjustment results in a change 

to the performance measurement baseline.

Figure 42: Establishing a New Baseline with a Single Point Adjustment

Time

0

100

200

300

400

Program budget (resources)

Original budget

Time now

Revised schedule

Management

reserve

Eliminates variances

Establishes new baseline, including reserve if needed

Single point adjustment

•

• ACWP

BCWS

BCWP

Revised budget

Dec 00 Mar 01 Jun 01 Sept 01 Dec 01 Mar 02 Jun 02 Sept 02

Source: Abba Consulting.

New baseline BCWS

In figure 42, the performance measurement baseline—that is, BCWS—is shifted upward to align with 

actual costs to date—that is, with ACWP. The new baseline continues from this point forward, and all 

new work performed and corresponding actual costs will be measured against this new baseline. The 

revised budget is also at a higher level than the original budget; the schedule has slipped 4 months from 

May to September. Finally, all variances up to the overtarget budget date have been eliminated and the 

management reserve amount has risen above the new performance measurement baseline. 

As work is performed against this new baseline, reliable performance indicators can be used to identify 

problems and implement corrective actions. However, because all variances have been eliminated, it 

may take several months after the single point adjustment for trends to emerge against the new baseline. 
292 Chapter 20 GAO-09-3SP

During the next few months, monitoring the use of management reserve can help show whether realistic 

budgets were estimated for the remaining work or new risks occurred after the overtarget budget.

A note of caution: single point adjustments should not be made regularly and not solely to improve 

contract performance metrics—especially when attempting to meet OMB’s “Get to Green” capital 

planning initiative to show favorable program performance status. Because a single point adjustment 

masks true performance, frequent use tends to cause varied and significant problems such as 

■ distorting earned value cost and schedule metrics, resulting in unreliable index-based EAC 

calculations;

■ turning attention away from true cost and schedule variances; and

■ hindering the ability of EVM data to predict performance trends. 

In other words, single point adjustments should be used sparingly in order not to inhibit successful use of 

EVM information to manage programs.

Planning the New Schedule and Control Accounts

Even if only an overtarget budget is required, some level of schedule development or analysis should always 

be performed. The revised schedule should be complete, integrated, realistic in length, and coordinated 

among key vendors and subcontractors. Further, the schedule logic and activity durations should be 

complete and should represent the effort associated with the remaining work. Any effect on government 

furnished equipment schedules or availability of government test ranges should also be considered before 

the schedule is validated and considered realistic. 

The government program office and the contractor should review, and come to a mutual understanding 

of, the remaining scope, resources, and risk in the new schedule. They should agree that it is integrated 

vertically and horizontally, task durations are backed by historic data, schedule reserve is adequate, and 

achieving the overall schedule is likely. 

Once the revised schedule for the remaining work has been established, it is used to determine the budget 

for the remaining cost accounts. A detailed estimate to complete the remaining work should be based on 

a bottom-up estimate to reflect all costs—staffing, material, travel. Control account managers should also 

consider the remaining cost and schedule risk and their probability.

Senior Management Review of Cost and Schedule

While an overriding goal of the overtarget budget–overtarget schedule process is to allow the contractor 

to implement an effective baseline in a timely manner, the government program office plays a key role 

in determining whether the contract can be executed within the constraints of program funding and 

schedule. Three key activities the government program office should consider in the final review of the 

new baseline are

1. perform an IBR to verify that the value and associated schedule determined in the overtarget 

budget–overtarget schedule process have been established in the new baseline;

2. determine to what extent EVM reporting requirements will be suspended or reduced, given the 

time needed to implement the new baseline; a best practice is to continue reporting against the old 
GAO-09-3SP Chapter 20 293

baseline until the new one is established, keeping EVM reporting rhythm in place and maintaining 

a record of the final change;

3. select meaningful performance indicators (such as those in table 46) to monitor contractor efforts 

to implement and adhere to the new baseline. 

One key indicator, management reserve usage, should not be used to a great extent in the near term; 

another is EVM performance trends, although the government program office should be aware of its 

effect on the subsequent trend chart if a single point adjustment was made.

Update the Program Cost Estimate with Actual Costs

Regardless of whether changes to the program result from a major contract modification or an overtarget 

budget, the cost estimate should be regularly updated to reflect all changes. Not only is this a sound 

business practice; it is also a requirement outlined in OMB’s Capital Programming Guide.

81 The purpose 

of updating the cost estimate is to check its accuracy, defend the estimate over time, shorten turnaround 

time, and archive cost and technical data for use in future estimates. After the internal agency and 

congressional budgets are prepared and submitted, it is imperative that cost estimators continue to monitor 

the program to determine whether the preliminary information and assumptions remain relevant and 

accurate. 

Keeping the estimate fresh gives decision makers accurate information for assessing alternative decisions. 

Cost estimates must also be updated whenever requirements change, and the results should be reconciled 

and recorded against the old estimate baseline. Several key activities are associated with updating the cost 

estimate:

■ documenting all changes that affect the overall program estimate so that differences from past 

estimates can be tracked; 

■ updating the estimate as requirements change, or at major milestones, and reconciling the results 

with the program budget and EVM system;

■ updating the estimate with actual costs as they become available during the program’s life cycle;

■ recording reasons for variances so that the estimate’s accuracy can be tracked;

■ recording actual costs and other pertinent technical information—source line of code sizing, effort, 

schedule, risk items—so they can be used for estimating future programs; and

■ obtaining government program office feedback, assessing lessons learned on completion, and 

recording the lessons so they are available for the next version of the estimate. 

After these activities are completed, the estimator should document the results in detail, including reasons 

for all variances. This critical step allows others to track the estimates and to identify when, how much, 

and why the program cost more or less than planned. Further, the documented comparison between the 

current estimate (updated with actual costs) and old estimate allows the cost estimator to determine the 

level of variance between the two estimates. In other words, it allows estimators to see how well they are 

estimating and how the program is changing over time.

81OMB, Capital Programming Guide: Supplement to Circular A-11, Part 7, Preparation, Submission, and Execution of the Budget. 
294 Chapter 20 GAO-09-3SP

Keep Management Updated

Part of agency capital planning and investment control is reporting updated program EACs to 

management during senior executive program reviews. With EVM data, a variety of EACs can be 

generated solely for this purpose. In addition, continuous management reviews of the EVM data not 

only allow insight into how a specific program is performing but also help depict a company’s financial 

condition accurately for financial reporting purposes. 

EVM data provide a clear picture of what was scheduled, accomplished, and spent in a given month so 

that program status can be known at any time. Likewise, cost and schedule performance trends derived 

from the CPR are objective data that allow management to identify where potential problems and cost 

overruns can occur. This information should be presented at every program manager review, since it is 

essential for managing a program effectively. 

DOD requires in addition that contractors submit a quarterly contract funds status report that provides 

time-phased funding requirements and execution plans and identifies requirements for work agreed-to 

but not yet under contract. Other agencies require a similar document. For example, NASA requires form 

533 that reports data necessary for projecting costs and hours to ensure that resources realistically support 

program schedules. It also evaluates contractors’ actual cost and fee data and compares them to the 

negotiated contract value, estimated costs, and budget forecast data. 

Data from the DOD report or a similar report are important for knowing whether the government 

has adequate funding to complete the program, based on the contractor’s historic performance trends. 

Therefore, both that report and the CPR should be used regularly to monitor contractor performance and 

update the cost estimate. Doing so will provide valuable information about problems early on, when there 

is still time to act. It also makes everyone more accountable and answerable to basic program management 

questions, such as

■ Can the EVM data be trusted? 

■ Is there really a problem? 

■ How much risk is associated with this program? 

■ What is causing a problem and how big is it? 

■ Are other risks associated with this problem? 

■ What is likely to happen? 

■ What are the alternatives? 

■ What should the next course of action be?

■ Who is responsible for major parts of the contract?

■ What were the major changes since the contract began?

■ How long have similar programs taken?

■ How much work has been completed and when will the program finish? 

■ When should results start materializing?
GAO-09-3SP Chapter 20 295

While EVM offers many benefits, perhaps the greatest benefit of all is the discipline of planning the 

entire program before starting any work. This planning brings forth better visibility and accountability, 

which add clarity to risks as well as opportunities. Further, EVM offers a wealth of data and lessons that 

can be used to project future program estimates. To reap these benefits, however, EVM requires strong 

partnership between the government program office and the contractor to make for a sense of ownership 

and responsibility on both sides. This shared accountability is a major factor in bringing programs to 

successful completion and makes good program management possible.

17. Best Practices Checklist: Managing Program Costs: Updating

 The cost estimate was updated with actual costs, keeping it current and 

relevant. 

3 Actual cost, technical, and schedule data were archived for future 

estimates.

 Authorized changes to the EVM performance measurement baseline were 

incorporated in a timely manner.

3 It reflected current requirements.

3 These changes were incorporated in a documented, disciplined, and 

timely manner so that budget, schedule, and work stayed together for 

true performance measurement.

3 Changes were approved and implemented in a well-defined baseline 

control process.

 Regular EVM system surveillance ensured the contractor’s effective 

management of cost, schedule, and technical performance and compliance 

with ANSI guidelines.

3 The surveillance organization was independent and had authority to 

resolve issues.

3 Surveillance staff had good knowledge about EVM and agency 

programs.

3 An annual surveillance plan was developed and programs were chosen 

objectively.

3 Findings and recommendations were presented to the program team 

for clarification, and the final surveillance report had an action plan to 

resolve findings quickly.

 The contractor’s overtarget baseline or overtarget schedule was detailed, 

reasonable, and realistic; planned for costs, schedule, and management 

review; and described measures in place to prevent another OTB.

 Updated EACs and other EVM data were continually reported to 

management.

 EVM and CFSR–like data were examined regularly to identify problems and 

act on them quickly.
296 GAO-09-3SP
GAO-09-3SP APPENDIX I 297

Auditing Agencies and Their Web Sites

GAO frequently contacts the audit agencies in this appendix at the start of a new audit. This list does not 

represent the universe of audit organizations in the federal government.

Auditing agency Agency’s Web site

Air Force Audit Agency www.afaa.af.mil/

Defense Contract Audit Agency www.dcaa.mil/

District of Columbia, Office of the Inspector General www.oig.dc.gov/main.shtm

Federal Trade Commission, Office of Inspector General www.ftc.gov/oig/

National Aeronautics and Space Administration, Office of 

Inspector General

www.hq.nasa.gov/office/oig/hq/

National Archives, Office of the Inspector General www.archives.gov/oig/

Navy Inspector General ig.navy.mil/

Social Security Administration, Office of the Inspector General www.ssa.gov/oig/

U.S. Army Audit Agency www.hqda.army.mil/aaaweb/

U.S. Department of Commerce, Office of Inspector General www.oig.doc.gov/oig/

U.S. Department of Defense, Office of Inspector General www.dodig.mil/

U.S. Department of Education, Office of Inspector General www.ed.gov/about/offices/list/oig/index.html

U.S. Department of Health and Human Services, Office of 

Inspector General

www.oig.hhs.gov/

U.S. Department of Housing and Urban Development, Office 

of Inspector General

www.hud.gov/offices/oig/

U.S. Environmental Protection Agency, Office of Inspector 

General

www.epa.gov/oigearth/

U.S. General Services Administration, Office of Inspector 

General

oig.gsa.gov/

U.S. House of Representatives, Office of Inspector General www.house.gov/IG/

United States Postal Service, Office of Inspector General www.uspsoig.gov/

Appendixes
298 APPENDIX II GAO-09-3SP

Case Study Backgrounds

We drew the material in the guide’s 48 case studies from the 16 GAO reports described in this appendix. 

Table 48 shows the relationship between reports, case studies, and the chapters they illustrate. The table 

is arranged by the order in which we issued the reports, earliest first. Following the table, paragraphs that 

describe the reports are ordered by the numbers of the case studies in this Cost Guide.

Table 48: Case Studies Drawn from GAO Reports Illustrating This Guide

Case study GAO report Chapters illustrated

2, 5, 18, 30, 35 GAO/AIMD-99-41: Customs Service Modernization 1, 2, 4, 9, 11

47 GAO-03-343: National Airspace System 20

17 GAO-03-645T: Best Practices 5

1, 3, 4, 11, 13, 23 GAO-04-642: NASA 1, 2, 5, 8

48 GAO-04-643R: Uncertainties Remain 20

8, 10, 14, 27, 28, 33,

36, 38, 40, 46

GAO-05-183: Defense Acquisitions 2, 4, 9–11, 13, 14, 19

19, 21, 45 GAO-06-215: DOD Systems Modernization 5, 7, 18

24 GAO-06-296: Homeland Security 8

9 GAO-06-327: Defense Acquisitions 2

7 GAO-06-389: Combating Nuclear Smuggling 2

20 GAO-06-623: United States Coast Guard 7

12, 32, 44 GAO-06-692: Cooperative Threat Reduction 2, 10, 18

6, 16, 25, 26, 29, 31,

34, 37, 39, 42

GAO-07-96: Space Acquisitions 2, 4, 9–12, 14, 15

15 GAO-07-133R: Combating Nuclear Smuggling 4

41 GAO-07-240R: Chemical Demilitarization 15

43 GAO-07-268: Telecommunications 16

22 GAO-08-756: Air Traffic Control 8

Note: Full bibliographic data for the reports in this table (listed in the order in which GAO issued them) are given below their headings in 

this appendix and in the case studies in the text. 

Case Studies 1, 3, 4, 11, 13, and 23: From NASA, GAO-04-642, May 28, 2004

For more than a decade, GAO has identified the National Aeronautics and Space Administration’s 

(NASA) contract management as a high-risk area. Because of NASA’s inability to collect, maintain, and 

report the full cost of its programs and projects, it has been challenged to manage its programs and control 

program costs. The scientific and technical expectations inherent in NASA’s mission create even greater 

challenges—especially if meeting those expectations requires NASA to reallocate funding from existing 

programs to support new efforts.

Because cost growth has been a persistent problem in a number of NASA’s programs, GAO was asked to 

examine NASA’s cost estimating for selected programs, assess its cost estimating processes and methods, 

and describe any barriers to improving its cost estimating processes. Accordingly, in NASA: Lack of 

Disciplined Cost-Estimating Processes Hinders Effective Program Management (May 28, 2004), GAO 

reported its analysis of 27 NASA programs, 10 of which it reviewed in detail.
GAO-09-3SP APPENDIX II 299

Case Studies 2, 5, 18, 30, and 35: From Customs Service Modernization, GAO/

AIMD-99-41, February 26, 1999

Title VI of the 1993 North American Free Trade Agreement Implementation Act, Public Law 103-182, 

enabled the U.S. Customs Service to speed the processing of imports and improve compliance with trade 

laws. Customs refers to this legislation as the Customs Modernization and Informed Compliance Act, or 

“Mod Act.” The act’s primary purpose was to streamline and automate Customs’ commercial operations. 

According to Customs, modernized commercial operations would permit it to more efficiently handle 

its burgeoning import workloads and expedite the movement of merchandise at more than 300 ports of 

entry. Customs estimated that the volume of import trade would increase from $761 billion in 1995 to 

$1.1 trillion through 2001, with the number of commercial entries processed increasing in those years 

from 13.1 million to 20.6 million.

The Automated Commercial Environment (ACE) program was Customs’ system solution to a 

modernized commercial environment. In November 1997, Customs estimated that it would cost $1.05 

billion to develop, operate, and maintain ACE between fiscal year 1994 and fiscal year 2008. Customs 

planned to develop and deploy ACE in increments. The first four were known collectively as the National 

Customs Automation Program (NCAP). The first increment, NCAP 0.1, was deployed for field operation 

and evaluation in May 1998. At the end of fiscal year 1998, Customs reported that it had spent $62.1 

million on ACE. GAO issued its report on these programs, Customs Service Modernization: Serious 

Management and Technical Weaknesses Must Be Corrected, on February 26, 1999.

Case Studies 6, 16, 25, 26, 29, 31, 34, 37, 39, and 42: From Space Acquisitions, 

GAO-07-96, November 17, 2006

Estimated costs for major space acquisition programs in the Department of Defense (DOD) have 

increased about $12.2 billion—or nearly 44 percent—above initial estimates for fiscal years 2006–2011. 

In some cases, current estimates of costs are more than double the original estimates. For example, the 

Space Based Infrared System High program was originally estimated to cost about $4 billion but is now 

estimated to cost over $10 billion. The National Polar-orbiting Operational Environmental Satellite 

System program was originally estimated to cost almost $6 billion but is now over $11 billion. Such 

growth has had a dramatic effect on DOD’s overall space portfolio. 

To cover the added costs of poorly performing programs, DOD has shifted scarce resources away from 

other programs, creating cascading cost and schedule inefficiencies. As a result, GAO was asked to 

examine (1) in what areas space system acquisition cost estimates have been unrealistic and (2) what 

incentives and pressures have contributed to the quality and usefulness of cost estimates for space system 

acquisitions. GAO reported its findings on November 17, 2006, in Space Acquisitions: DOD Needs to Take 

More Action to Address Unrealistic Initial Cost Estimates of Space Systems.

Case Study 7: From Combating Nuclear Smuggling, GAO-06-389, March 22, 2006

Since September 11, 2001, combating terrorism has been one of the nation’s highest priorities. Preventing 

the smuggling of radioactive material into the United States—perhaps for use by terrorists in a nuclear 

weapon or in a radiological dispersal device (a dirty bomb)—has become a key national security objective. 

The Department of Homeland Security (DHS) is responsible for providing radiation detection capabilities 
300 APPENDIX II GAO-09-3SP

at U.S. ports of entry. In September 2003, GAO reported on the department’s progress in completing 

domestic deployments. In particular, GAO found that certain aspects of its installation and use of 

equipment diminished its effectiveness and that agency coordination on long-term research issues was 

limited. 

After GAO issued that report, questions arose about the deployed detection equipment’s efficacy—in 

particular, its purported inability to distinguish naturally occurring radioactive materials from a nuclear 

bomb. GAO was asked to review DHS’s progress in (1) deploying radiation detection equipment, (2) using 

radiation detection equipment, (3) improving the equipment’s capabilities and testing, and (4) increasing 

cooperation between DHS and other federal agencies in conducting radiation detection programs. GAO 

reported these findings on March 22, 2006, in Combating Nuclear Smuggling: DHS Has Made Progress 

Deploying Radiation Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain.

Case Studies 8, 10, 14, 27, 28, 33, 36, 38, 40, and 46: From Defense Acquisitions, 

GAO-05-183, February 28, 2005

The U.S. Navy makes significant investments to maintain the technological superiority of its warships. 

It devoted $7.6 billion in 2005 alone to new ship construction in six ship classes: 96 percent of this was 

allocated to the Arleigh Burke class destroyer, Nimitz class aircraft carrier, San Antonio class amphibious 

transport dock ship, and Virginia class submarine. Cost growth in the Navy’s shipbuilding programs has 

been a long-standing problem. Over the few preceding years, the Navy had used “prior year completion” 

funding—that is, additional appropriations for ships already under contract—to pay for cost overruns. 

Responding to a congressional request, GAO’s review—Defense Acquisitions: Improved Management 

Practices Could Help Minimize Cost Growth in Navy Shipbuilding Programs (Feb. 28, 2005)—(1) estimated 

the current and projected cost growth on construction contracts for eight case study ships, (2) broke down 

and examined the components of the cost growth, and (3) identified funding and management practices 

that contributed to cost growth.

Case Study 9: From Defense Acquisitions, GAO-06-327, March 15, 2006

DOD has spent nearly $90 billion since 1985 to develop a Ballistic Missile Defense System. The 

developer, the Missile Defense Agency (MDA), plans to invest about $58 billion more in the next 6 years. 

MDA’s overall goal is to produce a system that can defeat enemy missiles launched from any range during 

any phase of their flight. Its approach is to field new capabilities in 2-year blocks. Block 2004, the first 

block, was to provide some protection by December 2005 against attacks out of North Korea and the 

Middle East. 

The Congress requires GAO to assess MDA’s progress annually. Its 2006 report assessed (1) MDA’s 

progress during fiscal year 2005 and (2) whether capabilities fielded under Block 2004 met their goals. In 

Defense Acquisitions: Missile Defense Agency Fields Initial Capability but Falls Short of Original Goals (Mar. 

15, 2006), GAO identified reasons for shortfalls and discussed corrective actions that should be taken. 
GAO-09-3SP APPENDIX II 301

Case Studies 12, 32, and 44: From Cooperative Threat Reduction, GAO-06-692, 

May 31, 2006

Until Russia’s stockpile of chemical weapons is destroyed, it will remain not only a proliferation threat 

but also vulnerable to theft and diversion. The U.S. Congress has authorized DOD since 1992 to provide 

more than $1 billion for the Cooperative Threat Reduction program to help the Russian Federation 

build a chemical weapons destruction facility at Shchuch’ye to eliminate about 14 percent of its stockpile. 

DOD has faced numerous challenges over the past several years that have increased the facility’s estimated 

cost from about $750 million to more than $1 billion and that delayed its operation from 2006 to 2009. 

DOD has attributed these increases to a variety of factors. Asked to assess the facility’s progress, schedule, 

and cost and to review the status of Russia’s efforts to destroy all its chemical weapons, GAO reported its 

findings in Cooperative Threat Reduction: DOD Needs More Reliable Data to Better Estimate the Cost and 

Schedule of the Shchuch’ye Facility (May 31, 2006).

Case Study 15: From Combating Nuclear Smuggling, GAO-07-133R, October 17, 2006

DHS is responsible for providing radiation detection capabilities at U.S. ports of entry. Current portal 

monitors, costing about $55,000 each, detect the presence of radiation. They cannot distinguish between 

harmless radiological materials, such as naturally occurring radiological material in some ceramic tile, and 

dangerous nuclear material, such as highly enriched uranium. Portal monitors with new identification 

technology designed to distinguish between the two types of material currently cost $377,000 or more. 

In July 2006, DHS announced that it had awarded contracts to three vendors to further develop and 

purchase $1.2 billion worth of new portal monitors over 5 years. 

GAO’s report on these developments is in Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to 

Support the Purchase of New Radiation Detection Portal Monitors Was Not Based on Available Performance 

Data and Did Not Fully Evaluate All the Monitors’ Costs and Benefits (Oct. 17, 2006).

Case Study 17: From Best Practices, GAO-03-645T, April 11, 2003

DOD’s modernizing its forces competes with health care, homeland security, and other demands for 

federal funds. Therefore, DOD must manage its acquisitions as cost efficiently and effectively as possible. 

As of April 2003, DOD’s overall investments to modernize and “transition” U.S. forces were expected to 

average $150 billion a year through 2008. 

In 2003, DOD’s newest acquisition policy emphasized evolutionary, knowledge-based concepts that had 

produced more effective and efficient weapon system outcomes. However, most DOD programs did 

not employ such concepts and, as a result, experienced cost increases, schedule delays, and poor product 

quality and reliability.

In a hearing before the Subcommittee on National Security, Emerging Threats, and International Relations 

of the House Committee on Government Reform, GAO’s testimony—Best Practices: Better Acquisition 

Outcomes Are Possible If DOD Can Apply Lessons from F/A-22 Program (Apr. 11, 2003)—compared best 

practices for developing new products with the experiences of the F/A-22 program.
302 APPENDIX II GAO-09-3SP

Case Studies 19, 21, and 45: From DOD Systems Modernization, GAO-06-215, 

December 5, 2005

The Naval Tactical Command Support System (NTCSS) was started in 1995 to help U.S. Navy 

personnel effectively manage ship, submarine, and aircraft support activities. The Navy expected to spend 

$348 million on the system between fiscal years 2006 and 2009. As of December 2005, about $1 billion 

had been spent to partially deploy NTCSS to about half its intended sites. It is important that DOD 

adhere to disciplined information technology acquisition processes to successfully modernize its business 

systems. Therefore, GAO was asked to determine whether NTCSS was being managed according to 

DOD’s acquisition policies and guidance, as well as other relevant acquisition management best practices. 

GAO issued its report on December 5, 2005, under the title, DOD Systems Modernization: Planned 

Investment in the Naval Tactical Command Support System Needs to Be Reassessed.

Case Study 20: From United States Coast Guard, GAO-06-623, May 31, 2006

Search and rescue is one of the U.S. Coast Guard’s oldest missions and highest priorities. The search and 

rescue mission includes minimizing the loss of life, injury, and property damage by aiding people and 

boats in distress. The National Distress and Response System is the legacy communications component 

of the Coast Guard’s search and rescue program. However, the 30-year-old system had several deficiencies 

and was difficult to maintain, according to agency officials. In September 2002, the Coast Guard 

contracted to replace its search and rescue communications system with a new system known as Rescue 

21. However, the acquisition and initial implementation of Rescue 21 had resulted in significant cost 

overruns and schedule delays. Therefore, GAO was asked to assess the (1) reasons for the significant 

cost overruns and implementation delays, (2) viability of the revised cost and schedule estimates, and (3) 

impact of the implementation delays. GAO issued its report on May 31, 2006, under the title, United 

States Coast Guard: Improvements Needed in Management and Oversight of Rescue System Acquisition.

Case Study 22: From Air Traffic Control, GAO-08-756, July 18, 2008

In fiscal year 2008, the Federal Aviation Administration (FAA) planned to spend over $2 billion 

on information technology (IT) investments—many of which support FAA’s air traffic control 

modernization. To more effectively manage such investments, in 2005 the Office of Management and 

Budget (OMB) required agencies to use earned value management (EVM). If implemented appropriately, 

EVM is a project management approach that provides objective reports of project status, produces early 

warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of a 

program’s total costs.

Among other objectives, GAO was asked to assess FAA’s policies for implementing EVM on its IT 

investments, evaluate whether the agency is adequately using these techniques to manage key IT 

acquisitions, and assess the agency’s efforts to oversee EVM compliance. To do so, GAO compared agency 

policies with best practices, performed four case studies, and interviewed key FAA officials. GAO issued 

its report, FAA Uses Earned Value Techniques to Help Manage Information Technology Acquisitions, but 

Needs to Clarify Policy and Strengthen Oversight, on July 18, 2008.
GAO-09-3SP APPENDIX II 303

Case Study 24: From Homeland Security, GAO-06-296, February 14, 2006

DHS’s U.S. Visitor and Immigrant Status Indicator Technology (US-VISIT) program was designed to 

collect, maintain, and share information, including biometric identifiers, on selected foreign nationals 

entering and exiting the United States. US-VISIT uses the identifiers—digital finger scans and 

photographs—to match persons against watch lists and to verify that a visitor is the person who was issued 

a visa or other travel documents. Visitors are also to have their departure confirmed by having their visas 

or passports scanned and by undergoing finger scanning at selected air and sea ports of entry. GAO has 

made many recommendations to improve the program’s management, all of which DHS has agreed to 

implement. GAO was asked to report in February 2006 on DHS’s progress in responding to 18 of those 

recommendations. Homeland Security: Recommendations to Improve Management of Key Border Security 

Program Need to Be Implemented (Feb. 14, 2006) was the result.

Case Study 41: From Chemical Demilitarization, GAO-07-240R, January 26, 2007

The U.S. stockpile of 1,269 tons of a lethal nerve agent (called VX) stored at the Newport Chemical 

Depot, Indiana, is one of nine stockpiles that DOD must destroy in response to congressional direction 

and the requirements of the Chemical Weapons Convention. The stockpile at Newport will be destroyed 

by neutralization—mixing hot water and sodium hydroxide with VX to change the chemical composition 

to a less toxic form. The resulting by-product is a liquid wastewater commonly referred to as hydrolysate 

that consists mostly of water but needs further treatment for disposal. At the time of GAO’s review, none 

of the generated hydrolysate—which was expected to be about 2 million gallons at the completion of the 

neutralization process—had been treated. Instead, the hydrolysate was being stored on-site until a posttreatment plan could be implemented.

The House Committee on Armed Services Report on the National Defense Authorization Act for Fiscal 

Year 2006 (H.R. Rep. No. 109-89) directed the Secretary of the Army to conduct and provide the 

congressional defense committees with a detailed cost-benefit analysis to include an analysis comparing 

the proposed off-site treatment option with eight on-site options. In response, the Army published 

its cost-benefit report in April 2006, which concluded that only three of the eight technologies were 

feasible for treating Newport’s hydrolysate. In the cost-effectiveness analysis contained in the report, the 

Army determined that the cost of off-site treatment of the hydrolysate would be less expensive than the 

on-site options. The Army also concluded that the off-site treatment option would allow the disposal 

to be accomplished in the shortest amount of time and would minimize the amount of time that the 

hydrolysate must be stored at Newport. GAO was asked to (1) assess the reasonableness of the Army’s 

rationale to eliminate five of the eight technologies for treating Newport’s hydrolysate; (2) determine 

what other options the Army considered, such as incineration; and (3) evaluate the adequacy of the 

cost comparison analysis presented for the three remaining technologies considered as alternatives 

to the Army’s proposed plan. GAO issued its report on January 26, 2007, under the title, Chemical 

Demilitarization: Actions Needed to Improve the Reliability of the Army’s Cost Comparison Analysis for 

Treatment and Disposal Options for Newport’s VX Hydrolysate.
304 APPENDIX II GAO-09-3SP

Case Study 43: From Telecommunications, GAO-07-268, February 23, 2007

The mission of General Services Administration (GSA) technology programs is to provide federal agencies 

with acquisition services and solutions at best value, including offering agencies options for acquiring 

needed telecommunications services. With the current set of governmentwide telecommunications 

contracts approaching expiration, GSA and its customer agencies will have to see the services acquired 

under these contracts through their transition to their replacements, known collectively as Networx. 

GSA will incur program management costs associated with planning and executing this transition. It 

has also made a commitment to absorb certain agency transition costs. To ensure that it would have the 

funds necessary to pay for these costs, GSA estimated that it would need to set aside about $151.5 million. 

GAO was asked to determine (1) the soundness of GSA’s analysis in deriving the estimate of funding 

that would be required for the transition and (2) whether GSA will have accumulated adequate funding 

to pay for its transition management costs. GAO issued its report on February 23, 2007, under the title, 

Telecommunications: GSA Has Accumulated Adequate Funding for Transition to New Contracts but Needs 

Cost Estimation Policy.

Case Study 47: From National Airspace System, GAO-03-343, January 31, 2003

The Standard Terminal Automation Replacement System (STARS) was to replace outdated computer 

equipment used to control air traffic within 5 to 50 nautical miles of an airport. At the time of this review, 

FAA’s plan was to procure 74 STARS systems, including 70 for terminal facilities and 4 for support 

facilities. With STARS, air traffic controllers at these facilities would receive new hardware and software 

that would produce color displays of aircraft position and flight information. In the future, FAA would 

be able to upgrade the software to provide air traffic control tools to allow better spacing of aircraft as 

they descend into airports. STARS was complex, costly, and software-intensive. Since 1996, when FAA 

initiated STARS, the number of systems scheduled to be procured ranged from as many as 188 to as 

few as 74, and the program’s cost and schedule also varied considerably. GAO’s report, covering cost and 

performance issues related to this procurement, is in National Airspace System: Better Cost Data Could 

Improve FAA’s Management of the Standard Terminal Automation Replacement System (Jan. 31, 2003).

Case Study 48: From Uncertainties Remain, GAO-04-643R, May 17, 2004

In 1996, the Air Force launched an acquisition program to develop and produce a revolutionary laser 

weapon system, the Airborne Laser (ABL), capable of defeating an enemy ballistic missile during the 

boost phase of its flight. Over the 8 years preceding GAO’s review, the program’s efforts to develop this 

technology resulted in significant cost growth and schedule delays. The prime contractor’s costs for 

developing ABL nearly doubled from the Air Force’s original estimate and cost was growing. The cost 

growth occurred primarily because the program did not adequately plan for and could not fully anticipate 

the complexities of developing the system. The Missile Defense Agency continued to face significant 

challenges in developing the ABL’s revolutionary technologies and in achieving cost and schedule stability. 

From 1996 through 2003, the value of the prime contract, which accounted for the bulk of the program’s 

cost, increased from about $1 billion to $2 billion. According to our analysis, costs could increase between 

$431 million and $943 million more through the first full demonstration of the ABL system. GAO’s 

report, covering cost and performance issues related to this procurement, is in Uncertainties Remain 

Concerning the Airborne Laser’s Cost and Military Utility, (May 17, 2004).
GAO-09-3SP APPENDIX III 305

Experts Who Helped Develop This Guide

The two lists in this appendix name the experts in the cost estimating community, with their 

organizations, who helped us develop this guide. This first list names significant contributors to the Cost 

Guide. They attended and participated in numerous expert meetings, provided text or graphics, and 

submitted comments.

Organization Expert

ABBA Consulting Wayne Abba

BAE Systems Shobha Mahabir

David Consulting Group Michael Harris

Defense Contract Management Agency Bob Keysar

Defense Acquisition University David Bachman

Todd Johnston

Department of Defense Debbie Tomsic

Department of Treasury Kimberly Smith

Federal Aviation Administration Lewis Fisher

Wayne Hanley

Fred Sapp

Fleming Management Consultancy Quentin Fleming

Galorath Incorporated Dan Galorath

Hulett & Associates LLC David Hulett

Independent Consultant John Pakiz

Internal Revenue Service Donald Moushegian

KM Systems Group Kim Hunter

Lockheed Martin Corporation Jamie Fieber

Ludwig Consulting Services LLC Joyce Ludwig

Management Concepts Gregory T. Haugan

MCR Federal LLC Neil Albert

Missile Defense Agency David Melton

Peter Schwarz

MITRE David Crawford

MITRE and National Oceanic and Atmospheric Administration Richard Riether

National Aeronautics and Space Administration Glenn Campbell

David Graham

PRICE Systems Bruce Fad

William Mathis 

Project and Program Management Systems, International Eric Marantoni

Social Security Administration Otto Immink 

The Analytical Sciences Corporation Peter Braxton

Greg Hogan

Technomics Rick Collins

Robert Meyer

Jack Smuck
306 APPENDIX III GAO-09-3SP

Organization Expert

Tecolote Research, Incorporated Lew Fichter

Greg Higdon

Bill Rote

Alf Smith

United Kingdom Ministry of Defense Andy Nicholls

U.S. Air Force, Air Force Cost Analysis Agency John Cargill

Rich Hartley

John Peterson

William Seeman

Wilson Rosa

U.S. Army, Army Cost Center Sean Vessey

U.S. Army, Corps of Engineers Kim Callan

U.S. Navy, Center for Cost Analysis Susan Wileman

U.S. Navy, Naval Air Systems Command Brenda Bizier

Susan Blake

Fred Meyer

John Scapparo

U.S. Navy, Naval Sea Systems Command Hershel Young

This second list names those who generously donated their time to review the Cost Guide in its various 

stages and to provide feedback.

Organization Expert

Association for the Advancement of Cost Engineering Osmund Belcher

Bill Kraus

Business Growth Solutions Ltd. Keith Gray

Center for Naval Analyses Dan Davis

Richard Sperling

CGI Federal Sameer Rohatgi

Comcast Tamara Garcia

Data Systems Analysts Inc. Aubrey Jones

Department of Defense John Leather

Department of Homeland Security Jim Manzo

Michael Zaboski

Department of Homeland Security, Domestic Nuclear Detection 

Office

Richard Balzano

Lisa Bell

Andrew Crisman

Federal Aviation Administration Scott Allard 

Dan Milano

William Russell

Hutchins & Associates Pam Shepherd

Independent Consultant Steven Deal

Dee Dobbins

Jan Kinner

David Muzio

Max Wideman
GAO-09-3SP APPENDIX III 307

Organization Expert

Lockheed Martin Corporation Walt Berkey

Bill Farmer

Kathleen McCarter

Chitra Raghu 

Tony Stemkowski

MITRE Raj Agrawal

National Geospatial Intelligence Agency Ivan Bembers

Northrop Grumman Gay Infanti

Beverly Solomon

Office of Management and Budget Patricia Corrigan

Parsons Karen Kimball

Michael Nosbisch

Jon Tanke

Sandy Whyte

PRICE Systems Didier Barrault

PT Mitratata Citragraha (PTMC) Paul D. Giammalvo

Robbins Gioia Wei Tang

Social Security Administration Alan Deckard

SRA International David Lyons

SRS Technologies Tim Sweeney

The Analytical Sciences Corporation Samuel Toas

University of Colorado Shekhar Patil

UQN and Associates Ursula Kuehn

U.S. Air Force Harold Parker

U.S. Army Robert Dow

U.S. Army, Army Cost Center Mort Anvari

U.S. Navy Mark Gindele

U.S. Navy, Naval Air Systems Command Jeff Scher

Wyle Labs Katrina Brown
308 APPENDIX IV GAO-09-3SP

The Federal Budget Process

Each year in January or early February, the president submits budget proposals for the year that begins 

October 1. They include data for the most recently completed year, the current year, the budget year, and 

at least the 4 years following the budget year. 

The budget process has four phases: (1) executive budget formulation, (2) congressional budget process, 

(3) budget execution and control, and (4) audit and evaluation. Budget cycles overlap—the formulation 

of one budget begins before action has been completed on the previous one. Tables 49 and 50 present 

information from OMB’s Circular A-11 about the main phases of the budget cycle and the steps—and 

time periods—within each phase. 

Table 49: Phases of the Budget Process

Phase Description

Executive budget 

formulation

OMB and the federal agencies begin preparing one budget almost as soon as the 

president has sent the last one to the Congress. OMB officially starts the process by 

sending planning guidance to executive agencies in the spring. The president completes 

this phase by sending the budget to the Congress on the first Monday in February, as 

specified in law

Congressional 

budget process

Begins when the Congress receives the president’s budget. The Congress does not 

vote on the budget but prepares a spending and revenue plan that is embedded in the 

Congressional Budget Resolution; the Congress also enacts regular appropriations acts 

and other laws that control spending and receipts

Budget execution This phase lasts for at least 5 fiscal years and has two parts:

■ Apportionment pertains to funds appropriated for that fiscal year and to balances 

of appropriations made in prior years that remain available for obligation. At the 

beginning of the fiscal year, and at other times as necessary, OMB apportions funds 

to executive agencies; that is, it specifies the amounts they may use by time period, 

program, project, or activity. Throughout the year, agencies hire people, enter into 

contracts, enter into grant agreements, and so on, to carry out their programs, 

projects, and activities. These actions use up the available funds by obligating the 

federal government to make immediate or future outlays

■ Reporting and outlay last until funds are canceled (1-year and multiple-year funds are 

canceled at the end of the fifth year, after the funds expire for new obligations) or until 

funds are totally disbursed (for no-year funds)

Audit and 

evaluation

■ While OMB does not specify times, each agency is responsible for ensuring that its 

obligations and outlays adhere to the provisions in the authorizing and appropriations 

legislation, as well as other laws and regulations governing the obligation and 

expenditure of funds. OMB provides guidance for, and federal laws are aimed at, 

controlling and improving agency financial management. Agency inspectors general 

give policy direction for, and agency chief financial officers oversee, all financial 

management activities related to agency programs and operations. 

■ The 1993 Government Performance and Results Act requires each agency to submit 

an annual performance plan and performance report to OMB and the Congress; the 

report must establish goals defining the level of performance each program activity 

in the agency’s budget is to achieve and describing the operational processes and 

resources required to meet those goals. The Congress oversees agencies through the 

legislative process, hearings, and investigations. GAO audits and evaluates government 

programs and reports its findings and recommendations for corrective action to the 

Congress, OMB, and the agencies

Source: GAO and OMB.
GAO-09-3SP APPENDIX IV 309

Table 50: The Budget Process: Major Steps and Time Periods 

Phase Major step Time

Formulation OMB issues planning guidance to executive agencies. OMB’s Director 

issues to agency heads policy guidance for budget requests; if no more 

specific guidance is given, the previous budget’s out-year estimates 

serve as the starting point for the next budget. This begins the process of 

formulating the budget the president will submit next February

Spring

OMB issues Circular No. A–11 to all federal agencies, providing detailed 

instructions for submitting budget data and materials

July

Executive agencies, except those not subject to review, submit budgets; 

OMB provides specific deadlines 

Sept.

The fiscal year begins. The just completed budget cycle focused on this 

fiscal year, the budget year in that cycle, and the current year in this cycle

Oct. 1

OMB conducts its fall review, analyzing agency budget proposals in light 

of presidential priorities, program performance, and budget constraints

Oct.–Nov.

OMB informs executive agencies of decisions on their budget requests Late Nov.

Agencies enter computer data and submit printed material and additional 

data; this begins immediately after passback and continues until OMB 

“locks” agencies out of the database to meet the printing deadline

Late Nov. to early 

Jan.

Agencies prepare, and OMB reviews, the justification materials they need 

to explain their budget requests to congressional subcommittees 

Jan.

The president transmits the budget to the Congress First Mon. in Feb.

Congressional The Congressional Budget Office (CBO) reports to budget committees on 

the economic and budget outlook

Jan.

CBO reestimates the President’s Budget, based on its economic and 

technical assumptions

Feb.

Committees submit “views and estimates” to House and Senate budget 

committees, indicating preferences on matters they are responsible for

Within 6 weeks of 

budget transmittal

The Congress completes action on the concurrent resolution on the 

budget and commits to broad spending and revenue levels by passing a 

budget resolution

Apr. 15

The Congress completes action on appropriations bills for the coming 

fiscal year or passes a continuing resolution (stop-gap appropriations)

Sept. 30

Execution The fiscal year begins Oct. 1

OMB apportions funds made available in the annual appropriations 

process and other available funds. Agencies submit to OMB 

apportionment requests for each budget account by August 21 or within 

10 calendar days after the approval of the appropriation, whichever is 

later. OMB approves or modifies apportionments, specifying the funds 

agencies may use by time period, program, project, or activity

Sept. 10, or within 

30 days after 

approval of a 

spending bill

Agencies incur obligations and make outlays for funded programs, 

projects, and activities, hiring people and entering into contracts and 

agreements. They record obligations and outlays according to control 

procedures, report to Treasury, and prepare financial statements

Throughout the 

fiscal year

The fiscal year ends Sept. 30

Agencies disburse against obligated balances and adjust them to reflect 

actual obligations, continuing to record obligations and outlays, report to 

Treasury, and prepare financial statements 

Until Sept. 30, fifth 

year after funds 

expire

Source: OMB.
310 APPENDIX V GAO-09-3SP

Federal Cost Estimating and EVM Legislation, 

Regulations, Policies, and Guidance

The material in this appendix, keyed to table 3 in the body of the Cost Guide, describes criteria related to 

cost estimating and EVM.

Legislation and Regulations

1968: DOD Selected Acquisition Reports

Before selected acquisition reports (SAR) were introduced, with DOD Instruction 7000.3 in 1968, no 

recurring reports on major acquisitions summarized cost, schedule, and performance data for comparison 

with earlier and later estimates. The original purpose of SARs was to keep the Assistant Secretary of 

Defense (Comptroller) informed of the progress of selected acquisitions and to compare this progress with 

the planned technical, schedule, and cost performance. When the Secretary of Defense and the Congress 

began to require regular reports early in 1969, SARs became key recurring summaries advising the 

Congress on the progress of major acquisition programs.1

For the purpose of oversight and decision making, legislation (10 U.S.C. § 2432 (2006)) now requires 

DOD to submit SARs annually to the Congress. The reports present the latest cost and schedule estimates 

and technical status for major defense programs. The comprehensive annual SARs are prepared in 

conjunction with the president’s budget. 

Quarterly exception reports are required only for programs with unit cost increases of at least 15 percent 

or schedule delays of at least 6 months. They are also submitted for initial reports, final reports, and 

programs that are rebaselined at major milestone decisions. 

For each major defense acquisition program, an SAR contains program quantities; program acquisition 

cost and acquisition unit cost; current procurement cost and procurement unit cost; reasons for any 

changes in these costs from the previous SAR; reasons for any significant changes from the previous SAR 

in total program cost, software schedule milestones, or performance; any major contract changes and 

reasons for cost or schedule variances since the last SAR; and program highlights for current reporting 

period.

1982: DOD Unit Cost Reports

Recognizing the need to establish a cost growth oversight mechanism for DOD’s major defense 

acquisition programs, the Congress requires DOD to report on program cost growth that exceeds 

certain thresholds. This requirement is commonly called Nunn-McCurdy, after the congressional leaders 

responsible for it. It became permanent law in 1982 with the Department of Defense Authorization Act, 

1983. The law (10 U.S.C. § 2433 (2006)) now provides for oversight of cost growth in DOD’s major 

defense acquisition programs by requiring DOD to notify the Congress when a program’s unit cost 

growth exceeds (or breaches) the original or the latest approved acquisition program baseline by certain 

thresholds.2

 If the cost growth has increased by certain percentages over the baseline, the Secretary of 

Defense must carry out an assessment that includes the projected costs of completing the program if 

1 See Comptroller General of the United States, How to Improve the Selected Acquisition Reporting System: Department of Defense, 

PSAD-75-63 (Washington, D.C.: GAO, Mar. 27, 1975), p. 2.

2 See 10 U.S.C.S. § 2433 (2002 & Supp. 2007).
GAO-09-3SP APPENDIX V 311

current requirements are not modified, as well as based on reasonable modification of such requirements. 

The assessment is also to include a rough order of magnitude of the costs of any reasonable alternative 

system of capability. Further, the Secretary of Defense is to certify to the Congress that 

1. the program is essential to national security,

2. no alternatives will provide equal or greater military capability at less cost,

3. new program acquisition or procurement unit cost estimates are reasonable, and

4. the management structure is adequate to control unit cost.

1983: DOD Independent Cost Estimates 

Section 2434 of title 10 of the U.S. Code requires the Secretary of Defense to consider an independent 

life-cycle cost estimate (LCCE) before approving system development and demonstration, or production 

and deployment, of a major defense acquisition program. Under DOD’s acquisition system policy, this 

function is delegated to a program’s milestone decision authority. The statute requires that DOD prescribe 

regulations governing the content and submission of such estimates and that the estimates be prepared 

1. by an office or other entity not under the supervision, direction, or control of the military 

department, agency, or other component directly responsible for the program’s development or 

acquisition or 

2. if the decision authority has been delegated to an official of a military department, agency, or other 

component, by an office or other entity not directly responsible for the program’s development or 

acquisition. 

The statute specifies that the independent estimate is to include all costs of development, procurement, 

military construction, and operations and support, without regard to funding source or management 

control.

1993: Government Performance and Results Act 

The Government Performance and Results Act of 1993 (GPRA), Pub. L. No. 103-62, requires agencies 

to prepare multiyear strategic plans that describe mission goals and methods for reaching them. It also 

requires agencies to develop annual performance plans that OMB uses to prepare a federal performance 

plan that is submitted to the Congress, along with the president’s annual budget submission. The agencies’ 

plans must establish measurable goals for program activities and must describe the methods for measuring 

performance toward those goals. The act also requires agencies to prepare annual program performance 

reports to review progress toward annual performance goals.

1994: Federal Acquisition Streamlining Act 

The Federal Acquisition Streamlining Act of 1994 (Pub. L. No. 103-355, §§ 5001(a)(1), 5051(a), as 

amended) established a congressional policy that the head of each executive agency should achieve, on 

average, 90 percent of cost, performance, and schedule goals established for major acquisition programs 

of the agency. The act requires an agency to approve or define cost, performance, and schedule goals 

for its major acquisition programs. To implement the 90 percent policy, the act requires agency heads to 

determine whether there is a continuing need for programs that are significantly behind schedule, over 

budget, or not in compliance with performance or capability requirements and to identify suitable actions 
312 APPENDIX V GAO-09-3SP

to be taken, including termination, with respect to such programs. This provision is codified at 41 U.S.C. 

§ 263 (2000) for civilian agencies. A similar requirement in 10 U.S.C. § 2220 applied to DOD but was 

amended to remove the 90 percent measure. DOD has its own major program performance oversight 

requirements, such as the Nunn-McCurdy cost reporting process at 10 U.S.C. § 2433. OMB incorporated 

the 90 percent measure into the Capital Programming Guide Supplement to Circular A-11.3

1996: Clinger-Cohen Act

The Clinger-Cohen Act of 1996 (codified, as relevant here, at 40 U.S.C. §§ 11101–11704 (Supp. V 2005)) 

is intended to improve the productivity, efficiency, and effectiveness of federal programs by improving the 

acquisition, use, and disposal of information technology resources. Among its provisions, it requires federal 

agencies to

1. establish capital planning and investment control processes to maximize the value and manage the 

risks of information technology acquisitions, through quantitative and qualitative assessment of 

investment costs, benefits, and risks, among other ways;

2. establish performance goals and measures for assessing and improving how well information 

technology supports agency programs, by benchmarking agency performance against public and 

private sector best practices; 

3. appoint chief information officers to be responsible for carrying out agency information resources 

management activities, including the acquisition and management of information technology, to 

improve agency productivity, efficiency, and effectiveness; and

4. identify in their strategic information resources management plans any major information 

technology acquisition program, or any phase or increment of such a program, that has 

significantly deviated from the cost, performance, or schedule goals established for the program.

2006: DOD Major Automated Information System Programs

Section 816 of the John Warner National Defense Authorization Act for Fiscal Year 2007 (Pub. L. No. 

109-364) added new oversight requirements for DOD’s major automated information system programs. 

These requirements, codified at 10 U.S.C. §§ 2445a–2445d (2006), include estimates of development 

costs and full life-cycle costs, as well as the establishment of a program baseline, variance reporting, and 

reports on significant or critical changes in the program (these include estimated program cost increases 

over certain thresholds). 

2006: Federal Acquisition Regulation—EVM Policy Added

The government’s earned value management system policy is spelled out in subpart 34.2 of the Federal 

Acquisition Regulation (FAR, 48 C.F.R.). The Civilian Agency Acquisition Council and the Defense 

Acquisition Regulations Council promulgated a final rule amending the FAR to implement EVM policy 

on July 5, 2006.4

 The rule was necessary to help standardize EVM use across the government where 

3OMB, Capital Programming Guide: Supplement to Circular A-11, Part 7, Preparation, Submission, and Execution of the Budget

(Washington, D.C.: Executive Office of the President, June 2006). www.whitehouse.gov/omb/circulars/index.html.

4 See Federal Acquisition Circular 2005-11, July 5, 2006, Item I—Earned Value Management System (EVMS) (FAR Case 2004-019).
GAO-09-3SP APPENDIX V 313

developmental effort under a procurement contract is required. It implements EVM system policy in 

accordance with OMB Circular A–11, Part 7, and its supplement, the Capital Planning Guide.

5

It requires that EVM be used for major acquisitions for development. The rule defines an EVM system as 

a project management tool that effectively integrates the project’s scope of work with cost, schedule, and 

performance elements for optimum project planning and control (see FAR, 48 C.F.R. § 2.101). It also 

states that the qualities and characteristics of an EVM system are described in ANSI/EIA Standard 748, 

Earned Value Management Systems.6

The rule stipulates that when an EVM system is required, the government is to conduct an integrated 

baseline review (IBR) to verify the technical content and realism of the related performance budgets, 

resources, and schedules. Through the IBR, agencies are to attain mutual understanding of the risks 

inherent in contractors’ performance plans and the underlying management control systems. The rule 

contemplates that the IBR results in the formulation of a plan to handle these risks.

2008: Defense Federal Acquisition Regulation Supplement 

DOD issued a final rule (73 Fed. Reg. 21,846 (April 23, 2008), primarily codified at 48 C.F.R. subpart 

234.2, and part 252 (sections 252.234-7001 and 7002)), amending the Defense Federal Acquisition 

Regulation Supplement (DFARS) to update requirements for DOD contractors to establish and maintain 

EVM systems. The rule also eliminated requirements for DOD contractors to submit cost-schedule status 

reports.

This final rule updated DFARS text addressing EVM policy for DOD contracts, supplements the 

final FAR rule published at 71 Fed. Reg. 38,238 on July 5, 2006, and establishes DOD-specific EVM 

requirements, as permitted by the FAR. The DFARS rule follows up on the policy in the memorandum 

the Under Secretary of Defense (Acquisition, Technology, and Logistics) issued on March 7, 2005, entitled 

“Revision to DOD Earned Value Management Policy.”

The DFARS changes in this rule include the following: For cost or incentive contracts and subcontracts 

valued at $20 million or more, the rule requires an EVM system that complies with the guidelines in 

the American National Standards Institute/Electronic Industries Alliance Standard 748, Earned Value 

Management Systems (ANSI/EIA–748). For cost or incentive contracts and subcontracts valued at $50 

million or more, the rule requires an EVM system that the cognizant federal agency (as defined in FAR 

2.101) has determined to be in compliance with the guidelines in ANSI/EIA–748. For cost or incentive 

contracts and subcontracts valued at less than $20 million, the rule provides that application of EVM is 

optional and is a risk-based decision. For firm-fixed-price contracts and subcontracts of any dollar value, 

the rule discourages applying EVM. DCMA is assigned responsibility for determining EVM compliance 

when DOD is the cognizant federal agency. Requirements for contractor cost-schedule status reports are 

eliminated.

5OMB, Capital Programming Guide: Supplement to Circular A-11, Part 7.

6EVM systems guidelines in American National Standards Institute (ANSI)/Electronic Industries Alliance (EIA) Standard 748 

were developed and promulgated through ANSI by the National Defense Industrial Association’s (NDIA) Program Management 

Systems Committee. 
314 APPENDIX V GAO-09-3SP

Policies

1976: OMB Circular Major Systems Acquisitions

OMB’s 1976 Circular A-109, Major Systems Acquisitions, establishes policies for agencies to follow when 

acquiring major systems. It requires agencies to ensure that their major system acquisitions fulfill mission 

needs, operate effectively, and demonstrate a level of performance and reliability that justifies the use 

of taxpayers’ funds. The policy also states that agencies need to maintain the ability to develop, review, 

negotiate, and monitor life-cycle costs. Moreover, agencies are expected to assess cost, schedule, and 

performance progress against predictions and inform agency heads of any variations at key decision points. 

When variations occur, the circular requires agencies to develop new assessments and use independent cost 

estimates, where feasible, for comparing results. 

1992: OMB Guidelines and Discount Rates for Benefit-Cost Analysis

OMB issued Circular No. A-94 to agencies in 1992, Guidelines and Discount Rates for Benefit-Cost Analysis 

of Federal Programs, to support government decisions to initiate, review, or expand programs that would 

result in measurable costs or benefits extending for 3 or more years into the future. It is general guidance 

for conducting benefit-cost and cost-effectiveness analyses. It also gives specific guidance on discount rates 

for evaluating federal programs whose benefits and costs are distributed over time. 

The guidance serves as a checklist for whether an agency has considered and properly dealt with all the 

elements of sound benefit-cost and cost-effectiveness analyses, including, among other things, identifying 

assumptions, analyzing alternatives, applying inflation, discounting for net present value, characterizing 

uncertainty, and performing sensitivity analysis.

1995: DOD’s Economic Analysis for Decisionmaking Instruction

Economic Analysis for Decisionmaking, DOD’s 1995 Instruction No. 7041.3, implements policy and 

updates responsibilities and procedures for conducting cost-effectiveness economic analysis. It states 

that economic analysis is an important tool for planning and budgeting for DOD systems, and it helps 

decision makers obtain insight into the economic factors of various alternatives. The instruction outlines 

procedures for estimating the life-cycle costs and benefits of each feasible alternative and for adjusting 

all costs and benefits to present value by using discount factors to account for the time value of money. 

These procedures provide decision makers with the information associated with each alternative’s size and 

the timing of costs and benefits so that the best alternative can be selected. The instruction discusses the 

following elements of an economic analysis: a statement of the objective, assumptions, alternative ways 

of satisfying the objective, costs and benefits for each alternative considered, a comparison of alternatives 

ranked by net present value, sensitivity and uncertainty analysis, and results and recommendations. It also 

contains guidance on choosing alternatives and providing sensitivity analysis and proper discounting. 

2003: DOD’s Defense Acquisition System Directive

DOD’s Directive No. 5000.1, The Defense Acquisition System, outlines the management processes DOD 

is to follow to provide effective, affordable, and timely systems to users. It stipulates that the Defense 

Acquisition System exists to manage the nation’s investment in technologies, programs, and product 

support necessary to achieve the National Security Strategy and support the armed forces. Among other 

things, the policy requires every program manager to establish life-cycle cost, schedule, and performance 
GAO-09-3SP APPENDIX V 315

goals that will determine the acquisition program baseline. These goals should be tracked and any 

deviations in program parameters and exit criteria should be reported. The directive discusses how 

programs should be funded to realistic estimates and states that major drivers of total ownership costs 

should be identified. It requires program managers to use knowledge-based acquisition for reducing risk by 

requiring that new technology be demonstrated before it is incorporated into a program.

DOD’s Directive No. 5000.1, The Defense Acquisition System has been redesignated 5000.01 and certified 

current as of Nov. 20, 2007.

2003: DOD’s Operation of the Defense Acquisition System Instruction

DOD’s Instruction No. 5000.2, Operation of the Defense Acquisition System, establishes a framework 

for translating requirements into stable and affordable programs that can be managed effectively. It 

describes the standard framework for defense acquisition systems, which is to define the concept and 

analyze alternatives, develop the technology, develop the system and demonstrate that it works, produce 

the system and deploy it to its users, and operate and support the system throughout its useful life. The 

instruction also discusses in great detail the three milestones and what entrance and exit criteria must be 

met for each one. It explains the concept of evolutionary acquisition and how DOD prefers this strategy 

for acquiring technology, because it allows for the delivery of increased technical capability to users in the 

shortest time. The instruction identifies technology readiness assessments as a way to manage and mitigate 

technology risk. It discusses the different kinds of acquisition categories and their cost thresholds and 

decision authorities. In addition, it defines the role of the Cost Analysis Improvement Group (CAIG), in 

developing independent cost estimates. 

DOD’s Instruction No. 5000.2, Operation of the Defense Acquisition System was cancelled and reissued by 

Instruction No. 5000.02 on Dec. 8, 2008.

2004: National Security Space Acquisition Policy 03-01

This document provides acquisition process guidance for DOD entities that are part of the National 

Security Space team. The Under Secretary of the Air Force is the DOD Space Milestone Decision 

Authority for all DOD Space Major Defense Acquisition Programs (MDAP). National Security Space is 

defined as the combined space activities of DOD and the National Intelligence Community. This policy 

describes the streamlined decision making framework for such DOD programs. 

A DOD Space Major Defense Acquisition Program is an acquisition program that the DOD Space 

Milestone Decision Authority or the Defense Acquisition Executive designates as special interest or that 

the Space Milestone Decision Authority estimates will require an eventual total expenditure for research, 

development, test, and evaluation of more than $365 million in fiscal year 2000 constant dollars or, for 

procurement, more than $2.19 billion in fiscal year 2000 constant dollars. Highly sensitive classified 

programs as defined by 10 U.S.C. § 2430 are not included. 

2005: DOD’s Earned Value Management Policy

Stating that EVM had been “an effective management control tool in the Department for the past 37 

years,” DOD revised its policy—with its March 7, 2005, memorandum, “Revision to DOD Earned 

Value Management Policy”—to streamline, improve, and increase consistency in EVM’s application and 

implementation. The memorandum requires contracts equal to or greater than $20 million to implement 
316 APPENDIX V GAO-09-3SP

EVM systems in accordance with ANSI/EIA Standard 748. It also requires contractors with contracts 

equal to or greater than $50 million to have formally validated EVM systems approved by the cognizant 

contracting officer. The revised policy also requires contract performance reports, an integrated master 

schedule, and an IBR whenever EVM is required. The new policy also calls for, among other things, a 

common WBS structure for the CPR and IMS. 

2005: OMB’s Memorandum on Improving Information Technology Project Planning and 

Execution

OMB’s 2005 Improving Information Technology (IT) Project Planning and Execution Memorandum 

for Chief Information Officers discusses how it expects agencies to ensure that cost, schedule, and 

performance goals are independently validated for reasonableness before beginning development. In 

addition, it requires agencies to fully implement EVM on all major capital acquisition projects. Full 

implementation occurs when agencies have shown that they have 

1. a comprehensive agency policy for EVM; 

2. included EVM system requirements in contracts or agency in-house project charters;

3. have held compliance reviews for agency and contractor EVM systems;

4. a policy of performing periodic system surveillance reviews to ensure that the EVM system 

continues to meet ANSI/EIA Standard 748 guidelines; and 

5. a policy of conducting IBRs for making cost, schedule, and performance goals final.

The memorandum gives further guidance and explanation for each of these five key components. For 

example, OMB states that compliance reviews should confirm that a contractor’s EVM system processes 

and procedures have satisfied ANSI/EIA Standard 748 guidelines and that surveillance reviews should 

show that agencies are using EVM to manage their programs. The memorandum stresses the importance 

of an IBR as a way of assessing program performance and understanding risk. 

2006: OMB’s Capital Programming Guide

The Capital Programming Guide—the part 7 supplement to OMB’s Circular No. A-11—sets forth 

the requirements for how OMB manages and oversees agency budgets. In the budget process, agencies 

must develop and submit to OMB for review an exhibit 300, also known as the Capital Asset Plan and 

Business Case. Under OMB’s Circular A-11, agencies must analyze and document their decisions on 

proposed major investments. Exhibit 300 functions as a reporting mechanism that enables an agency 

to demonstrate to its own management, as well as OMB, that it has used the disciplines of good project 

management, developed a strong business case for investment, and met other administration priorities in 

defining the cost, schedule, and performance goals proposed for the investment. Exhibit 300 has eight 

key sections on spending, performance goals and measures, analysis of alternatives, risk inventory and 

assessment, acquisition strategy, planning for project investment and funding, enterprise architecture, 

and security and privacy. When considering investments to recommend for funding, OMB relies on 

the accuracy and completeness of the information reported in exhibit 300. It also cites that credible cost 

estimates are vital for sound management decision making and for any program or capital project to 

succeed. To that end, OMB states that following the guidelines in GAO Cost Estimating and Assessment 

Guide (GAO-09-3SP) will help agencies meet most cost estimating requirements. 
GAO-09-3SP APPENDIX V 317

2006: DOD’s Cost Analysis Improvement Group Directive

DOD’s Directive 5000.04 states that the CAIG is the principal advisory body on cost for milestone 

decision authorities. The CAIG estimates that supporting milestone decisions include costs for research 

and development, prime hardware and its major subcomponents, procurement, initial spares, military 

construction, and all operations and support—regardless of funding source or management control. 

The CAIG is to provide its assessments in a formal report addressed to milestone decision authorities. In 

addition to describing the cost estimate, the CAIG report is to include a quantitative assessment of the 

associated risks. The risks should include the validity of program assumptions, such as the reasonableness 

of program schedules and technical uncertainty and any errors associated with the cost estimating 

methods. 

The directive describes other CAIG responsibilities, including reporting on the reasonableness of unit 

costs for programs breaching specific cost thresholds, the validity of costs in acquisition program baselines, 

and independent assessments of the Defense Acquisition Executive Summary program costs and giving 

guidance on preparing cost estimates, sponsoring cost research, establishing standard definitions of cost 

terms, and developing and implementing policy to collect, store, and exchange information on how to 

improve cost estimating and data.

Guidance

1992: CAIG’s Operating and Support Cost-Estimating Guide

The 1992 Operating and Support Cost-Estimating Guide, prepared by the Cost Analysis Improvement 

Group in the Office of the Secretary, is intended to help DOD components prepare, document, and 

present operating and support cost estimates to the CAIG. It discusses the requirements for the cost 

estimates, provides instructions for developing them, and presents standard cost element structures and 

definitions for specific categories of weapon systems. Documentation and presentation requirements are 

provided to help prepare for CAIG reviews. The guide’s primary objective is to achieve consistent, welldocumented operating and support cost estimates that an independent party can replicate and verify. 

1992: DOD’s Cost Analysis Guidance and Procedures

DOD’s 1992 Directive 5000.4-M, Cost Analysis Guidance and Procedures, is a manual for preparing the 

Cost Analysis Requirements Document, which the program office is to develop, describing the program 

in enough detail for cost estimators to develop an LCCE. The manual contains information on preparing 

and presenting LCCEs to the CAIG, including the scope of the estimate and the analytical methods to be 

used. It defines seven high-level cost terms—development cost, flyaway sailaway rollaway cost, weapons 

system cost, procurement cost, program acquisition cost, operating and support cost, and life cycle cost—

and how they relate to WBS elements and appropriations. 

2003: DOD’s Program Manager’s Guide to the Integrated Baseline Review Process

DOD developed the April 2003 Program Manager’s Guide to the Integrated Baseline Review Process

to improve the consistency of the IBR process. The intent was to ensure that the IBR would provide 

program managers with an understanding of the risks involved with a contractor’s performance plans 

and corresponding EVM systems. Since DOD’s acquisition policy requires IBRs on contracts with EVM 

requirements, the guide identifies the purpose of the IBR process and stresses the need for the process to 
318 APPENDIX V GAO-09-3SP

continue even after the IBR has been conducted. Program managers are strongly encouraged to follow this 

guidance for training in, preparing, and conducting IBRs.

2004: NDIA PMSC Surveillance Guide

The NDIA PMSC Surveillance Guide—the short title of the 2004 edition of this document—is intended 

for the use of government and contractor communities in determining whether EVM systems are 

being used to effectively manage program cost, schedule, and technical performance. The guide gives 

an overview of what EVM system surveillance entails, including ensuring that company processes 

and procedures are followed to satisfy the ANSI/EIA 748-A Standard. It discusses the activities in 

proper system surveillance, including organization, planning, execution, results, management control, 

and corrective action. It provides a standard industry surveillance approach to ensuring a common 

understanding of expectations and the use of a uniform process.

2005: NDIA PMSC EVM Systems Intent Guide

The 2005 Earned Value Management Systems Intent Guide, issued by NDIA and its Program Management 

Systems Committee, is intended for the use of government analysts and contractors, wherever ANSI/EIA 

Standard 748 is required. The guide defines the management value and intent for each of the standard’s 

guidelines and lists the attributes and objective evidence that can be used to verify compliance with a 

given guideline. The objective of compliance is to demonstrate that a contractor has thought through each 

guideline and can describe how its business process complies with it. A customer, independent reviewer, 

or auditor can use the intent, typical attributes, and objective evidence of typical outputs that the guide 

describes as the basis for verifying compliance. The guide’s five sections are (1) organization; (2) planning, 

scheduling, and budgeting; (3) accounting considerations; (4) analysis and management reports; and 

(5) revisions and data maintenance. It recommends that

1. contract or business processes and system documentation be mapped and verified against the 

guideline’s intent, typical attributes, and objective evidence of typical outputs described in the 

document by the process owner;

2. someone independent of the documenting party verify the compliance assessment;

3. the verifier be versed in ANSI/EIA 748 EVM system guidelines;

4. the customer recognize this method as being applicable and meaningful to compliance assessment 

verification; and

5. the customer consider past acceptance of compliance with ANSI/EIA 748 EVM system guidelines, 

business organization application policy, and surveillance activity in management decisions to 

perform a compliance assessment.7

2006: DOD Earned Value Management Implementation Guide

DCMA issued the Department of Defense Earned Value Management Implementation Guide in 2006 

to serve as the central EVM guidance during implementation and surveillance of EVM systems in 

compliance with DOD guidelines. The guide has two parts. The first contains basic EVM information, 

7NDIA PMSC EVM Systems Intent Guide, © 2004–2005 National Defense Industrial Association (NDIA) Program Management 

Systems Committee (PMSC), ANSI/EIA-748-A Standard for Earned Value Management Systems Intent Guide (January 2005 

edition).
GAO-09-3SP APPENDIX V 319

describes an EVM system’s objectives, and provides guidance for interpreting EVM guidelines as they 

apply to government contracts. The second part describes procedures and processes government staff 

must follow in evaluating the implementation of EVM systems. It also provides guidance on tailoring 

the guidelines, analyzing EVM performance, determining the effectiveness of the baseline and its 

maintenance, and performing other activities that must be followed after contracts have been awarded. 

2006: NDIA System Acceptance Guide

NDIA’s Program Management Systems Committee’s working draft of its EVM System Acceptance Guide

was released for comment in November 2006. The guide defines a process in which a government or 

industry owner of an EVM system that has a first-time requirement to comply with the ANSI/EIA 748-A 

standard can 

1. understand the need for and effectively design the system,

2. implement the system on the acquiring acquisition,

3. evaluate its compliance and implementation,

4. prepare and provide documentation that substantiates evaluation and implementation, and

5. receive approval and documentation that satisfies current and future requirements for the system’s 

approval.8

2007: ANSI/EIA 748-B 

ANSI/EIA 748-B is an update of ANSI-EIA-748A. This document provides basic guidelines for 

companies to use in establishing and applying an integrated EVM system. The guidelines are expressed 

in fundamental terms and provide flexibility for its usage. The guidelines are grouped into five major 

categories.

They incorporate best business practices to provide strong benefits for program or enterprise planning 

and control. The processes include integrating program scope, schedule, and cost objectives; establishing 

a baseline plan for accomplishing program objectives; and using earned value techniques for performance 

measurement during the execution of a program. The system provides a sound basis for identifying 

problems, taking corrective actions, and managing replanning as required.

The guidelines in this document are purposely high level and goal oriented, since they are intended to 

state the qualities and operational considerations of an integrated management system using earned value 

analysis methods without mandating detailed system characteristics. Different organizations must have 

the flexibility to establish and apply a management system that suits their management style and business 

environment. The system must, first and foremost, meet the organization’s needs and good business 

practices.

2007: NDIA Systems Application Guide

NDIA’s Program Management Systems Committee’s working draft of its EVM Systems Application Guide

was published in March 2007. It describes for all organizations implementing the ANSI/EIA 748-A 

8NDIA System Acceptance Guide, © 2004–2005 National Defense Industrial Association (NDIA) Program Management Systems 

Committee (PMSC), NDIA PMSC Earned Value Management System Acceptance Guide (November 2006 released working draft).
320 APPENDIX V GAO-09-3SP

standard, EVM systems (Current Version), the importance of planning the EVM application through 

all phases of the acquisition life cycle. It elaborates on the performance-based management requirements 

in OMB’s Capital Programming Guide. The Systems Application Guide also provides the context for the 

application of EVM within a federal agency’s acquisition life cycle, along with government acquisition 

terminology.9

9NDIA System Application Guide, © 2007 National Defense Industrial Association (NDIA) Program Management Systems 

Committee (PMSC), Earned Value Management Systems Application Guide (March 2007 edition).
GAO-09-3SP APPENDIX VI 321

Data Collection Instrument

Job title_______________________________________________ Job code____________

Explain the job, identify the requester, and provide any other relevant information.

DATA REQUEST. Please provide copies of the following:

 1. Program life-cycle cost estimates and supporting documentation, showing the basis of the 

estimates (methodology, data sources, risk simulation inputs and results, software cost model 

inputs and results, etc.).

 2. Program management review briefings from the past year’s budget documentation, including 

projected budget and OMB 300 reports.

 3. The program’s contract.

 4. A short contract history, with a description of contract line item numbers, contract number 

and type, award date, and performance period and a summary of significant modifications 

(with cost and description).

 5. Award fee determination (or incentive) letters and any presentations by the contractor 

regarding award fee determination (e.g., self-evaluations).

 6. Price negotiation memos, also known as business clearance letters.

 7. Independent cost estimate briefings and supporting documentation.

 8. Nunn-McCurdy unit cost breach program reporting and certification documentation, if 

applicable.

 9. Work breakdown structure (WBS) or cost element structure (CES), with dictionary.

 10. The latest approved technical baseline description (TBD), also knows as cost analysis 

requirements description (CARD) in DOD and cost analysis data requirement CADRe at 

NASA.

 11. Current acquisition program baseline.

 12. Selected acquisition reports (SAR), if applicable.

 13. If DOD, cost and software data reporting (CSDR), or contract critical design review (CCDR) 

if NASA.

 14. Technology readiness assessments, if applicable.

 15. Design review reports, preliminary and critical.

 16. The acquisition decision memorandum.

 17. EVM contract performance reports (CPR), Formats 1-5, for the past 12 months, year-end for 

all prior years, and monthly thereafter during the audit—preferably electronic.

 18. All integrated baseline review (IBR) reports.

 19. EVM surveillance reports for the past 12 months and a standing request for monthly reports 

during the audit.

 20. The integrated mater schedule (IMS) in its native software format (e.g., *mpp).

 21. The integrated master plan (IMP).
322 APPENDIX VI GAO-09-3SP

CONTRACT QUESTIONS. Please answer the following questions:

1. Break down the program’s budget by contract, government in-house, and other costs. What 

percent of the program’s budget do the prime contract, major subcontracts, and government costs, 

subsume? Identify the quantities of the system to be procured, including planned options and 

foreign military sales, if applicable. 

2. Discuss any major contract modifications and how long it took to make the changes final.

3. Discuss the award fee structure, if applicable. Does the program use cost performance as a basis 

for determining award fee? Are contract performance report (CPR) data used? If not used, what is 

examined to determine award fees? 

4. Describe any applicable teaming arrangements.

PROGRAM MANAGEMENT AND COST. Please answer the following questions:

1. Who was responsible for developing the program’s life-cycle cost estimate? If a support contractor 

prepared the estimates, what requirements and guidelines were provided to the support contractor 

regarding the development of the estimate? What qualifications and experience do the cost analysts 

have? Was the estimate prepared by a centralized cost team outside of the program office? What 

types of cost data are available to the cost team? Are centralized databases and experts available to 

the cost team to support the development of the estimate? 

2. How often does the program present program management review briefings? How are decisions 

made and documented?

3. What are the program’s current risk drivers and associated rankings—high, medium, low? Please 

describe the effect of each risk. Is there a risk mitigation plan? If so, please describe it.

4. Describe significant cost and schedule drivers. Are there corrective actions plans to address them?

5. Has an independent cost estimate (ICE) been performed on the program’s life-cycle costs? If so, 

how much higher or lower was the ICE? How were the differences between the ICE and the 

program cost estimate reconciled? Who was briefed on the ICE?

6. Have any Monte Carlo simulations been run to determine the risk level associated with cost 

estimates? What were the results and how did they influence program decision regarding risk and 

funding?

7. How does the program procure equipment furnished by the government? Are there separate 

contracts for such items? If so, what is the value? How is such equipment accounted for in the 

program’s cost estimate? 

8. Who is responsible for absorbing cost overruns associated with equipment furnished by the 

government—the program or the program developing the item?

9. Please describe the program’s software requirements. How was the effort estimated in regard to 

size requirements and productivity rates? Were any software cost models used? What were the 

associated inputs?

10. Please discuss any effects inflation has had on the program and whether inflation has played a role 

in cost overruns.
GAO-09-3SP APPENDIX VII 323

Data Collection Instrument: 

Data Request Rationale

The items in this appendix are keyed to the “Data Request” items in appendix VI.

1. Program life-cycle cost estimates and supporting documentation, showing the basis of the estimates 

(methodology, data sources, risk simulation inputs and results, software cost model inputs and results, 

etc.).

Rationale: Only by assessing the estimate’s underlying data and methodology can the auditor 

determine its quality. This information will answer important questions such as, How applicable 

are the data? Were the data normalized correctly? What method was used? What statistics were 

generated? 

2. Program management review briefings from the past 2 years’ budget documentation, including 

budget and OMB 300 reports.

Rationale: This information tells the auditor what senior management was told and when the 

presentations were made—what problems were revealed, what alternative actions were discussed. 

Budget documentation assures the auditor that agencies are properly employing capital programming 

to integrate the planning, acquisition, and management of capital assets into the budget decisionmaking process. Agencies are required to establish cost, schedule, and measurable performance goals 

for all major acquisition programs and should achieve, on average, 90 percent of those goals.

3. The program’s contract.

Rationale: This tells the auditor what the contractor was required to deliver at a given time. It also 

provides price or cost information, including the negotiated price or cost, as well as the type of 

contract (such as fixed-price, cost-plus-fixed-fee, cost-plus-award, or incentive fee).

4. A short contract history, with a description of contract line item numbers, contract number and type, 

award date, and performance period and a summary of significant contract modifications (with cost 

and description). 

Rationale: This provides important context for the current contract. Only with a detailed knowledge 

of program history can the auditor effectively determine the program’s present status and future 

prospects.

5. Award fee determination (or incentive) letters and any contractor presentations regarding award fee 

determination (e.g., self-evaluations).

Rationale: This obviously applies only to contracts with award (or incentive) fees. For such contracts, 

the auditor needs to know the basis on which fees were awarded, whether it was strictly followed, and 

reasons for any deviations.

6. Price negotiation memos, also known as business clearance letters.

Rationale: The price negotiation memorandum summarizes for the auditor the contract price 

negotiations, including documentation of fair and reasonable pricing.
324 APPENDIX VII GAO-09-3SP

7. Independent cost estimate briefings and supporting documentation. 

Rationale: This information is important because, first, it provides the auditor with the data needed 

to assess the quality of the LCCE and, second, it reveals what information was independently briefed 

to senior management about the quality of the baseline cost estimate.

8. Nunn-McCurdy unit cost breach program reporting and certification documentation, if applicable.

Rationale: This will not apply at all to non-DOD programs and applies only to certain DOD 

programs. For DOD programs (major defense acquisition programs), it is important that the auditor 

know the nature of the breach, when it occurred, when it was reported, and what action was taken.

9. Work breakdown structure (WBS) or cost element structure (CES), with dictionary.

Rationale: The WBS and CES and associated dictionary represent a hierarchy of product-oriented 

elements that provide a detailed understanding of what the contractor was required to develop and 

produce.

10. The latest approved technical baseline description, also known as cost analysis requirements document 

in DOD and CADRe at NASA. 

Rationale: The technical baseline description provides the auditor with the program’s technical and 

program baseline. Besides defining the system, it provides complete information on testing plans, 

procurement schedules, acquisition strategy, and logistics plans. This is the document on which cost 

analysts base their estimates and is therefore essential to the auditor’s understanding of the program.

11. Current acquisition program baseline. 

Rationale: The acquisition program baseline documents program goals before program initiation. 

The program manager derives the acquisition program baseline from the users’ performance 

requirements, schedule requirements, and best estimates of total program cost consistent with 

projected funding. The baseline should contain only the parameters that, if thresholds are not met, 

will require the milestone decision authority to reevaluate the program and consider alternative 

program concepts or design approaches,

12. Selected acquisition reports (SAR), if applicable

Rationale: For major defense acquisition programs, the SAR provides the history and current status 

of total program cost, schedule, and performance, as well as program unit cost and unit cost breach 

information. For joint programs, SARs provide information by participant. Each SAR includes a full, 

life-cycle cost analysis for the reporting program; an analysis of each of its evolutionary increments, as 

available; and analysis of its antecedent program, if applicable.

13. If DOD, cost and software data reporting, or contractor critical design review, if NASA. 

Rationale: Contractor critical design reviews provide the auditor with actual contractor development 

or procurement costs by WBS or CES. Especially useful is the fact that recurring and nonrecurring 

costs are differentiated. 

14. Technology readiness assessment, if applicable.
GAO-09-3SP APPENDIX VII 325

Rationale: A technology readiness assessment provides an evaluation of a system’s technological 

maturity by major WBS elements. It is extremely useful in countering technological overoptimism. 

For elements with unacceptable assessments, the auditor can then assess whether satisfactory 

mitigation plans have been developed to ensure that acceptable maturity will be achieved before 

milestone decision dates.

15. Design review reports, preliminary and critical.

Rationale: Design review reports provide the technical information needed to ensure that the system 

is satisfactorily meeting its requirements. The preliminary design review ensures that the system can 

proceed into detailed design, while meeting its stated performance requirements within cost (program 

budget), schedule (program schedule), risk, and other system constraints. The critical design review 

ensures that the system can proceed into system fabrication, demonstration, and test, while meeting 

its stated performance requirements within cost, schedule, risk, and other system constraints. It also 

assesses the system’s final design as captured in product specifications for each configuration item in 

the system (product baseline) and ensures that each product in the product baseline has been captured 

in the detailed design documentation.

16. The acquisition decision memorandum.

Rationale: This provides the documented rationale for the milestone decision authority’s (or 

investment review board’s) approving a program to advance to the next stage of the acquisition 

process.

17. EVM contract performance reports, Formats 1–5, for the past 12 months, year-end for all prior years, 

and monthly thereafter during the audit—preferably electronic. 

Rationale: CPRs are management reports essential to an auditor’s ability to develop a comprehensive 

analysis. They are timely, reliable summary data from which to assess current and projected contract 

performance. The auditor can use them to reasonably project future program performance. Format 

1 provides data to measure cost and schedule performance by product-oriented WBS elements—i.e., 

hardware, software, and services the government is buying. Format 2 provides the same data by the 

contractor’s organization (functional or integrated product team structure). Format 3 provides the 

budget baseline plan against which performance is measured. Format 4 provides staffing forecasts 

for correlation with the budget plan and cost estimates. Format 5 is a narrative report explaining 

significant cost and schedule variances and other identified contract problems and topics.

18. All integrated baseline review reports.

Rationale: An IBR’s purpose is to verify the technical content and realism of the interrelated 

performance budgets, resources, and schedules. It helps the auditor understand the inherent risks in 

offerors’ or contractors’ performance plans and the underlying management control systems, and it 

should contain a plan to handle these risks. OMB policy requires that IBRs be initiated as early as 

practicable.
326 APPENDIX VII GAO-09-3SP

19. EVM surveillance reports for the past 12 months and a standing request for monthly CPRs during 

the audit.

Rationale: EVM surveillance reports assure the auditor that contractors are using effective internal 

cost and schedule control systems that provide contractor and government managers with timely and 

auditable data to effectively monitor programs, provide timely indications of actual and potential 

problems, meet requirements, and control contract performance. Surveillance ensures that a supplier’s 

EVM implementation of processes and procedures is being maintained over time and on all applicable 

programs and is in compliance with the 32 EVM guidelines.

20. The integrated master schedule (IMS).

Rationale: The IMS contains the detailed tasks or work packages necessary to ensure program 

execution. The auditor can use the IMS to verify the attainability of contract objectives, evaluate 

progress toward program objectives, and integrate the program schedule activities with the program 

components.

21. The integrated master plan.

Rationale: The integrated master plan provides an event-based hierarchy of program events, with each 

event supported by accomplishments and each accomplishment associated with specific criteria to be 

satisfied for its completion. The plan is normally part of the contract and is therefore contractually 

binding.
GAO-09-3SP APPENDIX VIII 327

SEI Checklist

Checklists and Criteria contains a checklist for evaluating an organization’s software and is available at 

www.sei.cmu.edu/pub/documents/95.reports/pdf/sr005.95.pdf.
328 APPENDIX IX GAO-09-3SP

Examples of Work Breakdown Structures

DOD developed Work Breakdown Structures for Defense Materiel Items in 1968 to provide a framework 

and instructions for developing a WBS.10 Although it now serves only as guidance, the handbook 

remains an excellent resource for developing a WBS for government and private industry. It outlines 

the contents and components that should be considered for aircraft, electronic and automated software 

systems, missiles, ordnance, ships, space systems, surface vehicle systems, and unmanned air vehicle 

systems. It gives examples and definitions, particularly in its appendixes A–I, which constitute the bulk 

of the document and on which tables 51, 52, and 54–59 are based. These examples of WBS were valid 

at the time of publication. It is advised that the source of the WBS be checked before it is used to see 

if any updates have been made. Table 53 presents a common WBS for software development based on 

NASA research conducted by the Jet Propulsion Laboratory at the California Institute of Technology. 

Table 60 shows the Department of Energy’s Project WBS. Table 61 is an example of the General Services 

Administration’s construction WBS, and table 62 is an Automated Information System: Enterprise 

Resource Planning Program Level Work Breakdown Structure. Tables 63–67 are from the Project 

Management Institute’s Practice Standard for Work Breakdown Structures, second edition, published in 

October 2006. Table 68 is an example of a major construction renovation project. Table 69 includes 

IT infrastructure and IT services only. Automated information system configuration, customization, 

development, and maintenance are covered in chapter 8. Table 70 is an example from the CSI 

MasterformatTM 2004 Structure. 

Table 51: Aircraft System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Air vehicle 1.1.1 Airframe

1.1.2 Propulsion

1.1.3 Air vehicle applications software

1.1.4 Air vehicle system software

1.1.5 Communications/identification

1.1.6 Navigation/guidance

1.1.7 Central computer

1.1.8 Fire control

1.1.9 Data display and controls

1.1.10 Survivability

1.1.11 Reconnaissance

1.1.12 Automatic flight control

1.1.13 Central integrated checkout

1.1.14 Antisubmarine warfare

1.1.15 Armament

1.1.16 Weapons delivery

1.1.17 Auxiliary equipment

1.1.18 Crew station

10DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, 

D.C.: OUSD (AT&L), July 3, 2005).
GAO-09-3SP APPENDIX IX 329

Level 2 element Level 3 element

1.2 Systems engineering/program management

1.3 System test and evaluation 1.3.1 Development test and evaluation

1.3.2 Operational test and evaluation

1.3.3 Mock-ups/system integration laboratories

1.3.4 Test and evaluation support

1.3.5 Test facilities

1.4 Training 1.4.1 Equipment

1.4.2 Services

1.4.3 Facilities

1.5 Data 1.5.1 Technical publications

1.5.2 Engineering data

1.5.3 Management data

1.5.4 Support data

1.5.5 Data depository

1.6 Peculiar support equipment 1.6.1 Test and measurement equipment

1.6.2 Support and handling equipment

1.7 Common support equipment 1.7.1 Test and measurement equipment

1.7.2 Support and handling equipment

1.8 Operational/site activation 1.8.1 System assembly, installation, checkout 

1.8.2 Contractor technical support

1.8.3 Site construction

1.8.4 Site/ship/vehicle conversion

1.9 Industrial facilities 1.9.1 Construction/conversion/expansion

1.9.2 Equipment acquisition or modernization

1.9.3 Maintenance (industrial facilities)

1.10 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. A.

Table 52: Electronic/Automated Software System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Prime mission product 1.1.1 Subsystem 1...n (specify names)

1.1.2 Prime mission product applications 

software

1.1.3 Prime mission product system software

1.1.4 Integration, assembly, test, checkout

1.2 Platform integration

1.3 Systems engineering/program management
330 APPENDIX IX GAO-09-3SP

Level 2 element Level 3 element

1.4 System test and evaluation 1.4.1 Development test and evaluation

1.4.2 Operational test and evaluation

1.4.3 Mock-ups/system integration labs

1.4.4 Test and evaluation support

1.4.5 Test facilities

1.5 Training 1.5.1 Equipment

1.5.2 Services

1.5.3 Facilities

1.6 Data 1.6.1 Technical publications

1.6.2 Engineering data

1.6.3 Management data

1.6.4 Support data

1.6.5 Data depository

1.7 Peculiar support equipment 1.7.1 Test and measurement equipment

1.7.2 Support and handling equipment

1.8 Common support equipment 1.8.1 Test and measurement equipment

1.8.2 Support and handling equipment

1.9 Operational/site activation

1.10 System assembly, installation, checkout 1.10.1 Contractor technical support

1.10.2 Site construction

1.10.3 Site/ship/vehicle conversion

1.11 Industrial facilities 1.11.1 Construction/conversion/expansion

1.11.2 Equipment acquisition/modernization

1.11.3 Maintenance (industrial facilities)

1.12 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. B.

Table 53: Ground Software Work Breakdown Structure 

Level 2 element Level 3 element

1.1 Software management 1.1.1 General management/control activities

1.1.2 Software risk management

1.1.3 Arrange and conduct reviews

1.1.4 General documentation support 

1.1.5 Secretarial/clerical

1.1.6 Administrative support 

1.1.7 Information technology/computer support

1.1.8 Other expenses
GAO-09-3SP APPENDIX IX 331

Level 2 element Level 3 element

1.2 Software systems engineering 1.2.1 Functional design document

1.2.2 Requirements specification

1.2.3 Software interface documents

1.2.4 Configuration management

1.2.5 Procurement

1.2.6 User manuals

1.2.7 Ops concept

1.2.8 Concept document

1.2.9 Trade-off studies 

1.2.10 Review preparation

1.3 Software function i (i = 1...n) 1.3.1 Management and control activities

1.3.2 High-level design

1.3.3 Detailed design, code, and unit test

1.3.4 Data

1.4 Software development test bed 1.4.1 Test engineering support

1.4.2 Test bed development

1.4.3 Simulators and test environment

1.4.4 Test bed support software

1.4.5 Test bed computers

1.5 Software integration and test 1.5.1 Subsystem software integration test plan

1.5.2 Software test plans and procedures 

1.5.3 Support subsystem integration and test

1.5.4 System integration and test

1.6 Software quality assurance 1.6.1 Software product assurance plan

1.6.2 Software assurance activities 

1.7 Delivery and transfer to operations 1.7.1 End user training

Source: NASA.

Table 54: Missile System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Air vehicle 1.1.1 Propulsion (stages 1...n)

1.1.2 Payload

1.1.3 Airframe

1.1.4 Reentry system

1.1.5 Post boost system

1.1.6 Guidance and control

1.1.7 Ordnance initiation set

1.1.8 Airborne test equipment

1.1.9 Airborne training equipment

1.1.10 Auxiliary equipment

1.1.11 Integration, assembly, test, checkout
332 APPENDIX IX GAO-09-3SP

Level 2 element Level 3 element

1.2 Command and launch 1.2.1 Surveillance, identification, tracking

1.2.2 Sensors

1.2.3 Launch and guidance control

1.2.4 Communications

1.2.5 Command/launch applications software

1.2.6 Command and launch system software

1.2.7 Launcher equipment

1.2.8 Auxiliary equipment

1.2.9 Booster adapter

1.3 Systems engineering/program management 1.3.1 System test and evaluation

1.3.2 Development test and evaluation

1.3.3 Operational test and evaluation

1.3.4 Mock-ups/system integration laboratories

1.3.5 Test and evaluation support

1.3.6 Test facilities

1.4 Training 1.4.1 Equipment

1.4.2 Services

1.4.3 Facilities

1.5 Data 1.5.1 Technical publications

1.5.2 Engineering data

1.5.3 Management data

1.5.4 Support data

1.5.5 Data depository

1.6 Peculiar support equipment 1.6.1 Test and measurement equipment

1.6.2 Support and handling equipment

 1.7 Common support equipment 1.7.1 Test and measurement equipment

1.7.2 Support and handling equipment

1.8 Operational/site activation

1.9 System assembly, installation, checkout 1.9.1 Contractor technical support

1.9.2 Site construction

1.9.3 Site/ship/vehicle conversion

1.10 Industrial facilities 1.10.1 Construction/conversion/expansion

1.10.2 Equipment acquisition/modernization

1.10.3 Maintenance (industrial facilities)

1.11 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. C.
GAO-09-3SP APPENDIX IX 333

Table 55: Ordnance System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Complete round 1.1.1 Structure

1.1.2 Payload

1.1.3 Guidance and control

1.1.4 Fuze

1.1.5 Safety/arm

1.1.6 Propulsion

1.1.7 Integration, assembly, test, checkout

1.2 Launch system 1.2.1 Launcher

1.2.2 Carriage

1.2.3 Fire control

1.2.4 Ready magazine

1.2.5 Adapter kits

1.2.6 Integration, assembly, test, checkout

1.3 Systems engineering/program management

1.4 System test and evaluation 1.4.1 Development test and evaluation

1.4.2 Operational test and evaluation

1.4.3 Mock-ups/system integration laboratories

1.4.4 Test and evaluation support

1.4.5 Test facilities

1.5 Training 1.5.1 Equipment

1.5.2 Services

1.5.3 Facilities

1.6 Data 1.6.1 Technical publications

1.6.2 Engineering data

1.6.3 Management data

1.6.4 Support data

1.6.5 Data depository

1.7 Peculiar support equipment 1.7.1 Test and measurement equipment

1.7.2 Support and handling equipment

1.8 Common support equipment 1.8.1 Test and measurement equipment

1.8.2 Support and handling equipment

1.9 Operational/site activation

1.10 System assembly, installation, checkout 1.10.1 Contractor technical support

1.10.2 Site construction

1.10.3 Site/ship/vehicle conversion

1.11 Industrial facilities 1.11.1 Construction/conversion/expansion

1.11.2 Equipment acquisition/modernization

1.11.3 Maintenance (industrial facilities)

1.12 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. D.
334 APPENDIX IX GAO-09-3SP

Table 56: Sea System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Ship 1.1.1 Hull structure

1.1.2 Propulsion plant

1.1.3 Electric plant

1.1.4 Command/communication/surveillance

1.1.5 Auxiliary systems

1.1.6 Outfit and furnishings

1.1.7 Armament

1.1.8 Total ship integration/engineering

1.1.9 Ship assembly and support services

1.2 Systems engineering/program management

1.3 System test and evaluation 1.3.1 Development test and evaluation

1.3.2 Operational test and evaluation

1.3.3 Mock-ups/system integration laboratories

1.3.4 Test and evaluation support

1.3.5 Test facilities

1.4 Training 1.4.1 Equipment

1.4.2 Services

1.4.3 Facilities

1.5 Data 1.5.1 Technical publications

1.5.2 Engineering data

1.5.3 Management data

1.5.4 Support data

1.5.5 Data depository

1.6 Peculiar support equipment 1.6.1 Test and measurement equipment

1.6.2 Support and handling equipment

1.7 Common support equipment 1.7.1 Test and measurement equipment

1.7.2 Support and handling equipment

1.8 Operational/site activation 1.8.1 System assembly, installation, checkout 

1.8.2 Contractor technical support

1.8.3 Site construction

1.8.4 Site/ship/vehicle conversion

1.9 Industrial facilities 1.9.1 Construction/conversion/expansion

1.9.2 Equipment acquisition/modernization

1.9.3 Maintenance (industrial facilities)

1.10 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. E.
GAO-09-3SP APPENDIX IX 335

Table 57: Space System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Systems engineering, integration, and test; program 

management; and other common elements

1.2 Space vehicle (1...n as required) 1.2.1 Systems engineering, integration, 

and test; program management; 

and other common elements

1.2.2 Spacecraft bus

1.2.3 Communication/payload

1.2.4 Booster adapter

1.2.5 Space vehicle storage

1.2.6 Launch systems integration

1.2.7 Launch operations & mission 

support

1.3 Ground (1...n as required) 1.3.1 Systems engineering, integration, 

and test; program management; 

and other common elements

1.3.2 Ground terminal subsystems

1.3.3 Command and control subsystem

1.3.4 Mission management subsystem

1.3.5 Data archive/storage subsystem

1.3.6 Mission data processing subsystem

1.3.7 Mission data analysis and 

dissemination subsystem

1.3.8 Mission infrastructure subsystem

1.3.9 Collection management subsystem

1.4 Launch vehicle

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. F. 
336 APPENDIX IX GAO-09-3SP

Table 58: Surface Vehicle System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Primary vehicle 1.1.1 Hull/frame

1.1.2 Suspension/steering

1.1.3 Power package/drive train

1.1.4 Auxiliary automotive

1.1.5 Turret assembly

1.1.6 Fire control

1.1.7 Armament

1.1.8 Body/cab

1.1.9 Automatic loading

1.1.10 Automatic/remote piloting

1.1.11 Nuclear, biological, chemical

1.1.12 Special equipment

1.1.13 Navigation

1.1.14 Communications

1.1.15 Primary vehicle application software

1.1.16 Primary vehicle system software

1.1.17 Vetronics

1.1.18 Integration, assembly, test, checkout

1.2 Secondary vehicle 1.1.1–18 (Same as primary vehicle)

1.3 Systems engineering/program management 1.3.1 System test and evaluation

1.3.2 Development test and evaluation

1.3.3 Operational test and evaluation

1.3.4 Mock-ups/system integration lab 

1.3.5 Test and evaluation support

1.3.6 Test facilities

1.4 Training 1.4.1 Equipment

1.4.2 Services

1.4.3 Facilities

1.5 Data 1.5.1 Technical publications

1.5.2 Engineering data

1.5.3 Management data

1.5.4 Support data

1.5.5 Data depository

1.6 Peculiar support equipment 1.6.1 Test and measurement equipment

1.6.2 Support and handling equipment

1.7 Common support equipment 1.7.1 Test and measurement equipment

1.7.2 Support and handling equipment
GAO-09-3SP APPENDIX IX 337

Level 2 element Level 3 element

1.8 Operational/site activation 1.8.1 System assembly, installation, checkout

1.8.2 Contractor technical support

1.8.3 Site construction

1.8.4 Site/ship/vehicle conversion

 1.9 Industrial facilities 1.9.1 Construction/conversion/expansion

1.9.2 Equipment acquisition/modernization

1.9.3 Maintenance (industrial facilities)

1.10 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. G.

Table 59: Unmanned Air Vehicle System Work Breakdown Structure

Level 2 element Level 3 element

1.1 Air vehicle 1.1.1 Airframe

1.1.2 Propulsion

1.1.3 Communications/identification

1.1.4 Navigation/guidance

1.1.5 Central computer

1.1.6 Auxiliary equipment

1.1.7 Air vehicle application software

1.1.8 Air vehicle system software

1.1.9 Integration, assembly, test, checkout

1.2 Payload (1...n) 1.2.1 Survivability

1.2.2 Reconnaissance

1.2.3 Electronic warfare

1.2.4 Armament

1.2.5 Weapons delivery

1.2.6 Payload application software

1.2.7 Payload system software

1.2.8 Integration, assembly, test, checkout

1.3 Ground segment 1.3.1 Ground control systems

1.3.2 Command and control subsystem

1.3.3 Launch and recovery equipment

1.3.4 Transport vehicles

1.3.5 Ground segment application software

1.3.6 Ground segment system software

1.3.7 Integration, assembly, test, checkout

1.4 System integration, assembly, test

1.5 Systems engineering/program management
338 APPENDIX IX GAO-09-3SP

Level 2 element Level 3 element

1.6 System test and evaluation 1.6.1 Development test and evaluation

1.6.2 Operational test and evaluation

1.6.3 Mock-ups/system integration laboratories

1.6.4 Test and evaluation support

1.6.5 Test facilities

1.7 Training 1.7.1 Equipment

1.7.2 Services

1.7.3 Facilities

1.8 Data 1.8.1 Technical publications

1.8.2 Engineering data

1.8.3 Management data

1.8.4 Support data

1.8.5 Data depository

1.9 Peculiar support equipment 1.9.1 Test and measurement equipment

1.9.2 Support and handling equipment

1.10 Common support equipment 1.10.1 Test and measurement equipment

1.10.2 Support and handling equipment

1.11 Operational/site activation 1.11.1 System assembly, installation, checkout

1.11.2 Contractor technical support

1.11.3 Site construction

1.11.4 Site/ship/vehicle conversion

1.12 Industrial facilities 1.12.1 Construction/conversion/expansion

1.12.2 Equipment acquisition/modernization

1.12.3 Maintenance (industrial facilities)

1.13 Initial spares and repair parts

Source: DOD, Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MIL-HDBK-881A (Washington, D.C.: OUSD (AT&L), July 3, 2005), app. H.
GAO-09-3SP APPENDIX IX 339

Table 60: Department of Energy Project Work Breakdown Structure

Level 2 element Level 3 element Level 4 element

1.1 Fuel processing 

facility

1.1.1 Conceptual design 1.1.1.1 Conceptual design

1.1.1.2 Criteria development & conceptual design 

reviews

1.1.2 Design 1.1.2.1 Definitive design

1.1.2.2 CADD consultant

1.1.2.3 Engineering support during construction

1.1.3 Government 

furnished equipment

1.1.3.1 Construction preparation

1.1.3.2 Building

1.1.3.3 Process & service systems & equipment

1.1.3.4 Quality assurance

1.1.4 Construction 1.1.4.1 Construction preparation

1.1.4.2 Building

1.1.4.3 Process & service systems & equipment

1.1.4.4 Construction inspection

1.1.4.5 Construction management

1.1.4.6 Construction services

1.1.4.7 Constructability reviews

1.1.5 Project 

administration

1.1.5.1 Project control

1.1.5.2 Records management

1.1.5.3 Support services

1.1.5.4 Engineering

1.1.5.5 Independent construction cost estimate

1.1.6 Systems 

development

1.1.6.1 Process development

1.1.6.2 Design support

1.1.6.3 Plant liaison

1.1.6.4 Computer/control system development

1.1.7 Startup 1.1.7.1 SO test preparation

1.1.7.2 SO test performance

1.1.7.3 Manuals

1.1.7.4 Integrated testing

1.1.7.5 Cold run

1.1.7.6 SO test resources

1.1.7.7 Deleted

1.1.7.8 Preventative maintenance

1.1.8 Safety/

environmental

1.1.8.1 Environmental assessment

1.1.8.2 PSD document

1.1.8.3 Safety analysis report

1.1.8.4 Probabilistic risk assessment

1.1.8.5 Document coordination

1.1.8.6 RAM study

1.1.8.7 Hazardous waste
340 APPENDIX IX GAO-09-3SP

Level 2 element Level 3 element Level 4 element

1.2 Liquid effluent 

treatment & 

disposal

1.2.1 Construction

1.2.2 Government 

furnished material

1.2.3 Construction 

inspection

1.2.4 Project 

administration

1.2.5 Design

Source: Department of Energy, Work Breakdown Structures for Defense Materiel Items (Washington, D.C.: June 2003).

Table 61: General Services Administration Construction Work Breakdown Structure

Level 2 element Level 3 element

1.1 Superstructure 1.1.1 Foundations 

1.1.2 Basement construction

1.2 Exterior 

enclosure

1.2.1 Exterior walls

1.2.2 Exterior glazing & doors

1.2.3 Roofing

1.3 Interior 

construction

1.3.1 Partitions, doors & specialties

1.3.2 Access/platform floors

1.3.3 Interior finishes

1.4 Conveyance 

systems

1.4.1 Conveyance systems

1.4.2 Plumbing

1.4.3 HVAC

1.4.4 Fire protection/alarm 

1.4.5 Electrical service distribution & emergency power

1.4.6 Lighting and branch wiring

1.4.7 Communications, security & other electrical systems

1.5 Equipment & 

furnishing

1.5.1 Equipment & furnishings

1.6 Special 

construction, 

demolition & 

abatement

1.6.1 Special construction

1.6.2 Building demolition & abatement

1.7 Site work 1.7.1 Site work building related

1.7.2 Other site work project related

Source: General Services Administration, Project Estimating Requirement (Washington, D.C.: January 2007).
GAO-09-3SP APPENDIX IX 341

Table 62: Automated Information System: Enterprise Resource Planning Program Level Work 

Breakdown Structure

Level 2 element Level 3 element

1.1 Configuration, customization, 

development

1.1.1 Site prototype design and development

foundations 

1.1.2 Product application solution basement construction

1.1.3 System integration & test

1.1.4 Systems engineering/program management/change 

management

1.1.5 System test & evaluation

1.1.6 Training

1.1.7 Data

1.1.8 Reserved

1.1.9 Reserved

1.1.10 Reserved

1.1.11 Industrial facilities

1.1.12 Initial spares

1.2 Operational/site implementation 1.2.1 Site type 1

1.2.2 Site type 2; Site type n

1.2.3 System integration & test

1.2.4 Systems engineering/program management/change 

management

1.2.5 System test & evaluation

1.2.6 Training

1.2.7 Data

1.2.8 Reserved

1.2.9 Reserved

1.2.10 Reserved

1.2.11 Industrial facilities

1.2.12 Initial spares

1.3 Sustainment 1.3.1 Management

1.3.2 Sustaining engineering

1.3.3 COTS software maintenance and renewal

1.3.4 Custom software maintenance

1.3.5 Annual operations investment

1.3.6 Tech refresh

1.3.7 Recurring training

1.3.8 Hardware maintenance

1.3.9 Help desk support

1.4 Systems engineering/

program management/change 

management

1.4.1 Systems engineering

1.4.2 Program management

1.4.3 Change management

1.5 Data

Source: U.S. Air Force.
342 APPENDIX IX GAO-09-3SP

Table 63: Environmental Management Work Breakdown Structure

Level 2 element Level 3 element

1.1 System design 1.1.1 Initial design

1.1.2 Client meeting

1.1.3 Draft design

1.1.4 Client and regulatory agency meeting

1.1.5 Final design

1.2 System installation 1.2.1 Facility planning meeting

1.2.2 Well installation

1.2.3 Electrical power drop installation

1.2.4 Blower and piping installation

1.3 Soil permeability test 1.3.1 System operation check

1.3.2 Soil permeability test

1.3.3 Test report

1.4 Initial in situ respiration test 1.4.1 In situ respiration test

1.4.2 Test report

1.5 Long-term bioventing test 1.5.1 Ambient air monitoring

1.5.2 Operation, maintenance, and monitoring

1.5.3 Three-month in situ respiration test

1.5.4 Test report

1.5.5 Six-month in situ respiration test

1.5.6 Test report

1.6 Confirmation sampling 1.6.1 Soil boring and sampling

1.6.2 Data validation

1.7 Report preparation 1.7.1 Predraft report

1.7.2 Client meeting

1.7.3 Draft report

1.7.4 Client and regulatory agency meeting

1.7.5 Final report

1.8 Project management

Source: Project Management Institute, Practice Standards for Work Breakdown Structures, Project Management Institute, Inc. (2006). Copyright and all rights reserved. Material 

from this publication has been reproduced with the permission of PMI.

Table 64: Pharmaceutical Work Breakdown Structure

Level 2 element Level 3 element

1.1 Project initiation 1.1.1 Decision to develop business case

1.1.2 Business case

1.1.3 Project initiation decision

1.2 Marketing/sales support 1.2.1 Market research program

1.2.2 Branding program

1.2.3 Pricing program

1.2.4 Sales development program

1.2.5 Other marketing/sales support
GAO-09-3SP APPENDIX IX 343

Level 2 element Level 3 element

1.3 Regulatory support 1.3.1 IND submission

1.3.2 End of Phase 2 meeting

1.3.3 BLA/NDA submission

1.3.4 Postapproval regulatory support program

1.4 Lead identification program 1.4.1 Hypothesis generation

1.4.2 Assay screening

1.4.3 Lead optimization

1.4.4 Other discovery support

1.5 Clinical pharmacology 

support

1.5.1 Pharmacokinetic studies

1.5.2 Drug interaction studies

1.5.3 Renal effect studies

1.5.4 Hepatic effect studies

1.5.5 Bioequivalency studies

1.5.6 Other clinical pharmacology studies

1.6 Preclinical program 1.6.1 Tox/ADME support

1.6.2 Clinical pharmacology support

1.7 Phase I clinical study program 1.7.1 Pharmacokinetic/pharmacodynamic studies

1.7.2 Dose ranging studies

1.7.3 Multiple dose safety studies

1.8 Phase II clinical study program 1.8.1 Multiple dose efficacy studies

1.8.2 Other clinical studies

1.9 Phase III clinical study 

program

1.9.1 Pivotal registration studies

1.9.2 Other clinical studies

1.10 Submission/launch phase 1.10.1 Prelaunch preparation

1.10.2 Launch

1.10.3 Post-launch support

1.11 Phase/commercialization 

clinical study program

1.11.1 Investigator-sponsored studies

1.11.2 Registry studies

1.12 Legal support 1.12.1 Publications

1.12.2 Patents/intellectual property

1.12.3 Trademarks

1.12.4 Other legal support

1.13 Program management 

support

1.13.1 Program-level project management

1.13.2 Preclinical project management

1.13.3 Clinical project management

1.13.4 CM&C project management

1.13.5 Other project management support

Source: Project Management Institute, Practice Standards for Work Breakdown Structures, Project Management Institute, Inc. (2006). Copyright and all rights reserved. Material 

from this publication has been reproduced with the permission of PMI.
344 APPENDIX IX GAO-09-3SP

Table 65: Process Plant Construction Work Breakdown Structure

Level 2 element Level 3 element

1.1 Plant system design 1.1.1 Business requirements

1.1.2 Process models

1.2 Construction 1.2.1 Site development

1.2.2 Civil structure

1.2.3 Thermal systems

1.2.4 Flow systems

1.2.5 Storage systems

1.2.6 Electrical systems

1.2.7 Mechanical systems 

1.2.8 Instrument and control systems

1.2.9 Environmental systems

1.2.10 Temporary structure

1.2.11 Auxiliary systems

1.2.12 Safety systems

1.3 Legal and regulatory 1.3.1 Licensing (nongovernment)/permitting (government)

1.3.2 Environmental impact

1.3.3 Labor agreements

1.3.4 Land acquisition

1.4 Testing 1.4.1 System test

1.4.2 Acceptance test

1.5 Startup

1.6 Project management

Source: Project Management Institute, Practice Standards for Work Breakdown Structures, Project Management Institute, Inc. (2006). Copyright and all rights reserved. Material 

from this publication has been reproduced with the permission of PMI.

Table 66: Telecon Work Breakdown Structure

Level 2 element Level 3 element

1.1 Concept/feasibility 1.1.1 Concept

1.1.2 Marketing analysis

1.1.3 Market plan

1.1.4 Technical analysis

1.1.5 Product scope definition

1.1.6 Prototype

1.2 Requirements 1.2.1 End-user requirements

1.2.2 Application requirements

1.2.3 Infrastructure (systems) requirements

1.2.4 Operations/maintenance requirements

1.2.5 Service requirements
GAO-09-3SP APPENDIX IX 345

Level 2 element Level 3 element

1.3 Go/no go decision 1.3.1 Prototype review

1.3.2 Financial review 

1.3.3 Schedule review

1.3.4 Technical capabilities review

1.3.5 Financial commitment review

1.3.6 Go/no-go decision 

1.4 Development 1.4.1 End-user systems

1.4.2 Application

1.4.3 Infrastructure systems

1.4.4 Network

1.4.5 Operations/maintenance systems

1.4.6 Service plan

1.5 Testing 1.5.1 Test plans

1.5.2 Tests

1.5.3 Results

1.5.4 Corrective actions 

1.5.5 Retests

1.5.6 Retest results

1.6 Deployment 1.6.1 Trial in a nonpenalty environment

1.6.2 First action site

1.6.3 Deployment 

1.7 Life-cycle support 1.7.1 Customer training & education

1.7.2 Turnover to customer

1.7.3 Customer acceptance

1.7.4 Support & maintenance

1.8 Project management

Source: Project Management Institute, Practice Standards for Work Breakdown Structures, Project Management Institute, Inc. (2006). Copyright and all rights reserved. Material 

from this publication has been reproduced with the permission of PMI.

Table 67: Software Implementation Project Work Breakdown Structure

Level 2 element Level 3 element

1.1 Project management

1.2 Product requirements 1.2.1 Software requirements

1.2.2 User documentation

1.2.3 Training program materials

1.2.4 Hardware

1.2.5 Implementation & future support

1.3 Detail software design 1.3.1 Initial software design

1.3.2 Final software design

1.3.3 Software design approval
346 APPENDIX IX GAO-09-3SP

Level 2 element Level 3 element

1.4 System construction 1.4.1 Configured software

1.4.2 Customized user documentation

1.4.3 Customized training program materials

1.4.4 Installed hardware

1.4.5 Implementation & future support

1.5 Test 1.5.1 System test plan

1.5.2 System test cases

1.5.3 System test results

1.5.4 Acceptance test plan

1.5.5 Acceptance test cases

1.5.6 Acceptance test results

1.5.7 Approved user documentation

1.6 Go live

1.7 Support 1.7.1 Training

1.7.2 End user support

1.7.3 Product support

Source: Project Management Institute, Practice Standards for Work Breakdown Structures, Project Management Institute, Inc. (2006). Copyright and all rights reserved. Material 

from this publication has been reproduced with the permission of PMI.

Table 68: Major Renovation Project Work Breakdown Structure

Level 2 element Level 3 element

1.1 Requirements 1.1.1 Requirements 

1.1.2 Planning

1.1.3 Design

1.2 Construction 1.2.1 Move out

1.2.2 Entrances

1.2.3 Preconstruction

1.2.4 Core and shell

1.3 Tenant fit out 1.3.1 Tenant fit out construction

1.3.2 Security

1.3.3 Furniture, fixture, equipment

1.3.4 Move-in

1.3.5 Commissioning

1.4 Information management and 

telecomms

1.4.1 Design and engineering

1.4.2 Temporary and transitional communication

1.4.3 Backbone basic

1.4.4 Tenant specific

1.4.5 Systems

Source: DOD, Pentagon Renovation Program. 
GAO-09-3SP APPENDIX IX 347

Table 69: Sample IT Infrastructure and Service Work Breakdown Structure

The table 69 sample WBS includes IT infrastructure and IT services only. Automated information system 

configuration, customization, development, and maintenance are in chapter 8 in this guide. 

Level 1 element Level 2 element Level 3 element

1.0 IT project investment 

(nonrecurring)

1.1 Facility Type 1 – n (e.g., buildings, 

flooring, cooling)

1.1.1 Infrastructure site construction

1.1.2 Operational site construction

1.1.3 Integration/test facility construction

1.2 Purchased software licenses 

(e.g., application software, system 

software, database)

1.3 Infrastructure purchased hardware 

(e.g., UNIX servers, Windows servers, 

WAN/LAN equipment)

1.4 End user purchased hardware 

(e.g., switches, PC, printers, copiers)

1.5 Operational/site implementation 

(aka deployment) (e.g., system 

architecture, training, hardware & 

software setup)

1.5.1 IT systems architecture/design

1.5.2 Software/database services (e.g., 

deploying & supporting enterprise 

applications, databases, data migration, 

middleware and software services)

1.5.3 Infrastructure hardware & software 

installation, activation, & checkout 

(e.g., setup, deploy, test, checkout 

infrastructure systems such as servers, 

storage systems, & networks)

1.5.4 End user hardware & software 

installation, activation, & checkout: 

(e.g., labor to setup & deploy end user 

systems such as PCs, notebooks, PDAs, 

communication devices, and other 

mobile devices)

1.5.5 Data (e.g., user documentation 

preparation labor)

1.5.6 Training development (e.g., course 

development)

1.5.7 Initial training (e.g., personnel 

training)

1.5.8 Data migration

1.5.9 Operational test & evaluation (e.g., 

labor and material to test and certify the 

overall IT project)

1.6 Management 1.6.1 Government program office

1.6.2 Contractor program management
348 APPENDIX IX GAO-09-3SP

Level 1 element Level 2 element Level 3 element

2.0 Operations & support 

(recurring)

2.1 IT facility operations type 1 2.1.1 IT facilities maintenance & support

2.1.2 Power

2.2 Facility operations type 2

2.3 Facility operations type n

2.4 Purchased software maintenance 

(e.g., ongoing licensing)

2.4.1 Application software

2.4.2 System software

2.4.3 Database

2.5 Purchased hardware maintenance 2.5.1 Infrastructure

2.5.2 End user

2.6 Change architecture/design

2.7 Purchased software and hardware 

refresh (e.g., new hardware, new 

software, spares)

2.8 IT project operations & monitoring 2.8.1 System administration

2.8.2 Database administration

2.8.3 Help desk support (Tier I, II, III)

2.8.4 Security

2.8.5 Other IT operations & monitoring 

(e.g., hardware maintenance, computer 

operations)

2.8.6 Data maintenance (e.g., 

documentation review & update labor)

2.8.7 Recurring training (e.g., end users, 

developers, IT operations personnel)

2.8.8 Data migration update

2.8.9 Management

Source: GAO, DOD, and industry expert collaboration.

The MasterFormatTM and OmniClassTM Work Classification System

Many standard project construction breakdown structures have been created over the years for use in 

construction management. The most common, in existence since the 1960s, are the CSI (Construction 

Specifications Institute) format in North America and the SMM7 (Standard Method of Measurement) 

format in Great Britain.11 They originated as breakdowns for commercial building construction but have 

evolved to include other forms of construction. 

CSI introduced an expanded version, the MasterFormatTM, in 2004 that includes 50 divisions of work 

covering civil site and infrastructure work as well as process equipment—a significant increase from the 

previous 16 divisions covering building construction that had been in use for years. This expansion reflects 

the growing complexity of the construction industry, as well as the need to incorporate facility life cycle 

11More information is available on CSI’S Web site, www.csinet.org/s_csi/index.asp. The Construction Specifications Institute 

“maintains and advances the standardization of construction language as it pertains to building specifications.” It also provides 

structured guidelines for specification writing in its Project Resource Manual: CSI Manual of Practice, 5th ed. (New York: McGrawHill, 2004). 
GAO-09-3SP APPENDIX IX 349

and maintenance information into the building knowledge base. Another level of standardized numbers 

was added to the publication. One goal was to eventually facilitate building information modeling to 

contain project specifications. 

The MasterFormatTM standard serves as the organizational structure for construction industry publications 

such as the Sweets catalog, with a wide range of building products; MasterSpec and other popular 

master guide specification applications, and RS Means and other cost information applications. 

MasterFormat helps architects, engineers, owners, contractors, and manufacturers classify the typical use 

of various products to achieve technical solutions on the job site, known as “work results.” Work results 

are permanent or temporary aspects of construction projects achieved in the production stage or by 

subsequent alteration, maintenance, or demolition processes, through the application of a particular skill 

or trade to construction resources. 

The OmniClassTM Construction Classification System, a new North American classification system, is 

useful for many additional applications, from organizing library materials, product literature, and project 

information to providing a classification structure for electronic databases.12 It incorporates other systems 

in use as the basis of many of its tables, including MasterFormat for work results and UniFormatTM for 

elements.

OmniClass follows the international framework set out in International Organization for Standardization 

(ISO) Technical Report 14177—Classification of Information in the Construction Industry, July 1994. 

This document has been established as a standard in ISO 12006-2: Organization of Information about 

Construction Works—Part 2: Framework for Classification of Information. 

It is also worth noting that CSI is involved in developing a corresponding system for terminology 

based on a related ISO standard, ISO 12006-3: Organization of Information about Construction 

Works—Part 3: Framework for Object-Oriented Information. The system known as the International 

Framework for Dictionaries (IFD) Library is a standard for terminology libraries or ontologies. It is 

part of the international standards for building information modeling being developed and promoted 

by buildingSMART International (bSI). CSI sees the IFD Library being used in conjunction with 

OmniClass to establish a controlled vocabulary for the North American building industry, thereby 

improving interoperability. Both OmniClass and the IFD Library are included in the development work 

of the buildingSmart alliance (the North American chapter of bSI) and its National Building Information 

Modeling Standard (NBIMS).

OmniClass consists of 15 tables, each representing a different facet of construction information. Each table 

can be used independently to classify a particular type of information, or entries on it can be combined 

with entries on other tables to classify more complex subjects. The tables are not numbered sequentially 

and there are gaps in the progression. The first is table 11, Construction Entities by Function, and the last 

of the 15 is table 49, Properties. 

12More information is available from the OmniClass Web site at www.omniclass.org. The OmniClassTM material included here is used 

with permission. Copyright © 2006 the Secretariat for the OCCS Development Committee. All Rights Reserved. www.omniclass.org, 

Edition 1.0, 2006-03-28 Release. 
350 APPENDIX IX GAO-09-3SP

The OmniClass structures start to approach the DOD WBS template model at the system level regarding 

construction classifications. In its table 21 under “Utilities and Infrastructure” is included breakdowns 

for roadways, railways, airports, space travel, utilities, and water-related construction. This is not unlike 

a concept in aircraft systems, electronic systems, missile systems, and ship systems from the DOD Mil 

Handbook template.

OmniClass table 22 is based almost entirely on the CSI MasterFormat tables, although it is noted in 

OmniClass that “some content of MasterFormat 2004 Edition is not included in table 22.”13

None of the current construction breakdowns, including CSI, fully cover the complete civil infrastructure 

project life cycle, including development, engineering, construction, operations, maintenance, and risk 

mitigation. The current CSI MasterFormat 2004 edition comes closest to covering all the scope of work 

found in the construction of building facilities and site work. It falls short in addressing the unique 

requirements of program managers, estimators, schedulers, and cost engineers and in identifying all phases 

of work included in major infrastructure work such as Build Own and Transfer programs. 

These structures, MasterFormat, and OmniClass are not program work breakdown structures, although 

some subsections have the appearance of a WBS. However, at all levels the elements in the structures 

are candidates for WBS element descriptors, including work packages, and they meet the common 

definitions of WBS elements, being all nouns or nouns and adjectives. The MasterFormat tables include 

the equivalent of a WBS dictionary for the lowest levels. 

Many listings available in MasterFormat would enable an organization to pick and choose to provide 

ready-to-go WBS elements for virtually any work on any construction project, including related 

equipment and furnishings. It must be noted, however, that the summary headings are not truly WBS 

elements, since the breakdown or listings under the headings are further listings of categories within a 

heading and do not meet the WBS 100 percent rule. 

Figure 43 illustrates the relationship of the CSI MasterFormat structure to a WBS based on the CSI 

structure. (A true WBS would be based on the actual product structure.) The summary CSI elements 

are listed as the 34 divisions. Each division contains one or more sections that would be selected from 

the complete MasterFormat set to relate to the specific needs of the project. Also, although not shown, 

the specific physical breakdown of the building needs to be overlaid. For example, it would be normal 

for the individual floors to be identified and the appropriate work packages for each floor selected from 

the appropriate MasterFormat sections. Note, also, that there is no further breakdown of the project 

management element as would be the case in a true WBS. 

13OmniClassTM Edition 1.0 May 2, 2006 p. 22-ii.
GAO-09-3SP APPENDIX IX 351

Figure 43: MasterFormatTM Work Breakdown Structure

Construction

phase

Base building

construction

Facility

construction

General

requirements

Project

management

Process

equipment

Site and

infrastructure

Facility

services

Other

construction

or installation

WBS 

Level 1 CSI 2004 structure

WBS

Level 2

WBS

Level 3

CSI

subgroups

01 General

requirements

02

03

04

05

06

07

08

09

10

11

12

13

14

Existing conditions

Concrete

Masonry

Metals

Wood, plastics, and

composites

Thermal and moisture

protection

Openings

Finishes

Specialties

Equipment

Furnishings

Special construction

Conveying equipment

21

22

23

25

26

27

28

Fire suppression

Plumbing

Heating, ventilating,

and air conditioning

Integrated automation

Electrical

Communications

Electronic safety and

security

31

32

33

34

35

Earthwork

Exterior

improvements

Utilities

Transportation

Waterway and

marine construction

40

41

42

43

44

45

48

Process integration

Material processing and

handling equipment

Process heating,

cooling, and drying

equipment

Process gas and liquid

handling, purification,

and storage equipment

Pollution control

equipment

Industry-specific

manufacturing

equipment

Electrical power

generation

WBS

Level 4

CSI

divisions

34 divisions

Note: The CSI numbering contains gaps or

“reserved” division titles

Reference: Levels 3 and 4: Constructions

Specifications Institute, MasterFormatTM 2004,

page Div Numbers-1.

Source: CSI and CSC MasterFormatTM 2004 (c) 2006.

Building Information Modeling/Management (BIM) Applications 

OmniClass, MasterFormat, and UniFormat are used to index, organize, and retrieve a variety of different 

information types throughout a project’s life cycle. The consistent use of standard classifications from 

any of these, applied to objects, will enhance users’ ability to sort data or roll up or drill down through 

data based on the hierarchy that all these classifications are built on. A standard implementation of any of 

these classifications within a BIM model will allow for this same information sorting and retrieval across 

multiple platforms and by all users at any stage in a facility’s life cycle.

In conjunction with the IFD Library, the structure of the classification systems can be explicitly 

applied to the information used in model-based design, analysis, and management systems. A more 

consistent naming system for objects captured in a BIM has the potential to support the goals of the 

buildingSMART organization to improve interoperability of systems and processes. In North America, 

these systems are used by the buildingSmart alliance (the North American chapter of buildingSMART 

International) in pilot projects and in the development of the U.S. National Building Information 

Modeling Standard (NBIMS).
352 APPENDIX IX GAO-09-3SP

Table 70: CSI MasterFormatTM 2004 Structure Example: Construction Phase

Level 2 element Level 3 CSI subgroup Level 4 CSI division

1.1 Base construction Project management

General requirements General requirements

Facility construction Existing conditions

Concrete

Masonry

Metals

Wood, plastics, composites

Thermal and moisture protection

Openings

Finishes

Specialties

Equipment

Furnishings

Special construction

Conveying equipment

Facility services Fire suppression

Plumbing

Heating, ventilating, air conditioning

Integrated automation

Electrical

Communications

Electronic safety and security

Site and infrastructure Earthwork

Exterior Improvements

Utilities

Transportation

Waterway and marine construction

Process equipment Process integration

Material processing and handling equipment

Process heating, cooling & drying equipment

Process gas and liquid handling, purification and 

storage equipment

Pollution control equipment

Industry specific manufacturing equipment

Electrical power generation

1.2 Other construction or 

installation

Source: CSI MasterFormatTM 2004© 2006. 

Note: The numbers and titles used in this publication are from MasterFormatTM 2004, published by The Construction Specifications 

Institute (CSI) and Construction Specifications Canada (CSC), and are used with permission from CSI. For those interested in a more 

in-depth explanation of MasterFormatTM 2004 and its use in the construction industry, visit or contact The Construction Specifications 

Institute (CSI), 99 Canal Center Plaza, Suite 300, Alexandria, VA 22314. 800-689-2900; 703-684-0300.
GAO-09-3SP APPENDIX X 353

Schedule Risk Analysis

A schedule risk analysis uses statistical techniques to predict a level of confidence in meeting a program’s 

completion date. This analysis focuses on critical path activities and on near-critical and other activities, 

since any activity may potentially affect the program’s completion date. Like a cost estimate risk and 

uncertainty analysis, a schedule risk analysis requires the collection of program risk data such as

■ risks that may jeopardize schedule success, usually found in the risk register prepared before the risk 

analysis is conducted;

■ probability distributions, usually specified by a point estimate of activity durations;

■ probability of a risk register risk’s occurring and its probability distribution of impact if it were to 

occur;

■ probability that a branch of activities might occur (for example, a test failure could lead to several 

recovery tasks); and 

■ correlations between activity durations.

Schedule risk analysis relies on Monte Carlo simulation to randomly vary the following:

■ activity durations according to their probability distributions or 

■ risks according to their probability of occurring and the distribution of their impact on affected 

activity if they were to occur and

■ existence of a risk’s or a probabilistic branch’s occurring.

The objective of the simulation is to develop a probability distribution of possible completion dates that 

reflect the program and its quantified risks. From the cumulative probability distribution, the organization 

can match a date to its degree of risk tolerance. For instance, an organization might want to adopt a 

program completion date that provides a 70 percent probability that it will finish on or before that date, 

leaving a 30 percent probability that it will overrun, given the schedule and the risks. The organization 

can thus adopt a plan consistent with its desired level of confidence in the overall integrated schedule. This 

analysis can give valuable insight into what-if drills and quantify the impact of program changes. 

In developing a schedule risk analysis, probability distributions for each activity’s duration have to be 

established. Further, risk in all activities must be evaluated and included in the analysis. Some people focus 

only on the critical path, but because we cannot know the durations of the activities with certainty, we 

cannot know the true critical path. Consequently, it would be a mistake to focus only on the deterministic 

critical path when some off-critical path activity might become critical if a risk were to occur. Typically, 

three-point estimates—that is, best, most likely, and worst case estimates—are used to develop the 

probability distributions for the duration of workflow activities. After the distributions are developed, the 

Monte Carlo simulation is run and the resulting cumulative distribution curve, the S curve, displays the 

probability associated with the range of program completion dates. 

If the analysis is to be credible, the program must have a good schedule network that clearly identifies the 

critical path and that is based on a minimum number of date constraints. The risk analysis should also 

identify the tasks during the simulation that most often ended up on the critical path, so that near-critical 

path activities can also be closely monitored. It is important to represent all work in the schedule, since 
354 APPENDIX X GAO-09-3SP

any activity can become critical under some circumstances. Complete schedule logic that addresses the 

logical relationships between predecessor and successor activities is also important. The analyst needs to 

be confident that the schedule will automatically calculate the correct dates and critical paths when the 

activity durations change, as they do thousands of times during a simulation. Because of debugging, and 

because the collection of schedule risk data can take time and resources, it is often a good idea to work 

with a summary schedule rather than the most detailed schedule.

One of the most important reasons for performing a schedule risk analysis is that the overall program 

schedule duration may well be greater than the sum of the path durations for lower-level activities. This is 

in part because of

■ schedule uncertainty, which can cause activities to shorten (an opportunity) or lengthen (a threat). 

For instance, if a conservative assumption about labor productivity was used in calculating the 

duration of an activity, and during the simulation a better labor productivity is chosen, then the 

activity will shorten. However, most program schedule risk phenomena exhibit more probability of 

overrunning (threats) than underrunning (opportunities), which can cause activities to lengthen. 

■ schedule structure. A schedule’s structure has many parallel paths joined at a merge or join point, 

which can cause the schedule to lengthen. Merge points include program reviews (preliminary 

design review, critical design review, etc.) or the beginning of an integration and test phase. The 

timing of these merge points is determined by the latest merging path. Thus, if a late required 

element is delayed, the merge event will also be delayed. Since any merging path can be risky, any 

merging path can determine the timing of the merge event. This added risk at merge points is 

called the “merge bias.” Figure 44 gives an example of the schedule structure that illustrates the 

network or pure-logic diagram of a simple schedule with a merge point at integration and test.

Figure 44: Network Diagram of a Simple Schedule

Finish

Milestone Date: Mon 4/20/09

ID:13

Test Integrated System

Start:2/24/09

Finish:4/20/09

ID:12

Dur:40d

Res:

Integrate Units 1 & 2

Start:12/16/08

Finish:12/23/09

ID:11

Dur:50d

Res:

Test Unit 1

Start:11/11/08

Finish:12/15/08

ID:5

Dur:25 d

Res:

Test Unit 2

Start:11/18/08

Finish:12/15/08

ID:9

Dur:25d

Res:

Build Unit 1

Start:6/24/08

Finish:11/10/08

ID:4

Dur:100 d

Res:

Build Unit 2

Start:6/24/08

Finish:11/10/08

ID:8

Dur:100d

Res:

Design Unit!

Start:4/29/08

Finish:6/23/08

ID:3

Dur:40 d

Res:

Design Unit 2

Start:4/29/08

Finish:6/23/08

ID:7

Dur:40d

Res:

Source: Copyright 2007 Hulett and Associates, LLC.

Start

Unit 1

Start: 4/29/08

Finish:12/15/08

ID:2

Dur:165 d

Comp:0%

Unit 2

Start:4/29/08

Finish:12/15/08

ID:6

Dur:165 d

Comp:0%

Integration and Test

Start: 12/16/08

Finish:4/20/09

ID:10

Dur:90d

Comp:0%

Milestone date:Tue 4/29/08

ID: 1

Since each activity has an uncertain duration, it stands to reason that the entire duration of the overall 

program schedule will also be uncertain. Therefore, unless a statistical simulation is run, calculating 

the completion date based on schedule logic and the most likely duration distributions will tend to 

underestimate the overall program critical path duration. 
GAO-09-3SP APPENDIX X 355

Schedule underestimation is more pronounced when the schedule durations or logic have optimistic 

bias—for instance when the customer or management has specified an unreasonably short duration or 

early completion date. The schedule can be configured and assumptions made about activity durations 

to make a schedule match these imposed constraint durations. When this is the case, durations are often 

“best case” scenarios or based on unreasonable assumptions about resources availability or productivity. 

Further, the schedule may overlap activities or phases (for example, detailed engineering, fabrication, and 

testing) that would otherwise be more prudently scheduled in a series to compress the time. In addition, 

external contributions may be assumed with optimistic bias when there is little confidence that their 

suppliers will be able to comply. As a result, fitting the schedule to predetermined dates is dangerous. 

The preferred approach to scheduling is to build the schedule by starting with the WBS to define the 

detailed activities using program objectives to guide major events. When determining the durations for 

the activities, resource availability and productivity need to be reasonably assumed, external factors need to 

be realistically considered, and organizational risk associated with other programs and the priority of this 

program need to be considered. Once all these aspects have been modeled in the schedule, the scheduling 

system software can calculate the completion date. Following these best practice approaches to developing 

a schedule will provide a reasonable first step in determining the schedule duration. If the duration is too 

long, or the dates are too late, then more resources or less scope may be required. Unless more resources 

are provided, it is inappropriate to shorten the schedule to fit a preconceived date, given the original scope 

of work.

Accordingly, because activity durations are uncertain, the probability distribution of the program’s 

total duration must be determined statistically, by combining the individual probability distributions 

of all paths according to their risks and the logical structure of the schedule. Schedule activity duration 

uncertainty can be represented several ways. The example schedule in figure 45 illustrates.

Figure 45: Example Project Schedule 

Task name 2009 2010

Qtr 4 Qtr 1 Qtr 2 Qtr 3 Q Qtr 4 tr 1 Qtr 2

Three path project GAO

 Start

 Software

 Software design

 Software coding

 Software testing

 Hardware

 Hardware design

 Hardware fabrication

 Hardware test

 Integration H/W and S/W

 Integration

 Integrated test

 Finish

ID

0

1

2

3

4

5

6

7

8

9

10

11

12

13

Duration

750 d

0 d

470 d

100 d

250 d

120 d

500 d

100 d

300 d

100 d

250 d

150 d

100 d

0 d

Start

1/9/08

1/9/08

1/9/08

1/9/08

4/18/08

12/24/08

1/9/08

1/9/08

4/18/08

2/12/09

5/23/09

5/23/09

10/20/09

1/27/10

Finish

1/27/10

1/9/08

4/22/09

4/17/08

12/23/08

4/22/09

5/22/09

4/17/08

2/11/09

5/22/09

1/27/10

10/19/09

1/27/10

1/27/10

12/24

12/23

4/22

10/20

5/22

2/11

2/12 5/22

5/23 10/19

Source: Copyright 2007 Hulett and Associates, LLC.

In this example schedule, the project begins on January 9, 2008, and is expected to be completed about 

2 years later, on January 27, 2010. Three major efforts involve software, hardware, and integration. 
356 APPENDIX X GAO-09-3SP

According to the schedule logic and durations, hardware fabrication, testing, and the integration of 

hardware and software drive the critical path.

The first way to capture schedule activity duration uncertainty is to collect various estimates from 

individuals and, perhaps, from a review of actual past program performance. Estimates derived from 

interviews or in workshops should be formulated by a consensus of knowledgeable technical experts and 

coordinated with the same people who manage the program’s risk mitigation watch list. Figure 46 shows a 

traditional approach with the three-point estimate applied directly to the activity durations.

Figure 46: Estimated Durations for Remaining WBS Areas in the Schedule

ID Task name Rept ID

0

1

2

3

4

5

6

7

8

9

10

11

12

13

Three path GAO project

 Start

 Software

 Software design

 Software coding

 Software testing

 Hardware

 Hardware design

 Hardware fabrication

 Hardware test

 Integration H/W and S/W

 Integration

 Integrated test

 Finish

Mn Rdur

0 d

0 d

0 d

85 d

212.5 d

90 d

0 d

85 d

255 d

75 d

0 d

120 d

75 d

0 d

ML Rdur

0 d

0 d

0 d

100 d

250 d

120 d

0 d

100 d

300 d

100 d

0 d

150 d

100 d

x d

Max Rdur

0 d

0 d

0 d

150 d

375 d

240 d

0 d

130 d

390 d

200 d

0 d

210 d

200 d

0 d

2

0

0

0

0

0

0

0

0

0

0

0

0

0

Rept ID

0

0

0

2

2

2

0

2

2

2

0

2

2

0

Source: Copyright 2007 Hulett and Associates, LLC.

Note: Rept ID = Report Identification; Rdur = Remaining Duration, Mn = Minimum, ML = Most Likely, Max = Maximum; H/W = 

hardware; S/W = software. 

The example shows three-point estimates of remaining durations. In a real program schedule risk analysis, 

these would be developed from in-depth interviews of people who are expert in each of the WBS areas 

of the program. To model the risks in the simulation, the risks are represented as triangular distributions 

specified by the three-point estimates of the activity durations. These probability distributions combine 

the effects of all risks that affect the activities. 

Once the distributions have been established, the Monte Carlo simulation uses random numbers to 

select specific durations from each activity probability distribution and calculates a new critical path and 

dates, including major milestone and program completion. The Monte Carlo simulation continues this 

random selection thousands of times, creating a new program duration estimate and critical path each 

time. The resulting frequency distribution displays the range of program completion dates along with the 

probabilities that these dates will occur. 

Figure 47 shows that the most likely completion date is about May 11, 2010, not January 27, 2010, which 

is the date the deterministic schedule computed. The cumulative distribution also shows that a January 27, 

2010, completion is less than 5 percent likely, given the schedule and the risk ranges used for the durations. 

An organization that wants to cover 80 percent of its known unknowns would have to add a time reserve 

of about 5 months to June 24, 2010. While it would be prudent to establish a 5-month reserve for this 

project, each organization should determine its tolerance level for schedule risk.
GAO-09-3SP APPENDIX X 357

Figure 47: Cumulative Distribution of Project Schedule, Including Risk 

Date: 12/7/2007 5:35:45 PM

Samples: 5000

Unique ID: 0

Name: Three Path Project GAO

Completion Std Deviation: 51.79 d

95% Confidence Interval: 1.44 d

Each bar represents 20 d

Completion Date

Frequency

Cumulative Probability

11/28/09 5/11/10 11/12/10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16 Completion Probability Table

Prob Date Prob Date

0.05 2/19/10

0.10 3/6/10

0.15 3/18/10

0.20 3/29/10

0.25 4/5/10

0.30 4/12/10

0.35 4/19/10

0.40 4/25/10

0.45 5/2/10

0.50 5/9/10

0.55 5/15/10

0.60 5/22/10

0.65 5/29/10

0.70 6/6/10

0.75 6/15/10

0.80 6/24/10

0.85 7/5/10

0.90 7/20/10

0.95 8/8/10

1.00 11/12/10

GAO

Source: Copyright 2007 Hulett and Associates, LLC.

Schedule activity duration uncertainty can be determined by analyzing the probability of risks from the 

risk register. Using a probability distribution of the risk impact on the duration, risks are assigned to specific 

activities. This approach focuses on how risks affect time. Figure 48 shows how this approach can be used.

Figure 48: Identified Risks on a Spacecraft Schedule: An Example

ID

SUMMA

00001

00002

00003

00004

00005

00006

00007

00008

00009

00010

00011

00012

00020

00021

00022

00023

00024

00025

00026

00027

00028

00029

1900

1900

100

0

0

0

1450

300

0

450

0

600

100

1450

300

0

450

0

600

100

350

250

100

Description

Project summary

Spacecraft project milestones

Requirements definition spacecraft

PDR spacecraft

CDR spacecraft

Ship to launch site

First stage

FS preliminary design

FS PDR

FS final design

FS CDR

FS fabrication

Test FS engine

Upper stage

US preliminary design

US PDR

US Final design

US CDR

US fabrication

US test

Integration

Integration

Integration testing

Rem

duration

03/Mar/08

03/Mar/08

03/Mar/08

21/Jul/08

21/Jul/08

14/Sep/09

06/Jun/11

23/Sep/13

21/Jul/08

21/Jul/08

14/Sep/09

06/Jun/11

23/Sep/13

10/Feb/14

10/Feb/14

26/Jan/15

Start

12/Jun/15

12/Jun/15

18/Jul/08

11/Sep/09

03/Jun/11

12/Jun/15

07/Feb/14

11/Sep/09

11/Sep/09

03/Jun/11

03/Jun/11

20/Sep/13

07/Feb/14

07/Feb/14

11/Sep/09

11/Sep/09

03/Jun/11

03/Jun/11

20/Sep/13

07/Feb/14

12/Jun/15

23/Jan/15

12/Jun/15

Finish 2009 2010 2011 2012 2013 2014 2015

Source: Copyright 2007 Hulett and Associates, LLC.

11/Sep/09

11/Sep/09

03/Jun/11

03/Jun/11

03/Jun/11

12/Jun/15
358 APPENDIX X GAO-09-3SP

In this example of a spacecraft schedule, the work begins on March 3, 2008, and is expected to finish 

more than 7 years later, on June 12, 2015. Because of the long time and the risk associated with developing 

the spacecraft technology, the risk driver method can be used to examine how various risks from the risk 

register may affect this schedule (fig. 49). 

Figure 49: A Risk Register for a Spacecraft Schedule

Description

1

2

3

4

5

6

7

Requirements have not been decided

Several alternative designs considered

New designs not yet proven

Fabrication requires new materials

Lost know-how since last full spacecraft

Funding from Congress is problematic

Schedule for testing is aggressive

Optimistic

95.00%

95.00%

96.00%

96.00%

95.00%

90.00%

100.00%

Likelihood

30.00%

60.00%

40.00%

50.00%

30.00%

70.00%

100.00%

Pessimistic

120.00%

115.00%

112.00%

115.00%

105.00%

115.00%

130.00%

Most likely

105.00%

100.00%

103.00%

105.00%

100.00%

105.00%

120.00%

Source: Copyright 2007 Hulett and Associates, LLC.

In figure 49, one can quickly determine that the biggest risk affecting the spacecraft schedule has to do 

with testing, because the schedule is very aggressive. Moreover, lack of requirements, funding delays, 

alternative designs, and the fact that some of the designs are unproven are also highly likely to affect the 

schedule. With the risk driver method, these risks are shown as factors that will be used to multiply the 

durations of the activities they are assigned to, if they occur in the iteration. Once the risks are assigned to 

the activities, a simulation is run. The results may be similar to those in figure 50.

Figure 50: Spacecraft Schedule Results from a Monte Carlo Simulation

17/Dec/15

Distribution (start of interval)

0

100

200

300

400

500

600

700

Hits

 0% 11/Nov/14

 5% 04/May/15

 10% 25/Jun/15

 15% 17/Jul/15

 20% 03/Aug/15

 25% 14/Aug/15

 30% 27/Aug/15

 35% 09/Sep/15

 40% 23/Sep/15

 45% 06/Oct/15

 50% 21/Oct/15

 55% 06/Nov/15

 60% 26/Nov/15

 65% 17/Dec/15

 70% 11/Jan/16

 75% 05/Feb/16

 80% 03/Mar/16

 85% 06/Apr/16

 90% 16/May/16

 95% 04/Jul/16

 100% 19/Jan/17

Cumulative Frequency

Spacecraft Program

Entire Plan : Finish Date

Source: Copyright 2007 Hulett and Associates, LLC.
GAO-09-3SP APPENDIX X 359

In this example, the schedule date of June 12, 2015, is estimated to be 9 percent likely, based on 

the current plan. If the organization chooses the 80th percentile, the date would be March 3, 2016, 

representing a 9-month time contingency. Notice that the risks have caused a 14-month spread, a 

respectable range of uncertainty, between the 5 percent and 95 percent confidence dates.

Regardless of which method is used to examine schedule activity duration uncertainty, it is important 

to identify the risks that contribute most to the program schedule risk. In figure 51 is one approach to 

identifying activities that need close examination for effective risk mitigation. It compares a schedule 

with a well-managed critical path (unit 2) and two other paths that have risk but positive total float. The 

noncritical path efforts (units 1 and 3) therefore did not attract the program manager’s risk management 

attention. 

Figure 51: A Schedule Showing Critical Path through Unit 2

Task name 2008

Jan Feb Mar Apr May

Risk criticality project

 Start

 Unit 1

 Design unit 1

 Build unit 1

 Test unit 1

 Unit 2

 Design unit 2

 Build unit 2

 Test unit 2

 Unit 3

 Design unit 3

 Build unit 3

 Test unit 3

 Finish

Mn

Rdur

0 d

0 d

0 d

18 d

35 d

20 d

0 d

25 d

37 d

22 d

0 d

20 d

32 d

20 d

0 d

Ml

Rdur

0 d

0 d

0 d

28 d

40 d

25 d

0 d

30 d

40 d

25 d

0 d

30 d

37 d

25 d

0 d

Max

Rdur

0 d

0 d

0 d

43 d

50 d

50 d

0 d

38 d

45 d

35 d

0 d

45 d

47 d

50 d

0 d

Rept

ID

2

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Curve

0

0

0

2

2

2

0

2

2

2

0

2

2

2

0

2/5

2/6 3/16

3/17 4/10

2/7

2/8 3/15

3/16 4/9

4/12

2/7

2/8 3/18

3/19 4/12

Source: Copyright 2007 Hulett and Associates, LLC.

Note: Rept ID = Report Identification; Rdur = Remaining Duration, Mn = Minimum, ML = Most Likely, Max = Maximum; H/W = 

hardware; S/W = software. 

The measure of merit, the risk criticality, shows that the risky noncritical paths are more likely to delay 

the project than the so-called critical path. After running the simulation, which takes into account the 

minimum, most likely, and maximum durations, one can see that although unit 2 is on the schedule’s 

deterministic critical path, unit 1 is 44 percent likely to ultimately delay the project and unit 3 is 39 

percent likely to do the same. In other words, in the critical path method, “critical path” is the least likely 

path to delay the project, in this simple case.

Figure 52 shows the results of each unit’s probability of landing on the critical path, based on the Monte 

Carlo simulation.
360 APPENDIX X GAO-09-3SP

Figure 52: Results of a Monte Carlo Simulation for a Schedule Showing Critical Path through Unit 2

Task name 2008

Jan Feb Mar Apr May

Risk criticality project

 Start

 Unit 1

 Design unit 1

 Build unit 1

 Test unit 1

 Unit 2

 Design unit 2

 Build unit 2

 Test unit 2

 Unit 3

 Design unit 3

 Build unit 3

 Test unit 3

 Finish

ID

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Total

slack

0 d

0 d

2 d

2 d

2 d

2 d

0 d

0 d

0 d

0 d

3 d

3 d

3 d

3 d

0 d

Critical

Yes

Yes

No

No

No

No

Yes

Yes

Yes

Yes

No

No

No

No

Yes

% Critical

0

100

44

44

44

44

17

17

17

17

39

39

39

39

100

Risk

Critical

No

No

Yes

Yes

Yes

Yes

No

No

No

No

Yes

Yes

Yes

Yes

No

1/9

4

4

9

9

9

44

44 4/10

39

39

4/12

2/7

2/8 3/18

3/19 4/12

Source: Copyright 2007 Hulett and Associates, LLC.

Other measures of risk importance can be reviewed. For instance, sensitivity measures reflecting the 

correlation of the activities or the risks with the final schedule duration can be produced by most schedule 

risk software. Figure 53 is a standard schedule sensitivity index for the spacecraft project discussed earlier.

Figure 53: Sensitivity Index for Spacecraft Schedule

00005 - Test Unit 1

Risk Criticality Project

Duration Sensitivity

00013 - Test Unit 3

00003 - Design Unit 1

00011 - Design Unit 3

00004 - Build Unit 1

00012 - Build Unit 3

00007 - Design Unit 2

00009 - Test Unit 2

00008 - Build Unit 2

47%

42%

34%

29%

21%

18%

10%

8%

7%

Source: Copyright 2007 Hulett and Associates, LLC.

In this example, the testing and design of units 1 and 3 affect the schedule duration more than the 

design, testing, and building of unit 2, even though unit 2 represents the critical path in the deterministic 

schedule. Therefore, without taking into account the risk associated with each unit’s duration, the program 

manager would not know that keeping a strong eye on units 1 and 3 would be imperative for keeping the 

program on schedule.
GAO-09-3SP APPENDIX X 361

Figure 54 is a different view of final duration sensitivity resulting from the risk register risks themselves, 

using the risk driver approach discussed earlier. In this case, when a risk is assigned to several activities, 

its sensitivity measure reflects the entire correlation, not just the correlation of one activity to the project 

duration.

Figure 54: Evaluation of Correlation in Spacecraft Schedule 

6 - Funding from Congress is problematic

Driving Schedule Risk Factors

4 - Fabrication requires new materials

7 - Schedule for testing is aggressive

2 - Several alternative designs considered

3 - New designs not yet proven

5 - Lost know-how since last full spacecraft

1 - Requirements have not been decided

0.00

Correlation

0.20 0.40 0.60 0.80

Source: Copyright 2007 Hulett and Associates, LLC.

In the example in figure 54, funding from the Congress is the biggest risk driver in the program schedule, 

followed by new materials that may be needed for fabrication. While not much can be done about the 

congressional funding issue since this is an external risk, contingency plans can be made for several 

scenarios in which funding may not come through as planned. 

In addition to standard schedule risk and sensitivity analysis, events that typically occur in government 

programs require some new activities. This is called “probabilistic branching.” One such event that 

commonly occurs is the completion of a test of an integrated product (software program, satellite, etc.). 

The schedule often assumes that tests are successful, whereas experience indicates that tests may fail and 

only their failure will require the activities of root cause analysis, plan for recovery, execution of recovery, 

and retest. This is a branch that only happens with some probability. An example is shown in figure 55.

Figure 55: An Example of Probabilistic Branching Contained in the Schedule 

Task name 2008

May June July August September October

Project

 Start

 Design Unit

 Build Unit

 Test Unit

 FIXIT

 Retest

 Finish

ID

1

2

3

4

5

6

7

8

Duration

95 d

0 d

30 d

40 d

25 d

0 d

0 d

0 d

Start

6/1

6/1

6/1

7/1

8/10

9/3

9/3

9/3

Finish

9/3

6/1

6/30

8/9

9/3

9/3

9/3

9/3

Predecessor

2

3

4

5

6

7,5

6/1

6/1

7/1 8/9

8/10 9/3

9/3

9/3

9/3

6/30

Source: Copyright 2007 Hulett and Associates, LLC.
362 APPENDIX X GAO-09-3SP

If the test unit activity fails, FIXIT and retest occur; otherwise, their duration is 0 days. This is a 

discontinuous event that leads to the two new activities. If the test is estimated to fail with some 

probability such as 30 percent, the resulting probability distribution of dates for the entire project can be 

depicted as in figure 56.

Figure 56: Probability Distribution Results for Probabilistic Branching in Test Unit

Date: 1/9/2006 8:03:50 PM

Samples: 3000

Unique ID: 0

Name: Probabilistic Branch

Completion Std Deviation: 27.45 d

95% Confidence Interval: 0.98 d

Each bar represents 10 d

Completion Date

Frequency

Cumulative Probability

8/21 9/30 12/16

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.05

0.10

0.15

0.20

0.25

0.30 Completion Probability Table

Prob Date Prob Date

0.05 9/1

0.10 9/4

0.15 9/6

0.20 9/8

0.25 9/10

0.30 9/11

0.35 9/13

0.40 9/14

0.45 9/16

0.50 9/18

0.55 9/20

0.60 9/23

0.65 9/26

0.70 10/4

0.75 10/28

0.80 11/3

0.85 11/9

0.90 11/13

0.95 11/20

1.00 12/16

Source: Copyright 2007 Hulett and Associates, LLC.

Notice the bimodal distribution with the test success iterations on the left of figure 56 and the test failure 

iterations on the right. If the organization demands an 80th percentile schedule, it would be November 

3, although if it is satisfied by anything under the 70th percentile, the possibility of failure would not be 

important.

Other capabilities are possible once the schedule is viewed as a probabilistic statement of how the program 

might unfold. One that is notable is the correlation between activity durations. Correlation is when two 

activity durations are both influenced by the same external force and can be expected to vary in the same 

direction within their own probability distributions in any consistent scenario. While durations might 

vary in opposite directions if they are negatively correlated, this is less common than positive correlation 

in program management. Correlation might be positive and fairly strong if, for instance, the same 

assumption about the maturity of a technology is made to estimate the duration of design, fabrication, and 

testing activities. If the technology maturity is not know with certainty, it would be consistent to assume 

that design, fabrication, and testing activities would all be longer, or shorter, than scheduled together. It is 

the “together” part of the consistent scenario that represents correlation. 

Without specifying correlation between these activity durations in simulation, some iterations or scenarios 

would have some activities long and others short in their respective ranges. This would be inconsistent 

with the idea that they all react to the maturity of the same technology. Specifying correlations between 

design, fabrication, and testing ensures that each iteration represents a scenario in which their durations 

are consistently long or short in their ranges. Because schedules tend to add durations (given their 

logical structure), if the durations are long together or short together there is a chance for very long or 
GAO-09-3SP APPENDIX X 363

very short projects. How much longer or shorter depends, but without correlation, the risk analysis may 

underestimate the final effect. Figure 57 demonstrates this issue with a simple single-path hardware 

development fabrication test program.

Figure 57: A Project Schedule Highlighting Correlation Effects 

Task name 2008 2009

Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2

Correlation Project GAO

 Start

 Hardware Design

 Hardware Fabrication

 Hardware Test

 Finish

ID

0

1

2

3

4

5

Duration

500 d

0 d

100 d

300 d

100 d

0 d

Start

1/9/08

1/9/08

1/9/08

4/18/08

2/12/09

5/22/09

Finish

5/22/09

1/9/08

4/17/08

2/11/09

5/22/09

5/22/09

1/9

4/18 2/11

2/12 5/22

5/22

4/17

Source: Copyright 2007 Hulett and Associates, LLC.

Assuming no correlation between the activities’ durations, the result would be as shown in figure 58. In 

this uncorrelated case, the 80 percent probability date is August 10, 2009, and the standard deviation of 

completion date, a measure of dispersion, is 40.47 days.

Figure 58: Risk Results Assuming No Correlation between Activity Durations

Date: 12/7/2007 8:59:10 PM

Samples: 5000

Unique ID: 0

Name: Correlation Project GAO

Completion Std Deviation: 40.47 d

95% Confidence Interval: 1.12 d

Each bar represents 15 d

Completion Date

GAO No Correlations Frequency

Cumulative Probability

3/16/09 7/7/09 11/23/09

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.02

0.04

0.06

0.08

0.10

0.14

0.12

Completion Probability Table

Prob Date Prob Date

0.05 5/6/09

0.10 5/15/09

0.15 5/25/09

0.20 6/1/09

0.25 6/7/09

0.30 6/12/09

0.35 6/18/09

0.40 6/23/09

0.45 6/29/09

0.50 7/4/09

0.55 7/10/09

0.60 7/16/09

0.65 7/22/09

0.70 7/27/09

0.75 8/3/09

0.80 8/10/09

0.85 8/18/09

0.90 8/29/09

0.95 9/14/09

1.00 11/23/09

Source: Copyright 2007 Hulett and Associates, LLC.

However, if the influence of the technology maturity is strong and the program team believes that there 

is a 90 percent correlation between design, fabrication, and test of the hardware system, the simulation 

results will be affected dramatically. While the 90 percent correlation is high (correlation is measured 

between –1.0 and +1.0), there are often no actual data on correlation, so expert judgment is often used 

to set the correlation coefficients in many cases. Assuming this degree of correlation, we get the result in 

figure 59.
364 APPENDIX X GAO-09-3SP

Figure 59: Risk Results Assuming 90 Percent Correlation between Activity Durations

Date: 12/7/2007 9:05:20 PM

Samples: 5000

Unique ID: 0

Name: Correlation Project GAO

Completion Std Deviation: 62.59 d

95% Confidence Interval: 1.73 d

Each bar represents 15 d

Completion Date

GAO Correlation = 0.9 Frequency

Cumulative Probability

3/1/09 7/8/09 12/25/09

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10 Completion Probability Table

Prob Date Prob Date

0.05 4/6/09

0.10 4/21/09

0.15 5/2/09

0.20 5/11/09

0.25 5/20/09

0.30 5/28/09

0.35 6/6/09

0.40 6/14/09

0.45 6/22/09

0.50 7/1/09

0.55 7/10/09

0.60 7/19/09

0.65 7/29/09

0.70 8/9/09

0.75 8/21/09

0.80 9/4/09

0.85 9/19/09

0.90 10/7/09

0.95 10/28/09

1.00 12/25/09

Source: Copyright 2007 Hulett and Associates, LLC.

In this case the 80 percent probability date is September 4, 2009, nearly a month longer, and the standard 

deviation is 55 percent larger than when the activities were assumed independent. While the expected July 8, 

2009, date varied little from the uncorrelated risk analysis, the deterministic May 22, 2009, date increased in 

probability to more than 25 percent by correlating the risks. The two results are compared in figure 60. 

Figure 60: Schedule Analysis Results with and without Correlation 

2/6/09 3/28/09 5/17/09 7/6/09 8/25/09 10/14/09 12/3/09 1/22/10

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum. Prob. Effect of correlation

High correlation

No correlation

Source: Copyright 2007 Hulett and Associates, LLC.
GAO-09-3SP APPENDIX X 365

Other rules of thumb that can mitigate schedule risk include

■ break down longer activities to show critical handoffs—for example, if a task is 4 months long 

but a critical handoff is expected halfway through, the task should be broken down into separate 

2-month tasks that logically link the handoff between tasks. Otherwise, long lags must be used, 

which are rigid and tend to skew the risk results.

■ detailed program schedules should contain a predominance of finish-to-start logical relationships 

but summary schedules, typically used in risk analysis, may have more start-to-start and finish-tofinish relationships between phases. This practice requires care in completing the logic with the 

following rule:

Each activity needs a finish-to-start or start-to-start predecessor that drives it as 

well as a finish-to-start or finish-to-finish successor that it drives—in other words, 

dangling activities must be avoided. In this way, risks in predecessors and successors 

will be transmitted correctly down the paths to the later program milestones.

■ work packages in detailed program schedules should be no longer than 2 months so that work can 

be planned within two reporting periods, but for schedule risk a more summary schedule with 

basic activities representing phases is often used.

■ lags should represent only the passing of time on detailed program schedules and should never be 

used to replace a task, whereas in summary schedules used for schedule risk, the lags may have to 

be longer.

■ resources should be scheduled to reflect their scarcity, such as availability of staff or equipment.

■ constraints should be minimized, because they impose a movement restriction on tasks and can 

cause false dates in a schedule.

■ total float that is more than 5 percent of the total program schedule may indicate that the network 

schedule is not yet mature.

The schedule risk analysis aims to answer 11 fundamental questions:

1. Does the schedule reflect all work to be completed?

2. Are the program critical dates used to plan the schedule?

3. Are the activities sequenced logically?

4. Are activity interdependencies identified and logical?

5. If there are constraints, lags, and lead times, are they required, and is documentation available to 

justify the amounts? 

■ Constraints and lags should be used sparingly. There may be legitimate reasons for using 

constraints, but each constraint should be investigated. For instance, start-not-earlier-than 

constraints might reflect the availability of funding or a weather limitation and may be logical.

■ Finish-not-later-than constraints are usually artificial and reflect some policy rather than a 

program reality. If the program does not meet these dates, imposing this kind of constraint in a 
366 APPENDIX X GAO-09-3SP

computer model of the program schedule might make the schedule look good in the computer 

while the program is in trouble in the field.

■ Constraints that push the program activities beyond the dates that predecessors require in order 

to add float or flexibility are arbitrary and not recommended. The schedule risk analysis should 

determine the amount of time contingency needed.

6. How realistic are the schedule activity duration estimates?

7. How were resource estimates developed for each activity and will the resources be available when 

needed?

8. How accurate is the critical path and was it developed with scheduling software?

9. How reasonable are float estimates? Activities’ floats should be realistic. High total float values 

often indicate that logic is incorrect or missing and that there are dangling activities.

10. Can the schedule determine current status and provide reasonable completion date forecasts? 

11. What level of confidence is associated with the program schedule completion date? Does it reflect a 

schedule risk analysis and the organization’s or stakeholders’ risk tolerance?
GAO-09-3SP APPENDIX XI 367

Learning Curve Analysis

In this appendix, we describe the two ways to develop learning curves—unit formulation and cumulative 

average formulation—and discuss associated issues. 

Unit Formulation

Unit formulation (or unit theory) states that as the quantity of units doubles, unit cost decreases by a 

constant percentage. It is represented by the formula 

Y = AXb, where

Y = the cost of the Xth unit,

A = the first unit (T1) cost,

X = the unit number, and

b = the slope coefficient (defined as the Ln (slope) / Ln (2)).

What causes the cost to decrease as the quantity doubles is the rate of learning, depicted by b in the 

equation. Stated more simply, if the slope were 80 percent, then the value of unit 2 would be 80 percent of 

the value of the 1st unit, the 4th unit would be 80 percent of the value of the 2nd unit, and so on. As the 

quantity doubles, the cost reduces by the learning curve slope.

Cumulative Average Formulation

Cumulative average formulation is commonly associated with T. P. Wright, who initiated an important 

discussion of this method in 1936.14 The theory is that as the total quantity of units produced doubles, the 

cumulative average cost decreases by a constant percentage. This approach uses the same functional form 

as unit formulation, but it is interpreted differently:

Y = AXb, where

Y = the average cost of X units,

A = the first unit (T1) cost,

X = the cumulative number of units, and

b = the slope coefficient (defined as above).

In cumulative average theory, if the average cost of the first 10 units were $100 and the slope were 90 

percent, the average cost of the first 20 units would be $90, the average cost of the first 40 units would be 

$81, and so on. 

The difference between unit formulation and cumulative average theory is in where the curve affects the 

overall cost. For the first few units, using cumulative average will yield higher cost savings than using 

14T. P. Wright, “Factors Affecting the Cost of Airplanes,” Journal of Aeronautical Science 3:4 (1936): 122–28; reprinted in 

International Library of Critical Writings in Economics 128:3 (2001): 75–81.
368 APPENDIX XI GAO-09-3SP

a unit curve with the same slope. As the number of units increases, the difference between the results 

decreases.

Choosing between Unit Formulation and Cumulative Average

Choosing a formulation is not so much a science as an art. No firm rules would cause a cost estimator to 

select one approach over the other, but analyzing some factors can help decide which might best model the 

actual production environment. Some factors to consider when determining which approach to use are

1. analogous systems,

2. industry standards,

3. historic experience, and

4. expected production environment.

Analogous Systems

Systems that are similar in form, function, development, or production process may help justify choosing 

one method over the other. For example, if an agency is looking to buy a modified version of a commercial 

aircraft and unit curve were used to model the production cost for a previous version of a modified 

commercial jet, the estimator should choose unit formulation.

Industry Standards

Certain industries sometimes tend to prefer one method over the other. For example, some space systems 

have a better fit using cumulative average formulation. If an analyst were estimating one of these space 

systems, cumulative average formulation should be used, since it is an industry standard.

Historic Experience

Some contractors have a history of using one method over another because it models their production 

process better. The cost estimator should use the same method as the contractor, if the contractor’s 

method is known.

Expected Production Environment

Certain production environments favor one method over another. For example, cumulative average 

formulation best models production environments in which the contractor is just starting production with 

prototype tooling, has an inadequate supplier base, expects early design changes, or is subject to short lead 

times. In such situations, there is a risk of concurrency between the development and production phases. 

Cumulative averaging helps smooth out the initial cost variations and provides overall a better fit to the 

data. In contrast, unit formulation is a better fit for production environments where the contractor is well 

prepared to begin production in terms of tooling, suppliers, lead times, and so on. As a result, there is less 

need for the data to be smoothed out by averaging the results.

There are no firm rules for choosing one method over the other. Choosing between unit formulation and 

cumulative average formulation should be based on the cost estimator’s ability to determine which one 

best models the system’s costs.
GAO-09-3SP APPENDIX XI 369

Production Rate Effects and Breaks in Production

Not only do costs decrease as more units are produced but also costs usually decrease as the production 

rate increases. This effect can be modeled by adding a rate variable to the unit learning formulation. The 

equation then becomes

Y = AXbQr, where

Y, A, X, and b are as defined earlier,

Q = production rate (quantity per time period or lot), and

r = rate coefficient (Ln (slope) / Ln (2)).

This rate equation directly models cost reductions achieved by economies of scale. The rate at which 

items can be produced can also be affected by the continuity of production. Production breaks may occur 

because of program delays (budget or technical), time lapses between initial and follow-on orders, or labor 

disputes. Examining a production break can be divided into two questions:

■ How much learning has been lost (or forgotten) because of the break in production?

■ How will the learning loss affect the costs of future production items?

An analyst can answer the first question by using the Anderlohr method for estimating the loss of 

learning. The analyst can then determine the effect of the loss by using the retrograde method.

Anderlohr Method

When assessing the effect of a production break on costs, it is necessary first to quantify how much 

learning was achieved before the break and then to quantify how much of it was lost by the break. 

The Anderlohr method divides learning into five categories: personnel learning, supervisory learning, 

continuity of production, methods, and special tooling. Personnel learning loss occurs because of layoffs 

or removal of staff from the production line. Supervisory learning loss occurs when the number of 

supervisors is reduced because personnel have been reduced, so that supervisors who may no longer be 

familiar with the job are no longer able to provide optimal guidance. 

Learning can also be lost when production continuity changes because the physical configuration of 

the production line has moved or optimization for new workers is lacking. Methods are usually affected 

least by production breaks, as long as they are documented. However, revisions to the methods may be 

required if the tooling has to change once the production line restarts. Finally, tools may break during the 

production halt or may not be replaced when they are worn, causing productivity loss.

Each category must have a weight assigned to capture its effect on learning. The weights can vary by 

production situation but must always total 100 percent. To find the percentage of lost learning—known 

as the learning lost factor—the estimator must determine the learning lost factor in each category and 

then calculate the weighted average (see table 71).
370 APPENDIX XI GAO-09-3SP

Table 71: The Anderlohr Method for the Learning Lost Factor

Category Weight Learning lost Weighted loss

Personnel learning 30% 51% 0.1530

Supervisory learning 20 19 0.0380

Production continuity 20 50 0.1000

Tooling 15 5 0.0075

Methods 15 7 0.0105

Total learning lost 100% 0.3090 or 30.9%

Source: DOD.

In the table, if the production break were 6 months, the effect on learning would be almost a 31 percent 

reduction in efficiency, since the production line shut down. 

Retrograde Method

Assume that 10 units had been produced before the production break. The true cost of the first unit 

produced after the production break would then equal the cost of the 11th unit—assuming no production 

break—plus the 30.9 percent penalty from the lost learning. The retrograde method simply goes back up 

the learning curve to the unit (X) where that cost occurred. The number of units back up the curve is 

then the number of retrograde or lost units of learning. Production restarts at unit X rather than at unit 11. 

As illustrated by the Anderlohr and retrograde methods, costs increase as a result of production breaks. 

Cost estimators and auditors should question how the costs were estimated to account for learning that is 

lost, taking into account all factors that can be affected by learning.

Step-Down Functions

A step-down function is a method of estimating first unit production costs from prototype (or 

development) cost data. The first step is to account for the number of equivalent prototype units, based 

on both partial and complete units. This allows the cost estimator to capture the effects of units that are 

not entirely whole on the improvement curve. For example, if the development program includes a static 

article that represents 85 percent of a full aircraft, a fatigue article that represents 50 percent of a full 

aircraft, and three full aircraft, the development program would have 4.35 equivalent units. If the program 

is being credited with learning in development, the first production unit would then be unit 5.35. 

After equivalent units have been calculated, the analyst must determine if the cost improvement achieved 

during development on these prototype units applies to the production phase. The following factors 

should be considered when analyzing the amount of credit to take in production for cost improvement 

incurred in development:

■ the break between the last prototype unit and the start of production units,

■ how similar the prototype units are to the production units,

■ the production rate, and

■ the extent to which the same facilities, processes, and people are being used in production as in 

development.
GAO-09-3SP APPENDIX XI 371

By addressing these factors, the analyst can determine proper placement on the curve for the first 

production unit. For example, analysis might indicate that cost improvement is continuous and, therefore, 

the first production unit is really the number of equivalent development units plus one. If it is further 

determined that the development slope should be the same as the production slope, the production 

estimate can be calculated by continuing down the curve for the desired quantity. This is referred to as the 

continuous approach.

Analysis of the four factors often leads the analyst to conclude that totally continuous improvement is 

not appropriate and that some adjustment is required. This could be because prototype manufacture 

was accomplished in a development laboratory rather than in a normal production environment or that 

engineering personnel were used rather than production personnel. Numerous reasons are possible for less 

than totally continuous cost improvement. Since all programs are unique, the analyst must thoroughly 

evaluate their particularities.

Two Theories Associated with Less Than Continuous Improvement

Two theories, sequential and disjoint, address the issue of less than continuous improvement. Both theories 

maintain that the improvement slope is the same in production and development but that a step down in 

value occurs between the cost of the first prototype unit and the cost of the first production unit. 

In sequential theory, cost improvement continues where the first production unit equals the last 

production unit plus one, but a displacement on the curve appears at that point. In disjoint theory, the 

curve is displaced, but improvement starts over at unit one rather than at the last production unit plus one. 

These displacements are typically quantified as factors. Because disjoint theory restarts learning, it usually 

results in significantly lower production estimates.

The continuous cost improvement concept and sequential and disjoint displacement theories assume the 

same improvement slope in production as in development. Plots of actual cost data, however, sometimes 

indicate that production slopes are either steeper or flatter than development slopes. In cases in which the 

historic data strongly support a change in slope, the analyst should consider both a step down and a shift. 

For example, changing from an engineering environment to a heavily automated production line might 

both displace the improvement curve downward and flatten it.

End-of-Production Adjustments

As production ends, programs typically incur greater costs for both recurring and nonrecurring efforts. 

The recurring cost of end-of-production units is often higher than would have been projected from a 

program’s historic cost improvement curve. This is referred to as toe-up. The main reasons for toe-up are 

■ the transfer of more experienced and productive employees to other programs, resulting in a loss of 

learning on the production line;

■ reduced size of the final lot, resulting in rate adjustment penalties;

■ a decrease in worker productivity from the psychological effect of the imminent shutdown of the 

production line;
372 APPENDIX XI GAO-09-3SP

■ a shift of management attention to more important or financially viable programs, resulting in 

delayed identification and resolution of production problems; 

■ tooling inefficiency, resulting from tear-down of the tooling facility while the last production lot is 

still in process;

■ production process modifications, resulting from management’s attempts to accommodate such 

factors as reductions in personnel and production floor space; and

■ similar problems with subcontractors.

No techniques for projecting recurring toe-up costs are generally accepted. In truth, such costs are often 

ignored. If, however, the analyst has access to relevant historic cost data, especially contractor-specific data, 

it is recommended that a factor be developed and applied. 

Typically far more extensive than recurring toe-up costs are the nonrecurring close-out costs that account 

for the numerous nonrecurring activities at the end of a program. Examples of close-out costs are 

■ the completion of all design or “as built” drawings and files to match the actual “as built” system; 

often during a production run, change orders that modify a system need to be reflected in the final 

data package that is produced;

■ the completion of all testing instructions to match “as built” production; and

■ dismantling the production tooling or facility at the end of the production run and, sometimes, the 

storage of that production tooling.
GAO-09-3SP APPENDIX XII 373

Technology Readiness Levels

Readiness level Definition

1. Basic principles observed and 

reported

Lowest level of technology readiness. Translation of scientific research 

into applied research and development begins—examples might 

include paper studies of a technology’s basic properties

2. Technology concept or 

application formulated

Invention begins, application is speculative, and no proof or detailed 

analysis supports assumptions. Examples are limited to paper studies 

3. Analytical and experimental 

critical function or characteristic 

proof of concept

Active research and development begins, including analytical and 

laboratory studies to physically validate analytical predictions of 

technology’s separate elements. Examples include components not yet 

integrated or representative

4. Component or breadboard 

validation in a laboratory 

Basic technological components are integrated to establish that the 

pieces will work together. This is relatively low fidelity compared to 

the eventual system. Example is integration of ad hoc hardware in a 

laboratory 

5. Component or breadboard 

validation in relevant environment

Fidelity of breadboard technology increases significantly. The basic 

technological components are integrated with reasonably realistic 

supporting elements so that the technology can be tested in a 

simulated environment. Example is high-fidelity laboratory integration 

of components 

6. System or subsystem model 

or prototype demonstration in a 

relevant environment

Representative model or prototype system, well beyond level 5, is 

tested in a relevant environment, representing a major step up in a 

technology’s demonstrated readiness. Examples include testing a 

prototype in a high-fidelity laboratory environment or in simulated 

operational environment 

7. System prototype demonstration 

in an operational environment

Prototype near or at planned operational system, representing a major 

step up from level 6, requiring the demonstration of an actual system 

prototype in an operational environment, such as in an aircraft, in 

a vehicle, or in space. Example is testing the prototype in a test bed 

aircraft

8. System completed and flight 

qualified through test and 

demonstration

Technology has been proven to work in its final form and under 

expected conditions; in almost all cases, represents the end of true 

system development. Example is developmental test and evaluation 

of the system in its intended weapon system to determine if it meets 

design specifications 

9. System flight proven through 

successful mission operations

Actual application of the technology in its final form and under 

mission conditions, such as those encountered in operational test and 

evaluation. In almost all cases, this is the end of the last bug fixing 

aspects of true system development. Example is using the system under 

operational mission conditions

Source: GAO.
374 APPENDIX XIII GAO-09-3SP

EVM-Related Award Fee Criteria

Criterion Rating Rationale

EVM is integrated 

and used 

for program 

management 

Unsatisfactory Contractor fails to meet criteria for satisfactory performancea

Satisfactory Contractor team uses earned value performance data to make program 

decisions, as appropriate

Good Meets all satisfactory criteria, and earned value performance is 

effectively integrated into program management reviews and is a 

primary tool for program control and decisionmaking

Very good Meets all good criteria and the contractor team develops and sustains 

continual and effective communication of performance status with the 

government

Excellent Meets all very good criteria, and the entire contractor team proactively 

and innovatively uses EVM and plans and implements continual EVM 

process improvement

Contractor 

manages major 

subcontractors

Unsatisfactory Fails to meet criteria for satisfactory performance

Satisfactory Routinely reviews the subcontractors’ performance measurement 

baseline

Good Meets all satisfactory criteria and the management system is structured 

for oversight of subcontractor performance

Very good Meets all good criteria and actively reviews and manages subcontractor 

progress so that it provides clear and accurate status reporting to 

government

Excellent Meets all very good criteria, the effective and timely communication of 

subcontractor cost and schedule status are reported to government, 

and issues are proactively managed

Cost, expenditure, 

and schedule 

forecasts are 

realistic and current

Unsatisfactory Contractor fails to meet criteria for satisfactory performance.

Satisfactory Contractor provides procedures for delivering realistic and up-to-date 

cost and schedule forecasts as presented in the CPR, EACs, contract 

funds status report, IMS, etc. Forecasts are complete and consistent 

with program requirements and reasonably documented

Good Meets all satisfactory criteria, and all requirements for additional 

funding and schedule changes are thoroughly documented and 

justified. Expenditure forecasts are consistent, logical, and based 

on program requirements. The contractor acknowledges any cost 

growth in the current reporting period and provides well-documented 

forecasts

Very good Meets all good criteria, and expenditure forecasts reflect constant 

scrutiny to ensure accuracy and currency. The contractor prepares 

and develops program cost and schedule data that allow government 

a clear view into current and forecast program costs and schedule. 

Schedule milestone tracking and projections are very accurate and 

reflect true program status. The contractor keeps close and timely 

communications with the government

Excellent Meets all very good criteria, and the contractor consistently submits a 

realistic, high-quality EAC; reported expenditure profiles are accurate. 

Contractor develops comprehensive and clear schedule data with 

excellent correlation to technical performance measures and CPRs that 

permit early identification of problem areas. Schedule milestone tracking 

and projections are accurate and recognize potential program effect
GAO-09-3SP APPENDIX XIII 375

Criterion Rating Rationale

Contractor’s 

cost proposals 

are adequate 

during award fee 

evaluation period

Unsatisfactory Fails to meet criteria for satisfactory performance

Satisfactory Proposal data, including subcontractor data, are logically organized 

and give government a view adequate to support cost analysis and 

technical review. A basis of estimate is documented for each element, 

and when insufficiently detailed. the contractor provides it to the 

government on request. The proposal is submitted on time

Good Meets all satisfactory criteria and provides detailed analysis for 

subcontractor and material costs

Very good Meets all good criteria. Proposal data are traceable and give the 

government a view for supporting a detailed technical review 

and thorough cost analysis; only minor clarification is required by 

government. Potential cost savings are considered in the proposal

Excellent Meets all very good criteria; change proposals stand alone and require 

no iteration for government understanding. The contractor stays 

in communication during proposal preparation and resolves issues 

effectively before submission

Costs are controlled Unsatisfactory Contractor fails to meet criteria for satisfactory performance

Satisfactory Contractor and subcontractor control cost to meet program objectives

Good Meets all satisfactory criteria; contractor stays within target cost and 

provides good control during contract performance

Very good Meets all good criteria, and the contractor stays within cost and 

continues to provide good control during contract performance

Excellent Meets all very good requirements; contractor provides suggestions 

and, when appropriate, proposals to the program office for initiatives 

that can reduce costs. The contractor implements cost reduction 

ideas across the program and at the subcontract level and identifies 

(and when appropriate implements) new technologies, commercial 

components, and manufacturing processes that can reduce costs

Contractor 

conducts variance 

analysis

Unsatisfactory Fails to meet criteria for satisfactory performance

Satisfactory Variance analysis is sufficient and usually keeps the government 

informed of problem areas and their causes and corrective action. 

When detail is insufficient, the contractor provides it to the government 

promptly on request

Good Meets all satisfactory criteria and routinely keeps government 

informed of problem areas and their causes and corrective action. 

Updates explanations monthly and analyzes potential risks for cost and 

schedule impacts

Very good Meets all good criteria and always keeps government informed of 

problem areas and their causes and corrective action. Variance analysis 

is thorough and used for internal management to control cost and 

schedule. Detailed explanations and insight are provided for schedule 

slips or technical performance that could result in cost growth. The 

government rarely requires further clarification

Excellent Meets all very good criteria; variance analysis is extremely thorough. 

Contractor proactively keeps the government informed of all problem 

areas and their causes, emerging variances, impacts, and corrective 

actions. Keeps government informed of progress implementing 

the corrective action plans and fully integrates analysis with risk 

management plans and processes
376 APPENDIX XIII GAO-09-3SP

Criterion Rating Rationale

Billing and 

cumulative 

performance data 

are accurate, timely, 

and consistent and 

subcontractor data 

are integrated

Unsatisfactory Contractor fails to meet criteria for satisfactory performance

Satisfactory Billings to the government may have slight delays or minor errors and 

the CPR, contract funds status report, and IMS reports are complete 

and consistent, with only minor errors. Data can be traced to the WBS 

with minimum effort, and subcontractor cost and schedule data are 

integrated into the appropriate reports with some clarification required. 

Reports may be submitted late, but electronic data are correct

Good Meets all satisfactory criteria, and billing to government is accurate, 

although with slight delays. Data are complete, accurate, and 

consistent and can be traced to the WBS, with some clarification 

required. Subcontractor performance data are fully integrated into the 

appropriate on-time reports, with no clarification required

Very good Meets all good criteria, and data are complete, accurate, and consistent

Excellent Meets all very good criteria, and billing is submitted to government on 

time. Data are complete, accurate, and consistent and can be traced 

clearly to the WBS. CPR and contract funds status report data elements 

are fully reconcilable. Subcontractor schedule performance is vertically 

and horizontally integrated with the contractor schedule

Baseline is 

disciplined and 

system is in 

compliance

Unsatisfactory Contractor fails to meet criteria for satisfactory performance

Satisfactory Contractor develops a reliable performance measurement baseline that 

includes work scope, schedule, and cost. The contractor or government 

may discover system deficiencies or baseline planning errors through 

either routine surveillance or data inaccuracies in the CPRs. Contract 

changes and undistributed budget are normally incorporated into the 

baseline in a timely manner. Management reserve is properly tracked, 

and eliminating performance variances is limited to correcting errors

Good Meets all satisfactory criteria. Contractor develops a reliable 

performance measurement baseline that includes work scope, 

schedule, and cost. The contractor or government may discover 

system deficiencies or baseline planning errors through either routine 

surveillance or data inaccuracies in the CPRs. Contract changes and 

undistributed budget are normally incorporated into the baseline in a 

timely manner. Management reserve is tracked and used properly, and 

elimination of performance variances is limited to correction of errors 

Very good Meets all good criteria and the contractor builds a proper and realistic 

baseline in a timely way. The contractor ensures that work packages are 

detailed and consistent with scope of contract and planned consistent 

with schedule. The contractor conducts routine surveillance that 

reveals minor system deficiencies or minor baseline planning errors 

that are quickly assessed and corrected, resulting in little or no impact 

to data accuracy. Contractor’s EVM system is effectively integrated

Excellent Meets all very good criteria and the contractor proactively manages 

the baseline and maintains timely detailed planning as far in advance 

as practical and implements proper baseline controls. The contractor 

controls and minimizes changes to the baseline, particularly in the 

near term, and system deficiencies or planning errors are few and 

infrequent. The contractor streamlines internal processes and maintains 

a high level of EVM system competency and training

Source: GAO and DCMA.

a

Program managers need to determine what satisfactory performance criteria will be used since each program is unique.
GAO-09-3SP APPENDIX XIV 377

Integrated Baseline Review Case Study and Other

Supplemental Tools

As described in the Cost Guide, the objectives of the integrated baseline review (IBR) are to gain insight 

into cost and schedule risk areas associated with the subject program (or contract) and to develop 

confidence in the program’s operating plans. The focus of this review should be primarily to assessing the 

adequacy of the baseline plan to execute the approved program (or contract). In chapter 19, we discuss 

the key practices for planning and executing an effective IBR. In this appendix, we provide supplemental 

information on the IBR to help organizations in implementing or improving their IBR capabilities, as 

well as provide our auditors with further guidance on the planned effort to perform a quality IBR. This 

information is based on the process the Naval Air Systems Command (NAVAIR) uses, an organization 

considered a leader in IBR process and in maximizing the value gained from these reviews.15

NAVAIR IBR Preparation

IBR Team Roles and Responsibilities

The typical IBR team is made up of the government program manager, technical experts, an EVM 

analyst, and DCMA, as well as other personnel who may help during the review. The duties of all team 

members include attending IBR training before the start of the IBR, reviewing contract documentation 

before baseline discussions with the control account manager (CAM), conducting CAM and senior 

manager discussions, helping to complete applicable documentation, providing a risk assessment based 

on the prescribed risk evaluation criteria, and helping to prepare the IBR out-brief. Table 72 describes the 

specific responsibilities of the key team leaders.

Table 72: IBR Leadership Roles and Responsibilities

Key leader Role Responsibility

Program 

manager 

Acts as or assigns 

the team leader 

for the IBR; jointly 

responsible for the 

IBR process

Plan and perform the IBR

Monitor progress on required actions until issues are resolved

Provide an adequate number of qualified personnel as IBR team members

Specify evaluation criteria for risk areas

Document risk issues identified during an IBR

Present the IBR out-brief

Performance 

measurement 

deputy team 

leader

This lead role is 

filled by the NAVAIR 

[AIR-4.2.3] EVM 

analyst

Provide overall facilitation for the IBR

Provide IBR training before the start of the IBR

Review contract documentation before baseline discussions with the 

control account manager

Conduct control account manager and senior manager discussions

Provide policy and interpretation of EVM system guidance

Provide technical direction and leadership emphasizing the importance 

of thorough cost, schedule, and technical integration of contract work

Ensure that all action item reports are tracked in the EVM risk database

Provide an assessment of risk based on the prescribed risk evaluation 

criteria and all program risk based on the defined risk evaluation criteria 

Help complete all IBR documentation

Help prepare the IBR out-brief

Source: NAVAIR.

15The IBR approach is scalable—e.g., the NAVAIR case study is of a major system acquisition, but IBRs on small programs that do 

not involve a major acquisition may not require the time and number of people cited in this case study to achieve IBR objectives. 
378 APPENDIX XIV GAO-09-3SP

Based on experience, NAVAIR officials have told us that the data review with the IBR team before the 

event is really the key to a successful IBR. Being prepared goes a long way to being able to dig deep in the 

data to determine whether there are issues and risks in the plan. 

IBR Team Training

In the weeks leading up to the IBR event, the IBR team typically participates in a day of training tailored 

to the subject program that includes the

■ basic IBR fundamentals and review of the methodology to be followed on subject program; 

■ detailed roles and responsibilities of team members; 

■ guidance on baseline discussions with control account managers and the key documents that 

should be referenced (and sample data traces across these documents) to see how work is defined, 

baselined, measured, and scheduled;

■ results from recent schedule risk assessments, management system assessments, and major 

subcontractor IBRs (elements of the IBR NAVAIR performed before the IBR event to better 

understand the current risks in the baseline and focus on the program areas that align with these 

risks during the IBR); 

■ IBR out-brief contents; and

■ evaluation criteria, tools, and forms expected to be used during execution (see exhibits B–D at the 

end of this appendix for select NAVAIR discussion forms and sample questions). 

IBR Execution

The duration of the IBR is based on program and contract scope, complexity, and risk; typically, it lasts 

over several days. Exhibit A at the end of this appendix, for example, is the agenda of activities for an 

actual 4-day IBR for Program X; it had just implemented an overtarget baseline on its prime contract. 

(Specific references to the actual program and contractor have been removed.) 

This IBR was kicked off with the contractor’s overview briefing on its internal management process—risk 

management, baseline establishment and maintenance, scheduling, EVM methods and tools, work 

authorization, and standing management meetings with control account managers and subcontractors. 

The process overview briefing was followed by the team’s discussion with the contractor program 

manager. At the conclusion of this discussion, team members wrote up their observations and findings 

in their individual assessments. Once the individual assessments were completed (see exhibit B), the team 

came together to complete a consensus assessment (exhibit B). It is during these consensus meetings 

that action item reports are typically assigned to individual members for drafting, where applicable (see 

exhibit C). This same methodology is applied to the control account manager discussions, as well. 

Formal Out-brief of IBR Results

Figure 61 shows the team’s summary assessment of the risks in Program X, based on the amount of 

remaining work, level of severity of the risks (many of which affected tasks found on the integrated master 

schedule’s critical path), and the government risk evaluation criteria that was applied. The most critical 
GAO-09-3SP APPENDIX XIV 379

risks identified were related to the prime contractor’s management of its major subcontractors. During 

the out-brief presentation, the government program manager noted concerns about the prime contractor’s 

current practices in overseeing selected subcontractors because of ongoing poor quality and late receipt 

of key deliverables—some of which affected the program’s critical path. The other critical risks were 

associated with the specific earned value metrics applied to measure progress in software development. 

Figure 61: IBR Team’s Program Summary Assessment Results for Program X

Program level rollup of risk

1. Based on remaining work

2. Level of severity (risk level and schedule critical path taken into account)

3. Also considered government risk evaluation criteria

Subcontractor

management processes 

and lack of valuable 

software earned value 

performance measures

Have key vacancies in 

staffing at this time

Didn’t staff to plan in

software area

Low confidence in

personnel turnover plans 

and risk of losing 

expertise 

Cost impact of 

subcontractor (prime 

using management 

reserve to mitigate poor 

performing contractor);

software continues to be a 

risk, but expect to be less 

than 10% of program at 

this time

Activities pushing out from 

original baselines dates, 

however, appear to be 

overcoming difficulties 

with critical path driver 

subcontractor

Will continue to stress 

importance of near-term 

schedule

Hardware and software 

Integration plan was not in 

place

Few identified

opportunities (budget and 

schedule) available to 

mitigate potential risk 

areas 

High Medium Medium Medium Medium

Management

processes Resources Cost Schedule Technical

Source: NAVAIR.

Figure 62 shows a summary of the detailed assessment results at the major system development areas 

under evaluation during the IBR.

Figure 62: Program X IBR Team’s Assessment Results by Program Area

Management

processes

Critical path

June 2007

BCWS

remaining

CAM/Area June 2007

Subcontractor

Subcontractor

Prime contract/Control account manager

Prime contract/Control account manager

Prime contract/Control account manager

Subcontractor/software

Subcontractor

Prime contractor/logistics

Prime contractor/PM

$49K

$14,626K

$7,299K

$1,344K

$3,629K

$2,658K

$2,906K

SIL TRR

SDD

SIL

&

SDD

NO

NO

NO

SIL TRR & SDD

High

High

Low

Low

Low

Medium

High

Low

Low

High

Low

Medium

Medium

High

Medium

Low

Low

Medium

High

Medium

Low

Low

Medium

High

Low

Low

Low

High

High

Medium

Low

Medium

High

Low

Low

Low

High

Low

Low

Low

Medium

High

Low

Low

Low

Resources Cost Schedule Technical

Source: NAVAIR.

Figure 63 is an example from the detailed findings of a particular program area. This final assessment 

represents the team’s consensus of overall risk ratings by IBR risk category, based on the agreed-on 

observations and findings from the control account manager discussions. Each corrective action 

corresponds to a specific action item report. 
380 APPENDIX XIV GAO-09-3SP

Figure 63: Program X IBR Team’s Detailed Assessment Results for an Individual Program Area

Score H L

Management process

Assessment

 area

Resources

Cost

Schedule

Technical

M H L

Overall assessment

Lack of integration of the subcontractor schedule with the program IMS is a big concern

Communication processes between subcontractor and prime contractor must improve

Timeliness of EV and technical information transfer is inadequate 

Dialogue highlights

Strengths

Risks and issues

Work allocated to subcontractor is well defined and understood

Few identified opportunities remain to mitigate potential technical

or cost risk, no management reserve

Special management attention/monitoring of subcontractor critical path

items. Touch points between prime contractor and subcontractor/gov

need to be coordinated to reduce any potential schedule risk

Unable to assess total cost risk due to lack of subcontractor schedule

with the program IMS

Management process issues:

 Subcontractor PM management reserve decisions 

 Prime subcontractor mgmt processes have yet to be validated 

 Timing of subcontractor IMS submissions 

 Lack of a management process to use management reserve

 Lack of cost and schedule integration 

Risks and issues continued

Communications between team members and the data sharing across

the contract and subcontract especially during A/C installation and test

Corrective actions

Develop and integrate an IMS with subcontractor (CAR#4)

Although resources are assessed as low risk, identification of

subcontractor scheduler is a high priority

Prime contractor formally documents emerging processes (EAC,

schedule inputs, program management, finalize restructure, and prime

contractor evaluation of EVM data) 15 Oct goal to revisit (CAR#6)

Request a subcontractor org chart with contact information (data

request #5)

Continue A/C induction discussion...to be continued as an action out

of IBR (CAR#8)

Concern that prime contractor is not using subcontractor EVM data to

manage subcontractor effort, recommend more detailed analysis of

subcontractor CPR prior to submitting to gov (CAR#6)

Source: NAVAIR.

Post IBR Activities

After IBR event activities, the IBR team is responsible for developing the final action item reports, which 

are then formally submitted to the contractor, who is given about a month to respond back to the team. 

The team reviews the contractor’s responses; sometimes these require further negotiation on the closure 

of the action item reports. NAVAIR determines whether the contractor has responded sufficiently and 

the original risk has been addressed. NAVAIR closes the action item, requests further information or 

clarification, or decides to introduce the risk into the risk management plan. A decision to include the 

program’s risk in the NAVAIR risk database is based on several factors, including the contractor’s inability 

to address the item fully, the lack of a clear action to take at the time, or the realization that it is a risk that 

the program has to manage. In some cases, IBR reports can remain open for a significant amount of time. 

From their experience, NAVAIR officials have told us that they often find that the most difficult parts of 

the IBR are coming to closure on outstanding action item reports and keeping the program team focused 

on the issues long after actual events have occurred. Overcoming the perceptions that the IBR is just an 

event is definitely a challenge. 

In consultation with our experts, they noted that the critical factor in closing IBR actions is ensuring that 

these items receive ongoing attention from the program manager (both government and contractor) based 

on lessons learned. The most effective way to do this is to incorporate these action items into the business 

rhythm (usually monthly, including the monthly program management review). Any and all IBR action 

items captured in the out-brief and supporting documentation should go directly into the contractor’s 
GAO-09-3SP APPENDIX XIV 381

internal action item database for disposition and closure with the appropriate government approvals. The 

monthly program management review should be used to track the status of the IBR actions. Our experts 

noted that having a separate list of things to do outside the contractor’s business rhythm (and some of the 

issues are hard to tackle) simply will not get done. Also, waiting long enough could mean that the action 

items could even be overtaken by events. 

In summary, the experts highlighted several key points to this lesson learned:

■ use the contractor’s action item tracking database;

■ load the actions properly into it, assign appropriate responsibilities, receive status updates monthly, 

and obtain appropriate government reviews and approval to close them; and

■ hold the contractor accountable with the award fee process, as applicable.
382 APPENDIX XIV GAO-09-3SP

Exhibit A – 

Program X: Subject Contractor

Integrated Baseline Review

August 20–23, 2007

Monday, August 20, 2007

Conference Room A

1400–1430 Opening Comments

1430–1600 Program X Overview Management Systems

1600–1700 Contractor Program Management, Program X, Program Management Discussion

1700–1800 Contractor Program Management Process Write-Up

Tuesday, August 21, 2007

Conference Room B

0800–0830 Government IBR Team Only

0830–1000 Test Team, Control Account Managers 1 and 2

1000–1030 Test Team Write-Up

1030–1045 Break

1045–1200 Major Subcontractor A, Control Account Managers 3 and 4

1200–1300 Lunch

1300–1330 Major Subcontractor A Write-Up

1330–1500 Logistics and Technical Data, Control Account Manager 5

1500–1530 Logistics and Technical Publication Write-Up

1600–1630 Informal Out-brief (to Contractor)

1630–1730 Complete Formal Out-brief for Days 1 and 2, Control Account Managers

Wednesday, August 22, 2007

Conference Room B

0800–0830 Government IBR Team Only

0830–1030 Systems Engineering and SIL Support, Control Account Managers 6 and 7 

1030–1130 Systems Engineering and SIL Support Write-Up 

1130–1230 Lunch

1230–1400 Major Subcontractor B, Control Account Managers 3 and 8

1400–1430 Major Subcontractor B Write-Up

1430–1500 Break

1500–1630 Major Subcontractor C, Control Account Managers 3 and 8

1630–1700 Major Subcontractor C Write-Up

1700–1730 Informal Out-brief (to Contractor)

Thursday, August 23, 2007 

Conference Room B

0800–1130 Government IBR Team: Documentation Wrap-Up & Preparation of Formal Out-brief

1130–1230 Lunch

1230–1330 IBR Schedule Reserve 

1400 Deliver Formal Out-brief to Contractor
GAO-09-3SP APPENDIX XIV 383

Exhibit B – 

IBR Discussion Assessment Forms

In using assessment forms to frame discussions with control account managers, evaluators should keep in 

mind three fundamental objectives: (1) to achieve the technical plan, (2) to complete the schedule, and (3) 

to ensure the sufficiency and adequacy of resources and their time-phasing. These objectives make up the 

core of the IBR. 

Individuals with experience conducting IBRs should be present for each discussion. Without complete 

understanding of the baseline plan, the results of an IBR are negligible. Our experts discouraged 

overreliance on checklists or questionnaires, since these may result in “failure to see the forest for the trees.” 

With these three objectives in mind, the forms presented here can be useful for focusing discussions.

IBR Discussion Assessment Form from NAVAIR

Log No. ______________ Team ________ Date_______________

1. Manager __________________ Area of responsibility __________________

2. Technical scope (statement of work)

______ Complete identification, definition, and flow down

______ Consistency with contract requirements

______ Assignment of responsibility, authority, and accountability

3. Schedules 

______ Period of performance

______ Realistic planned durations

______ Logical sequence of work planned

______ Consistency with intermediate/master schedule and contract milestones 

______ Significant interdependencies, interfaces, and constraints

4. Cost and resource risk

______ Basis of estimate16

______ Budget adequacy and reasonableness (time phasing, levels, mix, type)

______ Resource availability

______ Provisions for scrap, rework, retest, or repair

5. Management process risk

 ______ Integrated cost, schedule, and technical planning

 ______ Status of EVM system acceptance. If not accepted, EVM specialists should assess the adequacy 

of key EVM concepts:

 ______ Baseline change control

 ______ Reliability and timeliness of management and performance data

 ______ EAC determination and maintenance process

 ______ Subcontract management

 ______ Objectively planned earned value methods correlated with technical progress

 ______ Objective determination of progress

 ______ Methods correlate with technical achievement

6. Brief summary of discussion

7. Action item report prepared

16 “Sound basis of estimate” should be understood in the context of the estimate (or contract target value) that resulted after the 

proposal and negotiation process was completed—the contract value is the new “basis of estimate.”
384 APPENDIX XIV GAO-09-3SP

An Alternative Discussion Form

Date: ___________________ Time: ___________________

Program: ______________________________________________________________

Control account Managers: ______________________________________________________________

WBS (or CLIN): ______________________________________________________________

Attendees: ______________________________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________

Documents reviewed:

Statement of work

Organizational breakdown structure

Work authorization document

Integrated master schedule

Entrance and exit criteria

Resource planning

WBS and WBS dictionary

Responsibility assignment matrix

Control account plans

Critical path analyses

Assumptions

Other ______________________

Brief summary of subjects discussed:

Identification of risks:

Were any action item forms prepared? Yes No

If yes, brief description of actions:

Brief statement of strengths, weaknesses, conclusions:

5 = Achievable — risks adequately identified

4 = Probably achievable — risk mitigation effort required in one or more minor areas 

3 = Potentially achievable — additional risk mitigation effort required in a number of areas 

2 = Risk mitigation borderline — significant risk mitigation effort required; may not be achievable 

1 = Not achievable — not achievable as currently planned; risks significantly affect achieving 

objectives

Plan’s achievability: 1 2 3 4 5

•	 Technical: 1 2 3 4 5

•	Schedule: 1 2 3 4 5

•	Cost: 1 2 3 4 5

Planned follow-up:

Signatures:

Government discussion lead Contractor discussion participant
GAO-09-3SP APPENDIX XIV 385

Exhibit C – 

IBR Action Item Forms

NAVAIR IBR Action Item Report

WBS / Control account:________________ Log no.: _____________ Date: ____________________

Submitted by:________________________

Subject of issue or observation:

Discussion of root problem and cause. (Provide impact assessment. Quantify problem and impacts 

where possible. Provide recommended actions and exit criteria for resolution. Attach exhibits if 

applicable. Provide reference to control account or work package number.)

Contractor’s response. (Address root cause of the problem, impact, corrective and preventive action 

plan; identify dates and POC. Identify exit criteria for corrective action.)

Subteam leader signature: __________________

Team leader signature: ______________________
386 APPENDIX XIV GAO-09-3SP

An Alternative Action Item Form

Date: ___________________ Time: ___________________

Program: ______________________________________________________________

Control account Manager: ______________________________________________________________

WBS (or CLIN): ______________________________________________________________

Issue: 

Actions required:

Criteria for success:

Estimated completion date: ______________________________________________________________

Point of contact: ______________________________________________________________

Signatures:

Government program manager Control account manager or functional lead
GAO-09-3SP APPENDIX XIV 387

Exhibit D – 

Sample IBR Discussion Questions

The following questions were used in the NAVAIR IBR training. They are intended only as a reference 

guide. NAVAIR expects its IBR teams to select and tailor questions to a program’s condition (that is, new 

program versus overtarget baseline), issues, and risks. 

Organization

To introduce the IBR team, identify (graphically if possible) the location of the integrated process team 

(IPT) in the program (that is, its organizational breakdown structure) relative to other IPTs. Similarly, 

identify the control and schedule accounts assigned to the IPT and which ones the IPT will discuss in 

answering the remaining questions. Include your areas of responsibility in the program, whom you report 

to, your responsibilities toward this person, and how you keep this person informed of status and progress. 

What is the manager’s scope of effort?

The manager should be able to refer to a statement of work paragraph, a contract WBS narrative, 

or a work authorization document. 

■ How many people work for you and what do they do? 

■ How do they report to you (how do you know the performance status of their work)? 

Is all the work planned into control accounts?

The statement of work defines the effort. The contract WBS provides specifics, such as work 

definition. The work authorization and change documentation should show information such as 

dollars and hours, period of performance, and the scope of work and any changes.

Are all elements of the scope planned?

The manager should be able to show the scope of work broken down into work packages and 

the budgets and estimates to complete (ETC) associated with each work package and planning 

package. The sum of the work packages and planning packages should equal the control account 

budget. The actual costs plus the estimates to complete should equal the estimate at completion. 

What are the manager’s resources for assigned work?

Baseline resources should be identified in the work authorization document, and changes in scope, 

cost, or schedule requirements should be reflected in change request documentation. 

Are the resources required to accomplish the current plan consistent with the original plan?

Review the basis of estimate for reasonableness. Does the manager believe that the budget (or ETC 

if different from BAC) is sufficient to perform the work? 

Elicit a range of possibilities (low and high) that represents as clearly as possible the complete 

judgment of the control account manager, as follows: 

■ The adequacy of the planned and approved baseline to achieve the approved scope.
388 APPENDIX XIV GAO-09-3SP

■ Risks and opportunities included or not included in the baseline. What are the major risks 

or challenges remaining to accomplish the control account manager’s or subcontractor’s 

responsibilities? 

ù Ask the control account manager to describe why it is a risk or opportunity.

ù Exchange ideas about risks and opportunities.

ù Establish the likelihood of the risk or opportunity event. 

■ Ask the control account manager to explain the risk mitigation plan emphasizing risk mitigation 

milestones and associated risk performance measurement. 

■ Determine the impact (cost and schedule) for medium and high risks. 

■ Ask the control account manager to consider extreme values for his effort (optimistic or 

pessimistic).

■ Document results on the risk assessment form.

Authorization

What is the status of work authorization? 

Give an example of work authorization documentation. 

Ask the control account manager to show his work authorization documents, which define the 

work to be accomplished. Ask the control account manager to relate these requirements to the work 

remaining within his team or WBS element when the cost to complete was analyzed or developed. 

Budget

Discuss how the control account manager’s budget was derived. 

How did you arrive at your budget figures? Do you have the backup or worksheets you derived 

your estimates from? 

Was there a negotiation process for your budgets after contract award? Is your budget adequate? 

How were you advised of budget? Of tasks? Of schedule? Of changes? 

Control Account

How many control accounts are you responsible for and what is their total dollar value? May we see 

a control account plan? 

How are your budgets time-phased, and is this reflected in your control account plan? 

How do you status your accounts? How does the performance status of your accounts get into the 

system? 

Do you have any LOE accounts? Please describe their tasks. 

Do you have any control accounts that contain a mixture of LOE and discrete effort? What is the 

highest percentage of LOE within an account that also contains discrete effort? 

How do you open and close a control account? 

What does your computer run show when a control account is opened or closed? 
GAO-09-3SP APPENDIX XIV 389

What reports do you receive that give you cost and schedule progress of your control accounts? 

Work Package

Assess whether work is measured objectively and whether LOE is appropriate for the nature of the 

work. 

How do your work package activities relate to the master program schedule or underlying 

intermediate supporting schedules? Support your answer with examples. 

How was the budget time-phased for each work package—i.e., what was the basis for the spread? Is 

the time-phased budget related to planned activities of the work package? 

For the example control account, what is your total (IPT) budget amount? Of this total budget 

amount, how much is distributed to work packages and how much is retained in planning 

packages? Do you have an undistributed budget and management reserve account? 

Do you use interim milestones on any of your work packages to measure BCWP? 

How do you define a work package? How many work packages do you have responsibility for? 

What options does your EVM system provide for taking BCWP? 

Do your control account plans indicate the method used in taking BCWP? 

How do you open and close work packages? 

Who prepares the budgets for your work packages? 

Demonstrate how you earn BCWP in the same way that BCWS was planned. 

Can you provide examples of how you measure BCWP or earned value for work-in-process? 

Planning Package

What is the procedure and time period for discretely developing work packages from the planning 

packages? 

Are your planning packages time-phased? 

Schedule

What are your schedule responsibilities? 

What schedule milestones did the manager use in planning the cost accounts? Ask the manager to 

show the team the schedule milestones used in planning the cost accounts. How does the current 

schedule compare with the baseline schedule?

The manager should discuss

■ relationships of work packages to milestones,

■ schedule interfaces and constraints,

■ staffing levels to support schedule milestones,

■ relationships to other organizations or IPTs,
390 APPENDIX XIV GAO-09-3SP

■ schedule impacts related to other work or organizations, and

■ level-of-effort tasks that support the schedule.

How did the manager time-phase the work to achieve the schedule? All work should be logically 

planned in compliance with the SOW and schedule.

Has the manager considered risks in developing the plan? 

Has the manager adequately planned and time-phased resources to meet the plan?

Do you directly support any major master or intermediate schedule milestones? 

Do you have detailed schedules below the work package? How do detailed schedules below the 

work package support the work package schedules? 

How are you informed by other organizations or IPTs of changes in their output that may affect 

your control accounts schedules (horizontal trace)? 

Demonstrate that the progress reflected on the master program schedule or underlying 

intermediate schedules correlates to the relative progress reflected in the EVM system. 

Change Control

Has the budget baseline had changes or replanning efforts? 

Have you had any changes to your accounts? (Give example of how these are handled.) 

Have you had any management reserve or undistributed budget activity? 

Do you have any work originally planned for in-house that was off-loaded? How was this 

accomplished? 

Earned Value

What methods and tools does the manager use in administering the plan?

Examples are weekly or monthly earned value reports; master, intermediate, and detail schedules; 

periodic meetings; and independent assessments of technical progress. Determine how changes 

are incorporated. Evaluate the effect of changes on performance measurement information. Assess 

whether changes accord with the EVM system description.

What formal training have you had in EVM? 

Estimate at Completion and Cost-to-Complete Subcontractor

Are you responsible for any subcontracts? How do you monitor their performance? How do you take 

BCWP? 

How are subcontracts managed? Ask the subcontracts manager to describe the process for 

managing subcontractor earned value.

■ What subcontracts are your responsibility? What types of subcontracts exist or are planned for 

negotiation (e.g., fixed price vs. cost reimbursement)? 
GAO-09-3SP APPENDIX XIV 391

■ What are the major challenges or risks to the subcontractor in accomplishing program 

responsibilities?

■ Are these items tracked by the program management office or functional manager in a risk 

register or plan?

■ What subcontractor technical, schedule, and cost reports must be submitted to you or your 

team?

■ What is your total budget (for each subcontract and the corresponding control accounts)? How 

is profit or fee included in your budget?

■ How was the budget established? Does it reflect an achievable value for the resources to fully 

accomplish the control account scope of effort?

■ What rationale was used to time phase the budget resources into monthly or weekly planning 

packages, tasks, work packages, or summary activities?

■ Are the time-phased budget resources consistent with your program master schedule? Show the 

trace from your control account to intermediate or master schedules.

■ When are you required to plan planning packages or summary activities in detail? What 

schedule document or system is used to develop detail planning for your control account?

■ How do you know that the work within your control accounts to be performed by 

subcontractor has been properly planned?

■ How do you check the status and performance of work on your control account by a 

subcontractor?

■ How are actual costs recorded against your cost account?

■ What techniques are available for determining earned value? Explain the techniques you are 

using for this control account.

■ How and when is the risk assessment or risk management plan updated for technical, schedule, 

and cost risk items affecting your control account?

■ How and when is the actual and forecast schedule update provided for your control account 

effort?

■ Are variance analysis thresholds or requirements established for reporting technical, schedule, 

or cost variances to planned goals established for your control accounts? Do you informally or 

formally report the cause of variance, impact, or corrective action for these variances?

■ What document authorizes you to begin work on a subcontract? 

■ For these selected work packages, what specific outputs, products, or objectives are to be 

accomplished?

■ What specifically do you need from other control account managers to generate subcontractor 

outputs or products? How do you monitor progress?

■ Who specifically needs the subcontractor outputs or products to perform their program 

functions? How do you status others on the progress of your outputs to them?
392 APPENDIX XIV GAO-09-3SP

■ Specifically, what technical items are the greatest risk to achieving technical, schedule, or 

cost goals? Are these items reviewed as part of a risk assessment, management plan, or other 

reporting tool to your boss or the program management office?

■ How do you determine whether the reported cost variance stems from subcontractor effort or 

company overhead rate?

Have material budgets been planned? 

Is material tracked before delivery? 

How do you track material when deliveries are late? 

When is BCWP or earned value taken on material? 

Analysis

Do you have any variance thresholds for your control accounts? 

What are the variance thresholds for your control accounts? 

How do you know when you have exceeded a threshold? 

Do you have samples of any variance analysis reports? Do they show a statement of the problem: 

the variance, cause, and impact; and proposed corrective action?

Who receives your variance reports? What action is taken on the reports? 

Which reports do you use most frequently? Why? 

Not Categorized

How are you reporting labor, material, and other direct costs? 

Has your IPT effort been affected by any directed contractual change? When did you receive 

authorization to proceed with the change and how did your IPT incorporate the change in its 

planning (schedule and budget time phasing)? 

Demonstrate that the current planning for your IPTs product delivery and services supports 

program IPT’s and contract delivery commitments. 

What changes have been made to the control account planning (technical definition of scope, 

schedule, budget resources, ETCs)?

■ What documents are involved in a change to the control accounts’ scope of work, schedule, 

budget, or ETC?

■ Did the control account manager rephase or replan work? In-process work? Completed work? 

Unopened work packages? Make current period or retroactive changes?

■ Did the control account manager transfer budget between control accounts?

■ How have contract changes or other changes been incorporated into the control account?
GAO-09-3SP APPENDIX XV 393

Common Risks to Consider in Software Cost Estimating

This appendix lists common risks in software acquisition and offers some possible risk containment 

actions to mitigate certain effects. It is organized by key area of software acquisition: (1) requirements, 

(2) design, (3) test and evaluation, (4) technology, (5) developer, (6) cost or funding, (7) monitoring, 

(8) schedule, (9) personnel resources, (10) security and privacy, (11) project implementation strategy and 

plans, (12) specific commercial off-the-shelf risks, (13) business risk, and (14) management.

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

1. Requirements

Does not reflect 

user needs

System rejection; 

program 

operations 

adversely 

affected; cost 

increases; rework

x x x x x x x x Those affected 

manage 

requirements, 

with review and 

approval 

Too many or 

too restrictive 

design and 

implementation 

constraints 

Infeasibility; 

increased 

cost to meet 

requirements, 

poor design

x x x x x x x x Rewrite 

and review 

requirements to 

address functions 

embedded in 

the constraints; 

perform costbenefit analysis 

of constraints 

and remove 

unnecessary or 

costly constraints

Uncertain 

requirements

Rework; 

unsuitable 

product, cost, 

and schedule 

increases; 

inaccurate 

cost estimates; 

possibly 

infeasible end 

product 

x x x x x x x x x x Define 

requirements 

before 

proceeding 

to next stage; 

prototype; hedge 

cost and schedule 

for risk; divide 

end-product in 

segments and 

prioritize for 

implementation
394 APPENDIX XV GAO-09-3SP

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Unstable Rework; 

unsuitable 

product; cost 

and schedule 

increases

x x x x x x x x x x Limit size of 

implementation 

segments; 

prototype

Untraceable Design does 

not meet 

requirements; 

rework; cost 

and schedule 

increases; 

unreliable testing

x x x x x x x x x Establish 

and maintain 

traceability to 

products and 

tests

2. Design

Does not achieve 

performance 

objectives

Increased 

program 

operation costs

x x x x Review design 

for alternatives; 

establish 

performance 

objectives for 

acceptance; 

simulations

Does not meet 

requirements

System rejection; 

rework

x x x x x x x Establish 

traceability to 

requirements

Infeasibility Product does not 

work

x x x x x x x x Review design, 

including 

feasibility analysis

Not cost 

effective

Increased 

maintenance 

costs

x x x x Analyze design 

for effectiveness 

and other design 

alternatives 

before coding

More training 

needed

Increased 

program 

operation costs

x x x x Review design for 

alternatives

3. Test and evaluation

Does not address 

operating 

environment

Poor system 

performance; 

increased 

operation and 

maintenance 

costs

x x x x x x Perform 

operational 

capability testing
GAO-09-3SP APPENDIX XV 395

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Inadequate 

acceptance 

testing

Premature 

acceptance; 

increased 

operation and 

maintenance 

costs

x x x x x x x Plan and 

allow time for 

acceptance 

testing; establish 

traceability 

to design and 

requirements

Insufficient time 

to fix

More acceptance 

testing needed

x x x x x x x x x Increase schedule

Insufficient 

time to test 

thoroughly

More acceptance 

testing needed

x x x x x x Require test 

plans, reports, 

and compliance 

matrices as 

deliverables; 

increase schedule 

to allow for 

adequate testing

Test planning 

not begun 

during initial 

development 

Increased costs; 

inadequate 

testing; rework

x x x x x x x Begin acceptance 

test planning 

immediately after 

requirements 

are baselined; 

establish 

traceability to 

requirements

Test procedures 

do not address 

all major 

performance 

and reliability 

requirements

Poor system 

performance; 

poor product 

quality; more 

acceptance 

testing needed; 

negative effect 

on program 

operations; 

rework

x x x x x x x x Establish 

traceability to all 

requirements; 

include 

performance 

and reliability 

requirements 

as acceptance 

criteria; require 

test plans, reports, 

and compliance 

matrices as 

deliverables
396 APPENDIX XV GAO-09-3SP

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Various levels 

of testing are 

not performed 

(system, 

integration, unit)

Poor system 

performance; 

more acceptance 

testing needed; 

poor product 

quality

x x x x x x Establish 

traceability to all 

requirements; 

include testing 

requirements in 

contract; require 

test plans, reports, 

and compliance 

matrices as 

deliverables

4. Technology

Availability Needed 

functionality 

delayed; 

increased 

program 

operation 

costs; business 

disruption

x x x x x Devise alternate 

business 

processes; 

consider other 

technology

Potential 

advances result 

in system’s not 

being costeffective

Increased 

program 

operating and 

operations and 

maintenance 

costs

x x x Consider 

replacing; change 

business process 

Potential 

changes 

make other 

components 

obsolete

Increased 

operations and 

maintenance 

costs; program 

disruption

a x x x x x x x x Periodic review 

of architecture 

and of changes 

in technology 

field; regular cycle 

upgrade 

Relies on 

complex design

Increased 

program 

operations 

costs (additional 

training); 

reduced 

cost-benefit; 

increased 

operations and 

maintenance 

costs

x x x x x x Simplify and 

review design; 

conduct and 

compare parallel 

design activities; 

prototyping
GAO-09-3SP APPENDIX XV 397

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Technology 

becomes 

obsolete or is 

abandoned

Program 

disruption; 

needed 

functionality 

unavailable; 

new acquisition 

required

a x x x x x Scheduled 

upgrades; 

periodic review 

of changes in 

technology field

Unproven or 

unreliable

Program 

disruption

Increased 

operations and 

maintenance 

costs

x x x x x x x Operational 

capability 

demonstrations 

and testing; delay 

acquisition; use 

other technology

5. Developer

Ability to 

produce item; 

poor track 

record for costs 

and schedule; 

key personnel 

turnover

Needed 

functionality 

unavailable 

or delayed; 

increased 

financial risk 

to acquirer; 

increased cost, 

reduced quality, 

delays

x x x x x x x x x x Terminate 

contract; 

performancebased contract; 

limited task 

orders; EVM or 

similar system

6. Cost or funding

Funding type 

does not match 

acquisition 

strategy

Uncertain 

financing; 

changes in 

project direction

b b b b b b b b b b Plan new 

acquisition 

strategy to map to 

funding type

Marginal 

performance 

capabilities 

incorporated at 

excessive costs 

(cost-benefit 

tradeoffs not 

performed)

Increased 

program 

operation costs 

and operations 

and maintenance 

costs for little 

benefit

x x x Maintain updated 

business case 

analysis return 

on investment; 

re-scope; riskadjusted cost 

benefit analysis of 

alternatives

Realistic cost 

objectives not 

established early

Greater financial 

risk; potential 

delays

x x x x x x x x x x Scheduled 

estimate updates 

and reviews
398 APPENDIX XV GAO-09-3SP

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Schedule not 

considered 

in choosing 

alternative 

implementation 

strategies

Unrealistic 

schedule; 

increased costs; 

reduced quality; 

infeasibility; 

changes in 

project direction

x x x x x x x x x x Re-examine 

business case and 

update for better 

information

Unstable 

funding

Increased costs 

and schedule 

and reduced 

quality due to 

stop and start; 

loss of project 

momentum

b b b b b b b b b b Implement 

incremental 

or modular 

segments that can 

be funded

7. Monitoring

Insufficient 

monitoring

Transfer of risk to 

acquirer; inability 

to recover 

from schedule 

slippage for 

lack of timely 

information

x x x x x x x x x Increase 

monitoring in 

riskiest areas

8. Schedule

Dependence on 

other projects

Delays; increased 

costs

x x x x x x x x x x Reduce 

dependencies; 

coordinate with 

another project

Insufficient 

resources to 

meet schedule

Delays; poor 

quality

x x x x x x x x x x Increase resource 

or schedule; 

reduce scope

Schedule may be 

delayed waiting 

for external 

approvals

Inability to 

meet due date; 

increased cost; 

reduced scope; 

product failure

x x x x x x x x x x Reduce scope; 

increase resources 

where reasonable, 

allowing 

for greater 

coordination 

complexity; 

extend due date

Tasks allocated 

poorly

Poor quality; 

inefficiency

x x x x x x x x x x Reallocate or 

restructure tasks
GAO-09-3SP APPENDIX XV 399

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Unrealistic Poor quality 

from shortening 

vital tasks; 

cost increases; 

infeasibility

x x x x x x x x x x Increase schedule; 

reduce scope

9. Personnel resources

Inadequate 

skills or staffing; 

inadequate mix 

of staff and skills

Poor quality; 

delays; increased 

costs

x x x x x x x x x x Training; hire 

more staff; 

reassign tasks or 

staff outside the 

project

10. Security and privacy

Inadequate Failure to certify 

system; system 

breaches; data 

corruption

x x x x x x x x x x Address security 

in requirements, 

design, and 

testing; test 

periodically or 

as changes are 

made to ensure 

continued 

security and 

privacy

11. Project implementation strategy and plans

Architectural 

dependencies

Architectural 

components

x x x x x x Adhere to 

enterprise 

architecture

Dependence on 

other projects or 

systems

Schedule delay; 

cost increases; 

uncontrolled 

changes; service 

disruption

x x x x x x x x x x Memorandum of 

understanding 

with other 

project or system 

owner; regular 

coordination 

meetings; reduce 

dependencies 

and predecessors

Inadequate link 

to mission need

Unnecessary 

system 

components; 

increased costs 

or schedule

x x x x Check back to 

system goals 

in each system 

phase
400 APPENDIX XV GAO-09-3SP

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Nonmodular 

development 

and 

implementation 

approach

Funding risks; 

schedule; 

cost; quality; 

requirements 

instability

x x x x x x x x x x Implement 

incremental 

or modular 

segments

Overall 

implementation 

strategy does 

not properly 

address key 

development 

phases and 

technology 

considerations

Troubled project; 

product defects

x x x x x x x Follow industry 

standards in 

development and 

tailor to nature of 

technology being 

implemented

Risks not acted 

on

Troubled 

project; product 

defects; cost 

and schedule 

increases

x x x x x x x x x x Manage risks 

Significant 

custom 

development 

required, with 

additional 

requirements

Product 

problems; 

increased 

operations and 

maintenance 

costs

x x x x x x x x x Consider other 

technology or 

alternatives; 

reengineer or 

simplify business 

process 

Subordinate 

strategies 

and plans not 

developed in a 

timely manner

Delays; 

inefficiency

x x x x x Create strategies 

and plans in a 

timely manner

Subordinate 

strategies 

and plans not 

linked to overall 

strategy

Delays; 

inefficiency

x x x x x Create links to 

higher- level plans

Wide scope of 

work

Increased 

complexity, cost, 

or schedule

x x x x x x x x x x Prioritize 

requirements
GAO-09-3SP APPENDIX XV 401

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

12. Commercial off-the-shelf 

Long-term 

maintenance 

and support not 

considered

Inability to 

perform 

maintenance; 

lack of skills or 

resources

x x x x Transition plan; 

training; link to 

business and 

human resources 

goals

Planned life of 

more than 5 

years per version 

or vendor

Obsolescence; 

lack of vendor 

support if 

business is sold 

or turns down

x x x Planned upgrades 

or replacement

Significant 

number of 

complex 

interfaces to 

other systems

Data quality 

and timeliness; 

increased 

maintenance; 

additional 

requirements

x x x x x x x x Simplify or reduce 

interfaces

Total life cycle 

costs not 

considered

System goals 

not reached; 

uncertain 

operations and 

maintenance 

costs

x x x Initial business 

case analysis with 

cost-adjusted 

risk analysis of 

alternatives; 

return on 

investment

13. Business 

Inappropriate 

development 

approach or 

methodology 

Cost and 

schedule 

overruns; failure; 

lost opportunity

x x x x x x x x Use appropriate 

development 

model (spiral, 

incremental, 

waterfall)

Downtime 

affects service or 

business

Loss of 

opportunity; 

customer 

dissatisfaction 

and complaints; 

inability to 

perform business 

function

x x Monitor system 

performance; 

periodic upgrades 

or enhancements; 

backup and 

redundancy
402 APPENDIX XV GAO-09-3SP

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Technology 

is built on 

old business 

processes

Inefficiency or 

redundancy; 

operational 

failure; increased 

personnel costs

x x x Benchmark 

and reengineer 

or streamline 

processes

14. Management

Demand for 

service is higher 

than resources 

can handle

Staff burnout; 

reduced service 

quality; loss 

of customer 

satisfaction

x x x x x x x x x x Prioritize service 

to ensure quality 

remains high

Inability to 

manage the 

complexity or 

scope of system, 

project, or 

operations 

Failure; system 

or implemented 

processes 

degrade 

from original 

implementation; 

project goes over 

schedule or over 

budget

x x x x x x x x x x Training; use 

project sponsor to 

ensure ongoing 

coordination 

across groups; 

decrease scope; 

break effort into 

segments

Outside 

pressures 

to produce 

something 

shortens the 

requirements 

period

Cost and 

schedule 

increase; 

disputes with 

customer about 

end-product; 

contract 

disputes; 

substandard 

product

x x x x x x x x x x Resist pressure 

to bypass 

requirements; 

use prototypes 

or storyboards 

to produce 

something 

and refine 

requirements; 

select a different 

development 

approach
GAO-09-3SP APPENDIX XV 403

Potential effect on

Area and risk Potential effectDesignCodeTestAcceptance testingOperations and maintenanceProgram operationsUser acceptanceTrainingUpgradesDocumentation

Possible risk 

containment 

action

Progress is 

affected by 

unstable or 

unstated 

business 

processes

Failure or 

rejection; 

increased 

schedule 

and cost to 

implement; 

additional 

changes to 

system; user 

confusion; 

business process 

inefficiency or 

ineffectiveness

x x x x x x x x x x Design 

processes before 

committing to 

system design

Users or 

organizations 

resist change

Failure or 

rejection; 

increased 

schedule 

and cost to 

implement; 

additional 

changes to 

system

x x x x x x x x x x Involve parties 

in requirements 

and design, 

communication, 

organizational 

change 

management 

plan; bring project 

sponsor to upper 

management; 

training and 

support

Source: GAO.

a

Architecture.

b

Impact area depends on financing and timing.
404 APPENDIX XVI GAO-09-3SP

Contacts and Acknowledgments

GAO CONTACTS

Karen Richey, at 202-512-4784, richeyk@gao.gov;

Jennifer Echard, at 202-512-3875, echardj@gao.gov; and

Carol Cha, at 202-512-4456, chac@gao.gov.

ACKNOWLEDGMENTS

Other key contributors to this guide include Gregory Borecki, Cristina Chaplain, Richard Hung, 

Jason Kelly, Nancy Kingsbury, Colleen Phillips, Penny Pickett, David Powner, Keith Rhodes, and 

Adam Vodraska.
GAO-09-3SP References 405

References

Abba, Wayne. Contracting for Earned Value Management. Falls Church, Va.: Abba Consulting, Jan. 13, 2007.

———. Earned Value Management: The Good, the Bad, and the Really Ugly. Falls Church, Va.: Abba 

Consulting, Dec. 21, 2004.

———. “Earned Value Management from Government Contracts to the Project Management Universe.” 

Presented at the 19th IPMA World Congress, New Delhi, India, November 13–16, 2005.

———. How Earned Value Got to Primetime: A Short Look Back and a Glance Ahead. Reston, Va.: Dekker, 

2000.

———. Understanding Program Resource Management through Earned Value Analysis. Falls Church, Va.: 

Abba Consulting, June 2006.

Agrawal, Raj. Overcoming Software Estimation Challenges. McLean, Va.: MITRE, May 22, 2007.

Albert, Neil F. Cost Estimating: The Starting Point of Earned Value Management. McLean, Va.: MCR, June 

2005.

———. Developing a Work Breakdown Structure. McLean, Va.: MCR, June 16, 2005.

Anderson, Mark, and David Nelson. Developing an Averaged Estimate at Completion (EAC) Utilizing Program 

Performance Factors and Maturity. Arlington, Va.: Tecolote Research, June 14–17, 2005.

ANSI (American National Standards Institute). Earned Value Management Systems Electronic Industries 

Alliance, (EIA)-748-B. Arlington, Va.: Government Electronics and Information Technology Association, 

2007.

Barrett, Bruce E., Jack E. Matson, and Joseph M. Mellichamp. Software Development Cost Estimation Using 

Function Points. Tuscaloosa: University of Alabama, April 1994.

Black, Hollis M. Impact of Cost Risk Analysis on Business Decisions. Huntsville, Ala.: Boeing, June 14–17, 2005.

Boatwright, Tim. Earned Value Management System Acceptance and Surveillance. Booz Allen Hamilton: 

Mar. 30, 2006.

Boden, Dave, and others. Interfacing Risk and Earned Value Management. Buckinghamshire, Eng.: 

Association for Project Management, 2008.

Bolinger, Paul. Schedule Analysis of the Integrated Master Schedule. [Orange, Calif.]: Humphreys & Associates, 

May 2008.

Bone, Lauren, and Val Jonas. “Interfacing Earned Value and Risk Management.” PMI-CPM 24th Annual 

International Conference Proceedings, Clearwater Beach, Florida, May 14–16, 2008.

Book, Stephen A. “Do Not Sum Most Likely Costs.” Presentation to the American Society to American 

Comptrollers, Los Angeles, California, April 30, 2002.
406 References GAO-09-3SP

———. Issues Associated with Basing Decisions on Schedule Variance in an Earned Value Management System. 

McLean, Va.: MCR, 2003.

———. The Program Schedule Network and Its Critical Path. McLean, Va.: MCR, Aug. 26, 2005.

———. Risks in Costing Software. McLean, Va.: MCR, Aug. 26, 2005.

———. Schedule Risk Analysis: Why It Is Important and How to Do It. McLean, Va.: MCR, June 14–17, 

2005.

———. The Schedule Risk Imperative and How to Deal with It. McLean, Va.: MCR, Aug. 26, 2005.

———. What Is Cost Risk Analysis? McLean, Va.: MCR, Aug. 26, 2005. 

———. What Earned Value Data Tells Us about Schedule Progress. McLean, Va.: MCR, Aug. 26, 2005.

Brophy, Brian A., and Greg Hogan. How to Get the Data and Ready It for Analysis. Chantilly, Va.: Northrop 

Grumman, The Analytical Sciences Corporation, 2005. 

Buchholz, Mark, Shaw Cohe, and Robert Tomasetti. Earned Value Management: Moving toward 

Governmentwide Implementation. Arlington, Va.: Acquisition Directions Advisory, August 2005. 

Cargill, John. Decision Support for the Program Manager. Eglin Air Force Base, Fla.: Air Force Cost Analysis 

Agency, n.d.

———. Forecasting the Future with EVMS. Eglin Air Force Base, Fla.: Air Force Cost Analysis Agency, n.d.

Carroll, Ed. Software Estimating Based on Use Case Points. [Beaverton, Ore.]: Agilis Solutions, June 2005.

Christensen, David S. The Costs and Benefits of the Earned Value Management Process. Pensacola: University of 

West Florida, June 1998.

———. Determining an Accurate Estimate at Completion. Cedar City: Southern Utah University, 1993.

———. Using the Earned Value Cost Management Report to Evaluate the Contractor’s Estimate at Completion. 

Cedar City: Southern Utah University, 1999.

———, and Carl Templin. An Analysis of Management Reserve Budget on Defense Acquisition Contracts. 

Cedar City: Southern Utah University, 2000.

———, and others. A Review of the Estimate at Completion Research. Cedar City: Southern Utah University, 

1995.

Chvotkin, Alan, and Harris N. Miller. Response to FAR Case 2004-019: Earned Value Management Systems, 

70 Fed. Reg. 17945 (April 8, 2005). Washington, D.C.: Information Technology Association of America 

and Professional Services Council, June 7, 2005.

Coleman, Richard L.,and Shishu S. Gupta. Two Timely Short Topics: Independence and Cost Realism. 

Chantilly, Va.: Northrop Grumman, Analytical Sciences Corp., and Intelligence Community Cost 

Analysis Improvement Group, June 16, 2005.
GAO-09-3SP References 407

———, and Jessica R. Summerville. Advanced Cost Risk. Chantilly, Va.: Northrop Grumman, The 

Analytical Sciences Corp., June 16, 2005. 

———. Basic Cost Risk. Chantilly, Va.: Northrop Grumman, The Analytical Sciences Corp., June 15, 2005. 

———, and others. Monte Carlo Techniques for Risk Management. Chantilly, Va.: Northrop Grumman, The 

Analytical Sciences Corp., June 2008. 

Committee ANSI/PMI 00-001-2004. A Guide to the Project Management Body of Knowledge, 3rd ed. 

Arlington, Va.: 2004. 

Comptroller General of the United States. Government Auditing Standards: January 2007 Revision, GAO-07-

162G. Washington, D.C.: U.S. Government Accountability Office, January 2007.

———. Theory and Practice of Cost Estimating for Major Acquisitions, B-163058. Washington, D.C.: U.S. 

Government Accountability Office, July 24, 1972.

Cooper, L. Sue. Basic Schedule Development and Analysis. Seal Beach, Calif.: Boeing, June 14–17, 2005.

Corrigan, Patricia. Capital Programming, Earned Value, and the Federal Acquisition Community. Washington, 

D.C.: Office of Federal Procurement Policy, Office of Management and Budget, April 2007.

DAU (Defense Acquisition University). Business, Cost Estimating, and Financial Management Training 

Requirements. Fort Belvoir, Va.: n.d.

———. Cost Estimating Methodologies. Fort Belvoir, Va.: April 2004.

———. Defense Acquisition Acronyms and Terms. Fort Belvoir, Va.: July 2005.

———. Defense Acquisition Guidebook. Fort Belvoir, Va.: July 24, 2006.

———. Introduction to Cost Analysis. Fort Belvoir, Va.: April 2004.

———. Unit and Cumulative Average Formulations. Fort Belvoir, Va.: Feb. 20, 2004.

DCAA (Defense Contract Audit Agency). DCAA Contract Audit Manual. Fort Belvoir, Va.: January 12, 

2008.

DCARC (Defense Cost and Resource Center). Automated Information System Enterprise Resource Planning 

Work Breakdown Structure. Arlington, Va.: n.d.

———. Automated Information System Enterprise Resource Planning Work Breakdown Structure Dictionary. 

Arlington, Va.: n.d.

DCMA (Defense Contract Management Agency). About the Defense Contract Management Agency. 

Alexandria, Va.: n.d. 

———. Earned Value Management System: System Surveillance. Washington, D.C.: 2005.

———. EVMS Standard Surveillance Operating Manual. Washington, D.C.: January, 2008.
408 References GAO-09-3SP

DCMA (Defense Contract Management Agency). EVMS Standard System Surveillance. Washington, D.C.: 

May 14, 2008.

———. Software Acquisition Management Services. Washington, D.C.: 2005.

Dean, Joe. Practical Software and Systems Measurement: Objective Information for Decision Makers. Boston, 

Mass.: Tecolote Research, June 2005.

Dechoretz, Jason. Technical Baselines. McLean, Va.: MCR, June 2005.

Defense Economic Analysis Council. Economic Analysis Handbook. Monterey, Calif.: September 1997. 

Dello Russo, Francis M., Paul R. Garvey, and Neal D. Hulkower. Cost Analysis. Bedford, Mass.: MITRE, 

1998.

DHS (Department of Homeland Security). Capital Planning and Investment Control Cost Benefit Analysis 

Guidebook. Washington, D.C.: 2006. 

———. Investment and Acquisition Management Re-Engineering: Improving Life Cycle Cost Estimating. 

Washington, D.C.: July 10, 2008.

DOD (Department of Defense). Automated Information System (AIS) Economic Analysis Guide. Washington, 

D.C.: May 1, 1995.

———. Contract Performance Report Data Item Description. Washington, D.C.: Mar. 30, 2005.

———. Contract Pricing Reference Guide. Washington, D.C.: 2002.

———. Contractor Cost and Software Data Reporting. Washington, D.C.: June 8, 2007.

———. Cost Analysis Guidance and Procedures, DOD 5000.4-M. Washington, D.C.: December 1992.

———. Cost Analysis Improvement Group, DOD 5000.4. Washington, D.C.: Nov. 24, 1992.

———. The Defense Acquisition System, DOD Directive 5000.01. Washington, D.C.: Nov 20, 2007.

———. Defense Contract Management Agency, DOD Directive 5105.64. Washington, D.C.: Nov. 21, 2003.

———. Department of Defense Handbook: Work Breakdown Structures for Defense Materiel Items, MILHDBK-881A. Washington, D.C.: Office of the Under Secretary of Defense (AT&L), July 30, 2005.

———. Earned Value Management Glossary of Terms. Washington, D.C.: n.d.

———. Earned Value Management Implementation Guide. Washington, D.C.: October 2006.

———. Earned Value Management Policy and Initiatives. Washington, D.C.: May 2008.

———. Earned Value Management Roles and Responsibilities. Washington, D.C.: July 3, 2007.

———. Economic Analysis for Decisionmaking, DOD 7041.3. Washington, D.C.: Nov. 7, 1995.

———. Enterprise Resource Planning Information. Washington, D.C.: April 20, 2006.
GAO-09-3SP References 409

———. A Guide to the Project Management Body of Knowledge. Washington, D.C.: June 2003.

———. Implementation of Central Repository System. Washington, D.C.: July 11, 2007.

———. Instructional Systems Development/Systems Approach to Training and Education, MIL-HDBK-29612-

2A. Washington, D.C.: Aug. 31, 2001.

———. Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide. Washington, 

D.C.: Oct. 21, 2005.

———. Integrated Master Schedule Data Item Description. Washington, D.C.: Mar. 30, 2005.

———. National Security Space Acquisition Policy. Washington, D.C.: Dec. 27, 2004.

———. Operating and Support Cost-Estimating Guide. Washington, D.C.: May 1992.

———. Operation of the Defense Acquisition System, DOD Instruction 5000.02. Washington, D.C.: Dec 8, 

2008.

———. Over Target Baseline and Over Target Schedule. Washington, D.C.: May 7, 2003. 

———. Primer on Cost Analysis Requirement Descriptions. Washington, D.C.: Jan. 28, 2004.

———. The Program Manager’s Guide to the Integrated Baseline Review Process. Washington, D.C.: June 

2003.

———. Revised Department of Defense Earned Value Management Policy. Washington, D.C.: May 12, 2005.

———. Revision to Department of Defense Earned Value Management Policy. Washington, D.C.: Mar. 7, 

2005.

———. Risk Management Guide for DOD Acquisition, 6th ed., version 1.0. Washington, D.C.: August 

2006.

———. “Single Point Adjustments.” Memorandum from D. M. Altwegg, Business Management, Missile 

Defense Agency, to Missile Defense Agency Program Divisions, Washington, D.C., December 8, 2005. 

———. Software Acquisition Gold Practice: Track Earned Value. Washington, D.C.: Data Analysis Center for 

Software, Dec. 12, 2002.

———. Software Resources Data Report Manual, DOD 5000.4-M-2. Washington, D.C.: Feb. 2, 2004.

———. Use of Earned Value Management in DOD. Washington, D.C. July 3, 2007

———, General Services Administration, and National Aeronautics and Space Administration. Federal 

Acquisition Regulation (FAR) Major System Acquisition, 48 C.F.R. part 34, Earned Value Management 

System, subpart 34.2, added by Federal Acquisition Circular 2005-11, July 5, 2006, as Item I—Earned 

Value Management System (EVMS) (FAR Case 2004-019).

DOE (Department of Energy). Cost Estimating Guide, DOE G 430.1.1. Washington, D.C.: Mar. 28, 1997.
410 References GAO-09-3SP

DOE (Department of Energy). Project Management and the Project Management Manual. Washington, D.C.: 

Mar. 21, 2003. 

———. Work Breakdown Structure. Washington, D.C.; June 2003.

Driessnack, John D. Using Earned Value Data. Arlington, Va.: MCR, June 2005.

———, and Neal D. Hulkower. Linking Cost and Earned Value Analysis. Arlington, Va.: MCR, June 2005.

Druker, Eric, Christina Kanick, and Richard Coleman. The Challenges of (and Solutions for) Estimating 

Percentiles of Cost. Arlington, Va.: Northrop Grumman, Aug, 11, 2008.

FAA (Federal Aviation Administration). Investment Analysis Standards and Guidelines: FAA Standard Cost 

Estimation Guidelines, version 1.0. Washington, D.C.: April 2003.

Federal Register. Rules and Regulations, vol. 73, no. 79. Washington, D.C.: Apr. 23, 2008.

Ferens, Dan. Commercial Software Models. Vienna, Va.: International Society of Parametric Analysis, June 

2005.

Fleming, Quentin W. Earned Value Management (EVM) Light . . . but Adequate for All Projects. Tustin, Calif.: 

Primavera Systems, November 2006.

———, and Joel M. Koppelman. “The Earned Value Body of Knowledge.” Presented at the 30th Annual 

Project Management Institute Symposium, Philadelphia, Pennsylvania, October 10–16, 1999.

———. “Performance Based Payments.” Tustin, Calif.: Primavera Systems, November 2007.

Flett, Frank. Organizing and Planning the Estimate. McLean, Va.: MCR, June 12–14, 2005.

Galorath, Daniel D. Overcoming Cultural Obstacles to Managing Risk. El Segundo, Calif.: Galorath Inc., 

2007.

———. Software Estimation Handbook. El Segundo, Calif.: Galorath Inc., n.d.

———. Software Projects on Time and within Budget—Galorath: The Power of Parametrics. El Segundo, 

Calif.: Galorath Inc., n.d. 

———. Software Total Ownership Cost Development Is Only Job 1. El Segundo, Calif.: Galorath Inc., 2008.

GAO. 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP. 

Washington, D.C.: February 2005.

GAO. Air Traffic Control: FAA Uses Earned Value Techniques to Help Manage Information Technology 

Acquisitions, but Needs to Clarify Policy and Strengthen Oversight, GAO-08-756. Washington, D.C.: July 

18, 2008. 

GAO. Aviation Security: Systematic Planning Needed to Optimize the Deployment of Checked Baggage Screening 

Systems, GAO-05-365. Washington, D.C.: Mar. 15, 2005.
GAO-09-3SP References 411

GAO. Best Practices: Better Acquisition Outcomes Are Possible If DOD Can Apply Lessons from the F/A-22 

Program, GAO-03-645T. Washington, D.C.: Apr. 11, 2003.

GAO. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes, 

GAO/NSIAD-99-162. Washington, D.C.: Aug. 16, 1999.

GAO. Best Practices: Successful Application to Weapon Acquisitions Requires Changes in DOD’s Environment, 

GAO/NSIAD-98-56. Washington, D.C.: Feb. 24, 1998. 

GAO. Chemical Demilitarization: Actions Needed to Improve the Reliability of the Army’s Cost Comparison 

Analysis for Treatment and Disposal Options for Newport’s VX Hydrolysate, GAO-07-240R. Washington, 

D.C.: Jan. 26, 2007.

GAO. Combat Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detection Equipment at U.S. 

Ports of Entry, but Concerns Remain, GAO-06-389. Washington, D.C.: Mar. 22, 2006.

GAO. Combat Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support the Purchase of New Radiation 

Detection Portal Monitors Was Not Based on Available Performance Data and Did Not Fully Evaluate All the 

Monitors’ Costs and Benefits, GAO-07-133R. Washington, D.C.: Oct. 17, 2006.

GAO. Cooperative Threat Reduction: DOD Needs More Reliable Data to Better Estimate the Cost and Schedule 

of the Shchuch’ye Facility, GAO-06-692. Washington, D.C.: May 31, 2006.

GAO. Customs Services Modernization: Serious Management and Technical Weaknesses Must Be Corrected, 

GAO/AIMD-99-41. Washington, D.C.: Feb. 26, 1999.

GAO. Defense Acquisitions: Assessments of Selected Major Weapon Programs, GAO-06-391. Washington, D.C.: 

Mar. 31, 2006.

GAO. Defense Acquisitions: Improved Management Practices Could Help Minimize Cost Growth in Navy 

Shipbuilding Programs, GAO-05-183. Washington, D.C.: Feb. 28, 2005.

GAO. Defense Acquisitions: Information for Congress on Performance of Major Programs Can Be More Complete, 

Timely, and Accessible, GAO-05-182. Washington, D.C.: Mar. 28, 2005.

GAO. Defense Acquisition: A Knowledge-Based Funding Approach Could Improve Weapon System Program 

Outcomes, GAO-08-619. Washington, D.C.: June 2008. 

GAO. Defense Acquisitions: Missile Defense Agency Fields Initial Capability but Falls Short of Original Goals, 

GAO-06-327. Washington, D.C.: Mar. 15, 2006.

GAO. DOD Systems Modernization: Planned Investment in the Navy Tactical Command Support System Needs 

to Be Addressed, GAO-06-215. Washington, D.C.: Dec. 5, 2005.

GAO. GAO’s High Risk Program, GAO-06-497T. Washington, D.C.: Mar. 15, 2006.

GAO. Global Hawk Unit Cost Increases, GAO-06-222R. Washington, D.C.: Dec. 15, 2005.

GAO. A Glossary of Terms Used in the Federal Budget Process, GAO-05-734SP. Washington, D.C.: September 

2005.
412 References GAO-09-3SP

GAO. Government Auditing Standards 2007 Revision, GAO-07-162G. Washington, D.C.: January 2007.

GAO. Homeland Security: Recommendations to Improve Management of Key Border Security Program Need to 

Be Implemented, GAO-06-296. Washington, D.C.: Feb. 14, 2006.

GAO. Information Technology: Agencies Need to Improve the Accuracy and Reliability of Investment Information, 

GAO-06-250. Washington, D.C.: Jan. 12, 2006. 

GAO. Major Acquisitions: Significant Changes Underway in DOD’s Earned Value Management Process, GAO/

NSIAD-97-108. Washington, D.C.: May 5, 1997. 

GAO. Maximizing the Success of Chief Information Officers, GAO-01-376G. Washington, D.C.: Feb. 1, 2001. 

GAO. NASA: Lack of Disciplined Cost Estimating Processes Hinders Effective Program Management, GAO-04-

642. Washington, D.C.: May 28, 2004.

GAO. National Airspace System: Better Cost Data Could Improve FAA’s Management of the Standard Terminal 

Automation System, GAO-03-343. Washington, D.C.: Jan. 31, 2003.

GAO. Polar-Orbiting Operational Environmental Satellites: Cost Increases Trigger Review and Place Program’s 

Direction on Hold, GAO-06-573T. Washington, D.C.: Mar. 30, 2006.

GAO. Polar-Orbiting Operational Environmental Satellites: Information on Program Cost and Schedule 

Changes, GAO-04-1054. Washington, D.C.: Sept. 30, 2004.

GAO. Space Acquisitions: DOD Needs to Take More Action to Address Unrealistic Initial Cost Estimates of Space 

Systems, GAO-07-96. Washington, D.C.: Nov. 17, 2006.

GAO. Standards for Internal Control in the Federal Government: Exposure Draft, GAO/AIMD-98-21.3.1.

Washington, D.C.: Dec. 1, 1997. 

GAO. Telecommunications: GSA Has Accumulated Adequate Funding for Transition to New Contracts but 

Needs Cost Estimation Policy, GAO-07-268. Washington, D.C.: Feb. 23, 2007.

GAO. Theory and Practice of Cost Estimating for Major Acquisitions, B-163058. Washington, D.C.: July 24, 

1972.

GAO. Uncertainties Remain Concerning the Airborne Laser’s Cost and Military Utility, GAO-04-643R. 

Washington, D.C.: May 17, 2004. 

GAO. United States Coast Guard: Improvements Needed in Management and Oversight of Rescue System 

Acquisition, GAO-06-623. Washington, D.C.: May 31, 2006.

Garvey, Paul R. Cost Risk Analysis without Statistics!! McLean, Va.: MITRE, February, 2005.

Geiser, Todd A., and David Schaar. “Tackling Cost Challenges of Net Centric Warfare,” pp. 4083–92. In 

2004 IEEE Aerospace Conference Proceedings, vol. 6. Proceedings of the 2004 IEEE Aerospace Conference, 

Institute of Electrical and Electronics Engineers. Big Sky, Mt.: March 6–13, 2004. 
GAO-09-3SP References 413

Gillam, Dale E. “Evaluating Earned Value Management and Earned Value Management Systems Using 

OMB’s Program Assessment Rating Tool (PART).” The Measurable News, Spring 2005, pp. 1–7.

Gordon, Creaghe. “The Importance of Reserves and Budget Distribution in Achieving Budget 

Requirements.” Flying High, 1st Quarter, 2006, pp. 6–8. 

———, and George Rosenthal. Proposal for Combining Risk and Earned Value Management. Los Gatos, 

Calif.: n.p., n.d. 

Groves, Angela, and Karen McRitchie. Better Software Estimates Using SEER-SEM. El Segundo, Calif.: 

Galorath Associates, n.d.

GSA (General Service Administration). Project Estimating Requirements. Washington, D.C.: January 2007.

Harris, Michael D. S., David Herron, and Stasia Iwanicki. The Business Value of IT: Managing Risks, 

Optimizing Performance, and Measuring Results. Boca Raton, Fla.: Auerbach Publications, 2008

Haugan, Gregory T. The Work Breakdown Structure in Government Contracting. Vienna, Va.: Management 

Concepts, 2008.

———. Work Breakdown Structures for Projects, Programs, and Enterprises. Vienna, Va.: Management 

Concepts, 2008.

HHS (Department of Health and Human Services). DHHS Project Officers’ Contracting Handbook. 

Washington, D.C.: Jan. 23, 2003.

Higdon, Greg, Lew Fichter, and Alfred Smith. Cost Uncertainty Analysis: Observations from the Field. Santa 

Barbara, Calif.: Tecolote Research, April 2008.

Hohmann, Timothy J. Estimating Software Size: Impact and Methodologies. El Segundo, Calif.: Galorath 

Associates, 1997. 

Hulett, David T. Advanced Quantitative Schedule Risk Analysis. Washington, D.C.: Hulett and Associates, 

2007.

———. Considering Corporate Culture in Project Risk Management Maturity Models. Washington, D.C.: 

Hulett and Associates, 2007.

———. Corporate Culture in Project Risk Management Maturity. Washington, D.C.: Hulett and Associates, 

2008.

———. Integrated Cost and Schedule Risk Analysis. Washington, D.C.: Hulett and Associates, 2007.

———. Integrated Cost and Schedule Risk Analysis Using Risk Drivers, Washington, D.C.: Hulett and 

Associates, 2007.

———. Integrating Cost and Schedule Risk Analysis. Washington, D.C.: Hulett and Associates, n.d.

———. The Problem with Dangling Activities. Washington, D.C.: Hulett and Associates, 2007.
414 References GAO-09-3SP

Hulett, David T. Using Risk Analysis and Monte Carlo Simulation Techniques in Strategic Decisionmaking. 

Washington, D.C.: Hulett and Associates, May 2–4, 2005.

———, and Waylon Whitehead. Using the Risk Register in Schedule Risk Analysis with Monte Carlo 

Simulation. Washington, D.C.: Hulett and Associates, 2007.

Humphreys, Gary, and Margo Visitacion. Mastering Program Management Fundamentals Critical for 

Successful EVM. Bala Cynwyd, Pa.: Primavera, n.d. 

Ignemi, Joseph. Jumpstart Your Val ITTM Process. Mount Laurel, N.J.: PRICE Systems, Aug. 6, 2008.

Intelligence Community Cost Analysis Improvement Group. Independent Cost Analysis Checklist, version 1.0. 

Washington, D.C.: Dec. 16, 1999.

International Society of Parametric Analysts. Parametric Estimating Handbook, 4th ed. Vienna, Va.: 2008. 

Johnson, Jim, and others. “Collaboration: Development and Management—Collaborating on Project 

Success.” Software Magazine, Sponsored Supplement, February–March 2001.

Jones, Capers. What Are Function Points? Hendersonville, N.C.: Software Productivity Research, March 

1995.

Kahneman, Daniel, and Dan Lovallo. “Delusions of Success: How Optimism Undermines Executives’ 

Decisions.” Harvard Business Review, July 2003.

Kratzert, Keith. Earned Value Management (EVM): The Federal Aviation Administration (FAA) Program 

Manager’s Flight Plan. Washington, D.C.: Federal Aviation Administration, January 2006.

Kumley, Alissa C., and others. Integrating Risk Management and Earned Value Management: A Statistical 

Analysis of Survey Results. [Arlington, Va.]: National Defense Industrial Association, Program Management 

Systems Committee, n.d.

Lavdas, Evaggelos. Identifying the Characteristics of a Good Cost Estimate: A Survey. Cranfield, Bedfordshire, 

Eng.: Cranfield University, June 2006. 

Lipke, Walter H. Independent Estimate at Completion: Another Method. Tinker Air Force Base, Okla.: 

October 2004.

Lorell, Mark A., Julia F. Lowell, and Obaid Younossi. Evolutionary Acquisition Implementation Challenges for 

Defense Space Programs. Arlington, Va.: RAND, 2006. 

McRitchie, K., and S. Acelar. “A Structured Framework for Estimating IT Projects and IT Support.” 

Industry Hills, Calif.: June 2008. 

Majerowicz, Walt. “Schedule Analysis Techniques.” Integrated Performance Management Conference, n.p., 

November 2005.

Manzer, Frederick. Integrated Baseline Review: The Link from the Estimate to Project Execution. Vienna, Va.: 

Center for Systems Management, June 2005.
GAO-09-3SP References 415

Martin, Kevin. Integrating Schedules and EV Metrics. KM Systems Group, Arlington, Va.: May 2008.

Minkiewicz, Arlene F. Estimating Software from Requirements. Mount Laurel, N.J.: PRICE Systems, n.d.

 ———. Measuring Object-Oriented Software with Predictive Object Points. Mount Laurel, N.J.: PRICE 

Systems, n.d.

NASA (National Aeronautics and Space Administration). CADRe (Cost Analysis Data Requirement). 

Washington, D.C.: Mar. 22, 2005.

———. Cost Estimating Handbook. Washington, D.C.: 2004.

———. NASA Procedural Requirements, NPR 7120.5C. Washington, D.C.: Mar. 22, 2005.

———, Jet Propulsion Lab. Handbook for Software Cost Estimation. Pasadena, Calif.: May 30, 2003.

NAVAIR (Naval Air Systems Command). AIR 4.2 Life Cycle Cost Estimating Process. Washington, D.C.: 

Oct. 1, 1999.

———. Integrated Project Management Training. Washington, D.C.: January 2004.

———. Methods and Models for Life-Cycle Cost Analysis in the U.S. Department of Defense. Washington, 

D.C.: May 26, 2004.

———. NAVAIR Acquisition Guide. Washington, D.C.: April 2005.

———. Using Software Metrics and Measurements for Earned Value Toolkit. Washington, D.C.: October 

2004.

NAVSEA (Naval Sea Systems Command). Cost Estimating Handbook. Washington, D.C.: 2005.

NCAD (Navy Cost Analysis Division). Cost Analysis 101. Arlington, Va.: n.d.

———. Documentation Guide. Washington, D.C.: Feb. 3, 2004.

———. Methods and Models for Life Cycle Cost Analysis in the U.S. Department of Defense. Arlington, Va.: 

May 25–26, 2004.

NCCA (United States Naval Center for Cost Analysis). Software Development Estimating Handbook. 

Arlington, Va.: February 1998. 

NDIA (National Defense Industrial Association). Integrating Risk Management with Earned Value 

Management. Arlington, Va.: June 2004.

———. Program Management Systems Committee ANSI/EIA-748A Standard for Earned Value Management 

Systems Intent Guide. Arlington, Va.: January 2005.

———. Program Management Systems Committee Earned Value Management System Acceptance Guide. 

Arlington, Va.: November 2006.
416 References GAO-09-3SP

NDIA (National Defense Industrial Association). Program Management Systems Committee Earned Value 

Management System Application Guide. Arlington, Va.: March 2007.

———. Program Management Systems Committee Surveillance Guide. Arlington, Va.: October 2004.

NIH (National Institutes of Health). Cost Benefit Analysis Guide for NIH Information Technology (IT) Projects. 

Bethesda, Md.: April 1999.

NIST (National Institute of Standards and Technology). Engineering Statistics Handbook. Washington, D.C.: 

Department of Commerce, Technology Administration, July 18, 2006.

OMB (Office of Management and Budget). Capital Programming Guide: Supplement to Circular A-11, Part 1, 

Overview of the Budget Process. Washington, D.C.: Executive Office of the President, June 2006.

———. Capital Programming Guide: Supplement to Circular A-11, Part 7, Preparation, Submission, and 

Execution of the Budget, Version 1.0. Washington, D.C.: Executive Office of the President, June 1997. 

———. Capital Programming Guide: Supplement to Circular A-11, Part 7, Preparation, Submission, and 

Execution of the Budget, Version 2.0. Washington, D.C.: Executive Office of the President, June 2006. 

———. “Developing and Managing the Acquisition Workforce,” Office of Federal Procurement Policy 

Letter 05-01, Washington, D.C., April 15, 2005.

———. FY2007 Exhibit 300 Preparation Checklist. Washington, D.C.: Mar. 8, 2005.

———. Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Circular No. A-94 

Revised. Washington, D.C.: Oct. 29, 1992.

———. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information 

Disseminated by Federal Agencies; Notice; Republication. Part IX. Washington, D.C.: Feb. 22, 2002.

———. “Improving Information Technology Project Planning and Execution,” memorandum for Chief 

Information Officers M 05-23, Washington, D.C., August 4, 2005.

———. Major Systems Acquisitions, Circular A-109. Washington, D.C.: Apr. 5, 1976.

———. Management of Federal Information Resources, Circular A-130. Washington, D.C.: 2000.

———. The President’s Management Agenda. Washington, D.C.: 2002.

Park, Robert E. Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software 

Organizations. Pittsburgh, Pa.: Software Engineering Institute, Carnegie Mellon University, 1995.

Phelan, Pat. Estimating Time and Cost of Enterprise Resource Planning (ERP) Implementation Projects Is a 

10-Step Process. Stamford, Conn.: Gartner Inc., Apr. 27, 2006. 

Performance Mangement–Civilian Agency/Industry Working Group. Civilian Agency Reciprocity for 

Contractors EVMS Acceptance. n.p.: May 29, 2008.

PMI (Project Management Institute). Practice Standard for Work Breakdown Structures, 2nd ed. Newton 

Square, Pa.: 2006.
GAO-09-3SP References 417

Price, Rick. The Missing Link: Schedule Margin Management. n.p.: n.p., May, 2008. 

PRICE Systems. The Promise of Enterprise Resource Planning. Mount Laurel, N.J.: 2004

RAND. Impossible Certainty: Cost Risk Analysis for Air Force Systems. Arlington, Va.: 2006.

———. Toward a Cost Risk Estimating Policy. Williamsburg, Va.: Feb. 17, 2005.

Reifer, Donald J. Poor Man’s Guide to Estimating Software Costs. Torrance, Calif.: Reifer Consultants, June 

2005.

Remez, Shereen G., and Daryl W. White. Return on Investment (ROI) and the Value Puzzle. Washington, 

D.C.: Capital Planning and Information Technology Investment Committee, Federal Chief Information 

Officer Council, April 1999. 

Scaparro, John. NAVAIR Integration Project Management Brief to U.S. Government Accountability Office. Pax 

River, Md.: Naval Air Systems Command, Apr. 19, 2007. 

SCEA (Society of Cost Estimating and Analysis). Cost Programmed Review of Fun Economic Analysis—How 

to Choose Between Investment Options. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Basic Data Analysis Principles—What to Do 

Once You Get the Data. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Cost Estimating Basics—Why Cost Estimating 

and an Overview of How. Vienna, Va.: 2003. 

———. Cost Programmed Review of Fundamentals (CostPROF): Cost Risk Analysis—How to Adjust Your 

Estimate to Reflect Historical Cost Growth. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Costing Techniques—The Basic Types of Cost 

Estimates. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Data Collection and Normalization—How 

to Get the Data and Ready It for Analysis. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Earned Value Management Systems 

(EVMS)—Tracking Cost and Schedule Performance on Projects. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Index Numbers and Inflation—How to 

Adjust for the General Rise in Prices over Time. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Manufacturing Cost Estimating—Techniques 

for Estimating in a Manufacturing Environment. Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Parametric Estimating—From Data to Cost 

Estimating Relationships (CERs). Vienna, Va.: 2003.

SCEA (Society of Cost Estimating and Analysis). Cost Programmed Review of Fundamentals (CostPROF): 

Probability and Statistics—Mathematical Underpinnings of Cost Estimating. Vienna, Va.: 2003.
418 References GAO-09-3SP

———. Cost Programmed Review of Fundamentals (CostPROF): Regression Analysis—How to Develop and 

Assess a Cost Estimating Relationship (CER). Vienna, Va.: 2003.

———. Cost Programmed Review of Fundamentals (CostPROF): Software Cost Estimating—Techniques for 

Estimating in a Software Environment. Vienna, Va.: 2003.

———. Glossary of Terms. Vienna, Va.: n.d.

Seaver, David P. Estimates for Enterprise Resource Planning Systems, Mount Laurel, N.J.: PRICE Systems n.d.

Slater, Derek. “Enterprise Resource Planning (ERP) Software Packages Promise Great Benefits, but Exactly 

How Much Will You Have to Pay to Get Them?” CIO Enterprise Magazine, January 15, 1998, p. 22.

Smith, Alfred. Examination of Functional Correlation. Santa Barbara, Calif.: Tecolote Research, June 2007.

———, and Shu-Ping Hu. Common Errors When Using Risk Simulation Tools. Santa Barbara, Calif.: 

Tecolote Research. June 2005.

———. Cost Risk Analysis Made Simple. Santa Barbara, Calif.: Tecolote Research, September 2004.

———. Impact of Correlating CER Risk Distributions on a Realistic Cost Model. Santa Barbara, Calif.: 

Tecolote Research. June 2003.

Summerville, Jessica R. Improvement Curves: How to Account for Cost Improvement. Chantilly, Va.: Northrop 

Grumman, The Analytical Sciences Corporation, 2005.

United States Army. Cost Guide Manual. Washington, D.C.: May 2002.

———, Environmental Center. Methodology for Developing Environmental Quality Requirements for a Cost 

Analysis Requirements Description. Aberdeen, Md.: November 2001.

USAF (United States Air Force). Cost Risk and Uncertainty Analysis Handbook. Goleta, Calif.: Tecolote 

Research, Inc., April 2007.

———. Air Force Systems Command (AFSC) Cost Estimating Handbook. Reading, Mass.: 1987.

———. Condensed GSAM Handbook Checklists. Hill Air Force Base, Utah: February 2003.

———. Cost Analysis Guidance and Procedures, Air Force Instruction 65-508. Washington, D.C.: Oct. 1, 1997.

———. Cost Risk Analysis: Need-to-Know for Evaluations and Budgetary Decisions. Arlington, Va.: 

Government-Industry Space Council, June 2008.

———. Costing ERP in the Department of Defense. Washington, D.C.: June 2008.

———. CrossTalk. Hill Air Force Base, Utah: June–December 2002.

———. Economic Analysis, Air Force Instruction 65-506. Washington, D.C.: Nov. 10, 2004.

———. Forecasting the Future with EVMS. Arlington, Va.: n.d.
GAO-09-3SP References 419

———. Guidelines for Successful Acquisition and Management of Software Intensive Systems. Hill Air Force 

Base, Utah: February 2003.

———. Myths of EVM. Arlington, Va.: May 11, 2005. 

———, Air Force Cost Analysis Agency. Cost Risk Analysis Handbook. Arlington, Va.: October 2006.

———, Air Force Cost Analysis Agency. Decision Support for the Program Manager. Arlington, Va.: n.d.

———, Air Force Cost Analysis Agency. Pre-Award IBRs. Arlington, Va.: Sept. 7, 2006.

———, Air Force Cost Analysis Agency. Raytheon Missile System EVMS Validation Review. Arlington, Va.: 

Apr. 18, 2008.

———, Air Force Cost Analysis Agency. Risk and Estimate Quality Issues. Arlington, Va.: n.d.

VA (Veterans Administration). Individual VAMC Pricing Guides by VISN. Washington, D.C.: July 2007

Visitacion, Margo. Debunking Commonly Held EVM Myths. n.p.: n.p., September 2007. 

Wideman, Max R. “A Pragmatic Approach to Using Resource Loading, Production, and Learning Curves 

on Construction Projects.” Canadian Journal of Civil Engineering 21 (1994): 939–53.

Wiley, David. Software Cost Estimating: Techniques for Estimating in a Software Development Environment. 

Chantilly, Va.: Northrop Grumman and Analytical Sciences Corp., 2005.

Wilkens, Tammo T. Earned Value Clear and Simple. Bala Cynwyd, Pa.: Primavera Systems, Apr. 1, 1999.

Wright, R., J. Comer, and Justin Morris. Enhancing Remediation Project Credibility with Defensible and 

Documented Cost Estimates. n.p.: n.d.

Zubrow, Dave. Earned Value Management (EVM): Basic Concepts. Pittsburgh, Pa.: Carnegie Mellon Software 

Engineering Institute, 2002.

———. Implementing Earned Value Management (EVM) to Manage Program Risk. Pittsburgh, Pa.: Carnegie 

Mellon Software Engineering Institute, 2002.
420 References GAO-09-3SP

Image Sources

This section lists credit and copyright information for images and graphics on the front cover.

Front cover clockwise: 

PhotoDisc (Treasury)

NASA (Space Shuttle)

DOD (U.S. Navy ship)

Corbis (Glen Canyon Dam)

Digital Vision (Expressway)

Eyewire (White House)

GAO (capitol)
The Government Accountability Office, the audit, 

evaluation, and investigative arm of Congress, exists 

to support Congress in meeting its constitutional 

responsibilities and to help improve the performance 

and accountability of the federal government for the 

American people. GAO examines the use of public 

funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and 

other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its 

core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO 

documents at no cost is through GAO’s Web site 

(www.gao.gov). Each weekday afternoon, GAO posts 

on its Web site newly released reports, testimony, and 

correspondence. To have GAO e‐mail you a list of 

newly posted products, go to www.gao.gov and select 

“E-mail Updates.”

The printed version of the GAO Cost Estimating 

and Assessment Guide: Best Practices for 

Developing and Managing Capital Program Costs

is available through the Government Printing Office

at bookstore.gpo.gov or by calling (202) 512-1800 or 

(866) 512-1800 toll free.

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or 

(202) 512-7470

Ralph Dawn, Managing Director, dawnr@gao.gov, 

(202) 512-4400

U.S. Government Accountability Office 

441 G Street NW, Room 7125

Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, 

(202) 512-4800

U.S. Government Accountability Office 

441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Ordering Printed Copies of 

the GAO Cost Estimating 

and Assessment Guide

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

Please Print on Recycled Paper.
http://www.gao.gov/products/GAO-09-3SP