INDEPENDENT COST ASS E S S M E N T O F T H E

SPACE LAUNCH SYSTEM, MULTI - PURPOSE 

CREW VEHICLE AND 21 S T CENTURY GROUND 

SYSTEMS PROGRAMS

E X E C U T I V E S U M M A R Y O F F I N A L R E P O R T

A U G U S T 1 9 T H , 2 0 1 1

Booz | Allen | Hamilton This document is confidential and is intended solely for the 

use and information of the client to whom it is addressed.
Booz | Allen | Hamilton

This page is intentionally left blank.
iii Booz | Allen | Hamilton

Executive Summary 

Task Overview

NASA engaged Booz Allen Hamilton to perform independent assessments of cost and schedule 

estimates (hereafter referred to as Independent Cost Assessments, or ICAs) developed by the Space 

Launch System (SLS), Multi-Purpose Crew Vehicle (MPCV), and 21st Century Ground System (21CGS)

Programs, and to assess the sufficiency of reserves contained in the estimates. It is important to note 

that an ICA is not a cost estimate, but rather, it is an assessment of existing cost estimates and the 

documentation and practices used to generate them.

The estimates of this assessment were generated by the three Programs as part of their response to an 

Analysis of Alternatives (AoA). The AoA was commissioned by the Human Exploration and Operations 

Directorate (HEO) to evaluate candidates for NASA’s post-Space Shuttle manned space flight 

architecture.

Figure 1: Program Project Milestones

Not all of the estimates have the same level of maturity due to each program’s respective lifecycle 

phase. Both 21CGS and SLS can be characterized as Pre-Phase A programs, whereas MPCV can be 

characterized as Phase B since it has held portions of its Preliminary Design Review (PDR) process, but 

has not completed it due the changes in the other parts of the architecture. Due to the numerous 

alternatives considered as part of the AoA which began in March, and a final determination of the 

Planning, Programming, Budgeting & Execution (PPBE) architecture in June, each Program was limited in 

the time they had available to develop their estimates. In many cases, the documentation was being 

created in parallel with the ICA review and was updated from previous estimates due to time 

constraints. The Booz Allen ICA Team (subsequently referred to as the ICA Team) worked as closely as 

possible with program personnel to obtain and evaluate all appropriate documentation. Since the AoA 

estimates form the basis of NASA’s long-term PPBE submission they have been reviewed according to 

standards typically associated with full life-cycle cost estimates.

Human Space 

Flight Project 

Events and 

Reviews

Implementation

Phase A: Concept & Technology 

Development Phase C Phase D Phase E Phase F

Baseline 

Project Plan

Formulation NASA Life 

Cycle Phases

Project Life 

Cycle Phases

SRR SDR (PNAR) PDR

(NAR)

Approval

Phase B: Prelim Design & Tech 

Completion 

Pre-Phase A: Concept Studies 

FAD MCR
iv Booz | Allen | Hamilton

Key Findings

In general, the estimates prepared by SLS, MPCV, and 21CGS are consistent with Analysis of Alternative 

(AoA) level estimates and are reasonable point estimates for budget planning in the near-term 3-5 year 

budget horizon. They are serviceable in that they represent the basis to build upon for future life-cycle 

cost estimates of the quality required for long-term budget formulation and the development of 

program baselines. None of the estimates reviewed by the ICA Team support establishment of longterm budgets or detailed baselines consistent with NPR 7120.5 requirements. They are, however, 

reasonable AoA estimates appropriate for supporting trade studies and comparative analyses. All three 

Program estimates assume large, unsubstantiated, future cost efficiencies leading to the impression that 

they are optimistic. A scenario-based risk assessment, which excludes cost estimating uncertainty and 

unknown-unknown risks (historically major sources of cost and schedule growth), reveals all three 

Programs’ reserves are insufficient.

Below are the seven key findings common to all three programs:

Finding: The Programs’ estimates are serviceable and can be used for near-term budget planning in the 

current 3- to 5-year budget horizon. Beyond this horizon, the inclusion of large expected cost savings in 

the estimates, the beginning of development activities, and the potential for significant risk events 

decreases the ICA Team’s confidence in the estimates.

Recommendation: The ICA Team recommends that NASA initiate full Program life-cycle cost estimates 

(LCCEs) for SLS, MPCV, and 21CGS immediately upon approval of SLS architecture. Further, the ICA 

Team recommends that an Independent Cost Estimate be conducted at the next program milestone as a 

cross-check to the Program estimates. 

Finding: The BOEs provided by the Programs are not fully traceable or documented. An independent 

organization could not replicate Program estimates using the data sets provided by SLS, MPCV, or 21CGS 

without additional explanation from Program staff.

Recommendation: The ICA Team recommends that HEO/ESD use the occasion of a selection of a new 

SLS architecture to establish a common practice across Programs for generating cost and schedule 

estimates; establish documentation standards for BOEs; and create and disseminate BOE, cost, and 

schedule estimate templates to Programs.

Finding: There are many instances of unjustified cost reductions in the Program estimates. This exposes 

the Programs to cost risk and undermines the credibility of the estimate. Cost reductions were generally 

observed in either of two categories: scope reductions where the removed work will likely be required, 

or the application of anticipated efficiencies (production, competition, etc.) that NASA has not 

historically achieved. In some cases the efficiencies leading to cost reductions are not explicitly 

identified. Both of these categories lower the estimates below what historical data suggests and indicate 

that the estimates are optimistic.

Recommendation: The ICA Team recommends HEO/Exploration Systems Development (ESD) require 

that adjustments exceeding +/- 5% from historical data provide written justifications for such 

adjustments and take countervailing reserve positions for items where justifications are not sufficient. 
v Booz | Allen | Hamilton

Where the source of cost reductions are not identified, NASA should describe the rationale for such 

reductions and develop a management plan to increase the probability that NASA achieves these 

efficiencies.

Finding: Programs lack sufficient reserves to cover their Protect Scenarios. Program reserves were 

applied independent of the level of cost and schedule risk. Quantitative risk/sensitivity analyses have 

not been performed on any estimates.

Recommendation: The ICA Team recommends quantitative risk/sensitivity analyses be performed on all 

three Programs in conjunction with their full-scale life-cycle cost estimates (recommended above) and 

that reserve positions be established based on risk analysis findings.

Finding: There is little observed evidence of formal cross-Program coordination or integrated schedule 

development, although the Programs do interact on a working level. Lack of formal integration among 

the Programs diminishes each program manager’s ability to manage cost and schedule risk proactively.

Recommendation: The ICA Team recommends HEO/ESD enhance PMO functionality by assigning a core 

group of dedicated staff, working independent of the individual Programs, to coordinate and integrate 

schedules, milestones, technical reviews, and cross-program risks.

Finding: Individual Program cost and schedule estimates are managed separately and are not in 

alignment. Lack of cost and schedule alignment makes it difficult to estimate the cost impact of schedule 

slips and vice-versa. Non-integrated cost and schedule estimates are sufficient for trade studies, but do 

not facilitate ongoing baseline management.

Recommendation: The ICA Team recommends that HEO/ESD establish a common WBS/CES as a 

foundation for future integration of cost and schedule.

Finding: Program estimates were shaped to fit within an anticipated budget profile. While this is 

common practice, it is not consistent with GAO cost estimating best practices. Shifting costs to later 

years to fit upcoming budget caps decreases each Program’s availability of funds for risk reduction, 

technology maturation, and exposes the Programs to out-year cost growth.

Recommendation: The ICA Team recommends that time-phased results from the full LCCE be used to 

determine whether or not the AoA estimates, phased to fit into a budget profile, are executable. If not, 

the ICA Team recommends Programs identify specific plans to move work between years to fit into the 

budget cap.
vi Booz | Allen | Hamilton

Methodology

The ICA Team focused on assessing the quality of individual Bases of Estimates (BOEs). BOEs are 

documents describing the justification and rationale for a cost estimate. In a step away from doctrine, 

the technical lead allowed information collected in interviews with Programs to be considered when 

written documentation was not available. As discussed in the findings, this is due to the fact that the 

documentation required to substantiate the estimates was not always available or sufficient. This 

further investigation was necessitated by the fact that the ICA Team believes NASA’s intent is to 

understand the quality of the estimates, more so than the quality of their documentation. The results of 

the assessments and a full description of the rating system used are contained in the main body of this 

report. 

The ICA Team also recognized that summary-level observations could subsequently be provided for each 

Program. To facilitate the characterization of such observations, the team referenced the GAO’s four 

properties of quality cost estimates (U.S. Government Accountability Office, 2009). These are:

Well-Documented: By well documented is meant that an estimate is thoroughly documented, 

including source data and significance, clearly detailed calculations and results, and explanations 

of why particular methods and references were chosen. Data can be traced to their source 

documents.

Comprehensive: An estimate is comprehensive if it has enough detail to ensure that cost 

elements are neither omitted nor double counted. All cost-influencing ground rules and 

assumptions are detailed in the estimate’s documentation.

Accurate: An estimate that is accurate is unbiased, the work is not overly conservative or overly 

optimistic, and is based on an assessment of most likely costs. It has few, if any, mathematical 

mistakes; its mistakes are minor.

Credible: As for credibility, any limitations of the analysis because of uncertainty or bias 

surrounding data or assumptions are discussed. Major assumptions are varied, and other 

outcomes are recomputed to determine how sensitive they are to changes in the assumptions. 

Risk and uncertainty analysis is performed to determine the level of risk associated with the 

estimate.

Additionally, the team leveraged the NASA Schedule Management Handbook (NASA, 2011) to establish 

two rating criteria for quality schedule estimates. These are:

Traceable: Schedule estimates are traceable when they are based on well-founded and/or 

previously accepted schedule information of established programs with comparable scope. 

Strong estimates will be derived from schedule information developed using industry best 

practices and standards. Lastly, traceable estimates should include adjustments and 

justifications for variations from reference material.

Executable: Estimates are executable when they are founded on clearly defined work packages 

consistent with progress within the program lifecycle. Valid estimates will contain realistic 
vii Booz | Allen | Hamilton

schedule durations based on projected work hours and program deliverables. Executable 

schedule estimates will also demonstrate reasonable integration with cost estimates and include 

considerations for programmatic risk.

The narrative of the report is structured to align with the above cost and schedule rating criteria. 

Furthermore, the ICA Team used Harvey Balls, as defined below, to standardize presentation of 

qualitative program-level observations based on these defined rating criteria. 

Table 1: Program Level Observation Reference

Cost Estimates

All All BOEs meet all criteria

Most Most BOEs meet criteria, or BOEs meet most criteria

Some Some BOEs meet the criteria, or BOEs meet some of the criteria

Few Few BOEs meet the criteria, or BOEs meet a few of the criteria

None No BOEs meet the criteria, or BOEs meet none of the criteria. 

Qualitative Observation Reference
viii Booz | Allen | Hamilton

SLS Cost and Schedule Assessment Summary

The ICA Team performed a thorough review of all BOEs, information, and data the SLS team provided in 

support of their PPBE submission. The ICA Team concludes that the estimate is acceptable to serve as 

the basis for near-term, 3-5 year, AoA and Program decisions, although there are several areas noted in 

this report where documentation, assumptions, and methodologies did not meet GAO standards. The 

estimate is not suitable for long-term budget planning or the development of a program baseline. The 

SLS cost estimate assumes several cost efficiencies that have not been realized on previous NASA 

programs. These efficiencies represent cost risk to the program as it is unclear whether they are realistic 

and leads to the impression that the estimate is optimistic. Local NASA in-house cost estimating 

resources were leveraged in production of some, but not all, of the SLS estimates.

Table 2: SLS Cost and Schedule Assessment Summary

Cost Estimates

Well-documented

BOEs meet some of the GAO criteria. Cost estimates and corresponding BOEs exist for all 

major cost elements with system descriptions, ground-rules and assumptions, as well as 

detailed diagrams, visual mock-ups, or pictures of major elements. However, most of the 

BOEs do not reveal the calculations performed or the estimating methodology and rationale

Comprehensive

BOEs meet all GAO criteria. Documentation and corresponding estimates account for all 

major development and production activities, as well as expected testing, insight and 

oversight

Accurate

BOEs meet most of the GAO criteria. The estimates are based on an assessment of the most 

likely costs, are adjusted for inflation, and are verified by cross checks. However, the review 

team is unable to verify the accuracy of several vendor cost suggestions or engineering build 

up estimates

Credible

BOEs meet few GAO criteria. SLS personnel took steps to enhance the credibility of estimates 

by providing cross-checks for various elements as a part of the documentation, but there was

no quantified sensitivity or risk analysis

Schedule Estimates

Traceable N/A

No viable schedule or corresponding BOE was available for the elements of the SLS 

architecture. The NASA Schedule Management Handbook does not require detailed 

schedules for Pre-phase A programs

Executable N/A No schedule was provided for evaluation

Cost Estimates

Schedule Estimates
ix Booz | Allen | Hamilton

MPCV Cost and Schedule Assessment Summary

The ICA Team performed a thorough review of all BOEs, information and data the MPCV team provided 

in support of their PPBE submission. MPCV uses an atypical form of extrapolation from actuals, in which 

costs are divided into fixed and variable costs and are phased to fit the schedules of each alternative. 

Typical methods of extrapolation from actuals, per the GAO Cost Estimating and Assessments Guide, 

include averages, learning curves, and estimates at completion from earned value management systems 

(EVMS), as opposed to fixed and variable cost breakouts. As such, BOEs were not developed in 

accordance with a standardized WBS. Phasing of costs using this fixed/variable methodology was driven 

by technical requirements and the need to meet annual cost targets. No breakout of fixed/variable costs 

into their component activities was provided and no detailed plan exists for how work can be re-ordered 

to fit variable costs into the constrained baseline. Without this plan it is questionable whether the 

current estimate is executable. The resulting assessment revealed an estimate appropriate to the task of 

analyzing alternatives but not of sufficient rigor for long term budgeting or baseline development. Local

NASA in-house cost estimating resources were not leveraged to develop the MPCV estimates.

Table 3: MPCV Cost and Schedule Assessment Summary

Cost Estimates

Well-documented

Few BOEs meet GAO criteria. Prime contractor costs from the Update and Verification (U&V) 

session are well documented, but the bases of estimates modifying those costs are not well 

documented

Comprehensive All BOEs meet GAO criteria. Costs comprising DDT&E, production, and upgrades are 

estimated, and those costs already incurred are documented

Accurate BOEs meet some of the GAO criteria. Not all estimates are grounded in documented 

assumptions. Inflation was incorporated consistently, but using out of date inflation tables

Credible

Some BOEs meet the GAO criteria. U&V data used to form the base of the Prime estimate i s

credible. Some modifications made to the Prime U&V data to establish the Start Point are 

not as credible, lacking assessment of risk and uncertainty and verification by independent 

estimates

Schedule Estimates

Traceable

Schedule BOEs were founded on a previously approved PDR schedule as well as work-hour 

estimates by the contractor, however EVA and MOD were added to the MPCV program after 

the initial estimate and not a part of the BOE assessment

Executable N/A Schedule not available for assessment

Cost Estimates

Schedule Estimates
x Booz | Allen | Hamilton

21CGS Cost and Schedule Assessment Summary

The ICA Team performed a thorough review of all BOEs, information and data that 21CGS provided in 

support of their PPBE submission. The ICA Team concludes that the estimate is reasonable and 

acceptable to serve as the basis for near-term, 3-5 year, AoA and Program decisions. Much of the 21CGS 

budget submission is based on in-depth design studies, although there are several areas noted in this 

report where documentation and assumptions did not meet GAO standards. The 21CGS PPBE 

submission assumes a reduction in cost in comparison to historical Shuttle and Constellation figures, 

primarily due to the limited launch manifest. Several technical risks exist that could have significant cost 

and schedule impacts. Local NASA in-house cost estimating resources developed the 21CGS estimate. 

Due to last minute changes in architecture and the corresponding late delivery of BOEs, 21CGS was the 

only Program with which the ICA Team was not able to perform an additional round of estimator 

interviews. This same Ground Operations team has a history of delivering on-cost and on-schedule and a 

portion of the ICA Team’s ratings reflects the lack of time to perform a second round of estimator 

interviews rather than the quality of the estimates themselves.

Table 4: 21CGS Cost and Schedule Assessment Summary

Cost Estimates

Well-documented

BOEs meet some of the GAO criteria. Cost estimates and corresponding BOEs exist for all 

major cost elements with system descriptions, ground-rules and assumptions, as well as 

detailed diagrams, visual mock-ups, or pictures of major elements. However, most of the 

BOEs do not reveal the calculations performed or the estimating methodology and rationale

Comprehensive

BOEs meet all GAO criteria. Documentation and corresponding estimates account for all major 

development and production activities, as well as expected testing, insight and oversight.

Accurate

BOEs meet all GAO criteria. The estimates accounted for inflation, and were verified by cross 

checks. The program has a historical record of cost accuracy

Credible

BOEs meet few GAO criteria. KSC personnel took steps to enhance the credibility of estimates 

by providing cross-checks for various elements as a part of the documentation, but did not 

perform quantified sensitivity or risk analysis.

Schedule Estimates

Traceable N/A

No basis for schedule information was provided for 21CGS. A high-level management planning 

calendar (Power Point) was provided, however it did not contain any information regarding 

work breakdown on program milestones. Advanced schedule materials are not required for 

Pre-Phase A programs. The NASA Schedule Management Handbook does not require detailed 

schedules for Pre-phase A programs.

Executable N/A Schedule not available for estimate.

Cost Estimates

Schedule Estimates
xi Booz | Allen | Hamilton

Reserve Sufficiency Assessment

To perform a sufficiency check on the Program reserves the ICA Team applied the Scenario Based 

Method (SBM) (Paul Garvey, The MITRE Corporation, 2005) for risk analysis. The SBM is one of three 

primary risk analysis methods recognized by the cost community (AFCCA, 2007) and cited within the

NASA Cost Estimating Handbook (NASA, 2008). SBM, while subjective, provides each program with a list 

of potential cost and schedule risks against which they can develop actionable mitigation plans. 

Additionally, it provides each program a quick sufficiency check of their reserves in the absence of a 

detailed, quantified risk analysis.

To conduct the reserve assessment, cost/schedule risks were identified by the ICA Team and were 

added to technical and programmatic risks determined by Booz Allen SMEs to develop an independent 

risk list for each Program. From this, a subset of risks, chosen in proportion to their likelihood of 

occurrence, was selected by the ICA Team and Booz Allen SMEs to create the “Protect Scenario” (PS) for 

each program. A Protect Scenario represents a set of potential risks a program should reasonably be 

able to cover with its reserves. The combined cost impact, by year, of risks contained in the PS was then 

compared to the Program reserves. The Protect Scenario is not intended to be comprehensive, a worstcase scenario, or the definitive list of highest impact risks nor does it include estimating uncertainty or 

unknown-unknowns, both of which have historically been major sources of cost and schedule growth on 

programs. For these reasons this assessment is only useful for conducting a spot-check of the sufficiency 

of Program reserves. It is not appropriate for use in determining appropriate program reserve levels. 

Since the protect scenarios exclude estimating uncertainty and unknown-unknown risks, which 

history indicates are major sources of cost and schedule growth on programs, the reserve 

sufficiency assessment described in this report cannot be used to determine appropriate 

Program reserve levels. Until a quantified risk analysis has been performed on each estimate the 

ICA Team can make no recommendation of appropriate Program reserve levels
xii Booz | Allen | Hamilton

Conclusion

The cost estimates prepared by the SLS, MPCV, and 21CGS Programs are consistent with pre-concept, 

AoA-phase estimates and thus are not suitable for long-term budget formulation or the development of 

Program baselines. NASA should treat the estimates as serviceable point estimates for budget planning 

in the near-term 3-5 year budget horizon as they represent the basis upon which future estimates can 

be constructed. Due to unjustified, sometimes substantial, assumed future cost savings; the ICA Team 

views each Program’s estimate as optimistic. Reserve levels were not based on a quantitative risk 

analysis and do not cover each Program’s Protect Scenario. Furthermore, each Protect Scenario excludes

estimating uncertainty and unknown-unknown risks, which history indicates are major sources of cost 

growth on programs. Due to procurement of items still in development and large cost risks in the out 

years, NASA cannot have full confidence in the estimates for long-term planning.

Effective application of program management best practices will help the human space flight concept 

remain viable beyond the near-term budget window. NASA should endeavor to understand the true 

lifecycle cost of each program, work to integrate the programs through a master schedule, and ensure 

that documentation practices improve to facilitate future cost. Upon selection of a new architecture, 

NASA should perform a full scale life cycle cost estimate, complete with a quantitative risk analysis, to 

ensure each program has a long-term budget appropriate to its mission. This estimate should be 

validated with an independent cost estimate per NASA policy and GAO best practices. 

There is little evidence of formal inter-program coordination of cost, schedule and risk. Entities that 

used to be individual Projects in a tightly-coupled Program are now each stand-alone Programs. NASA 

should establish and enforce standards for documenting, integrating, and reporting on cost, schedule, 

and risk at the Mission Directorate or Division level (HEO or ESD). 

The SLS, MPCV and 21CGS estimates are serviceable as they represent the foundation for a future lifecycle cost estimate of the quality necessary for long-term budget development and program planning. 

The ICA Team believes that the recommendations contained within in this report and its appendices 

provide NASA an actionable framework to produce the quality estimates required to ensure the financial 

success of NASA’s next generation of human space exploration programs.