# Presentation: 356782

## NASA Cost Estimating Initiatives/Activities

**NASA Cost Estimating Initiatives/Activities**

- Joe Hamaker
- NASA HQ Cost Analysis Division
- March 2004

- NASA Headquarters Cost Analysis Division

## Thank You!

**Thanks to Richard Whitlock and his staff, and to Rey Carpio for organizing this year’s symposium**
**Thanks to Gigi Hackford and Awards Committee establishing Rutledge Award and selecting this year’s winner**
**And thanks to all y’all* for coming too!**

********All y’all*****:  *****y’all*** **for larger groups**

- NASA Headquarters Cost Analysis Division

## Message From Steve Isakowitz

**Degradation of cost function during the 1990s has turned around**
- Administrator is very cost conscious
- Tom Young Committee and ISS ICE clear evidence of renewed commitment  to quality estimating
- Congress, GAO, IG are calling for better cost estimating and overall financial credibility)
- We are being asked to execute on our promises of performance, schedule and budget with the latter perceived as our greatest challenge
- Changes in place reflect our commitment to improve
- Code BC, IPAO, Code D, IFMP, full cost, etc.
**Now we have the President’s vision for exploration**
- Cost played a huge role in the deliberations leading to this vision--“sand chart” has been key to the arguments
- Underlying assumptions assumed no heroic miracles
**Teams are hard at work establishing requirements and top level architectures**
- Code T spinning up
- The challenge for the cost community is to cost these architecture elements and constructively critique any innovative business practices for real substance
**To do our job we need good cost estimating at all levels from the Centers, through the IPAO to HQ**
- Independent multi level review is healthy (like peer reviews in science)
- Cost function needs to fully participate in systems engineering of these architectures using CAIV and other approaches
- While staunchly maintaining real credibility in the cost estimates
**Cost played a huge role in the formulation of the vision and will play an even larger role in the implementation**

- NASA Headquarters Cost Analysis Division

## Status of Cost Analysis Division at NASA HQ	

**Cost function/presence firmly re-established at HQ **
- Actively participating in other mainline activities (next chart)
**Code BC **
- 4 FTEs on board in Code BC
- 1 rotational on board through May
- 1 rotational coming on board in June
- Job announcement forthcoming for 2 GS14/15 analysts

- NASA Headquarters Cost Analysis Division

## Current Cost Activities at NASA HQ

**Fielding questions relative to Exploration Initiative “sand chart”**
**Providing cost support to space architect activities**
**Providing Agency interface to the GAO as they complete  their report on NASA cost estimating**
**Working with BAH to finalize Cost Estimating Handbook**
**Working with new Exploration Systems Office (Code T)**
**Serving on Core and Blue Team/Red Team Reviews of requirements and top level architectures**
**Coordinating with EVM Focal Point Council**
**Coordinating with IFMP**
**Continued close interface with Office of Chief Engineer (Code D) **
**Completed “looking back” cost growth study**
**Cost estimating initiatives (balance of briefing)**

**EVM**

**Cost**

**IFMP**

**WBS**

- NASA Headquarters Cost Analysis Division

## Ongoing NASA Cost Estimating Initiatives

**Cost Analysis Data Requirement (CADR)**
_**O**_**ne **_**N**_**ASA **_**C**_**ost **_**E**_**stimating (ONCE) Database**
**Cost Readiness Levels**
**Cost Growth Study**
**Full cost estimating methods **
**Cost risk analysis methods and best practices**
**Cost Risk Feedback Continuum**
**NASA Inflation Index**
**Cost Estimating Handbook**
**NPR 7120.5C **
**Cost Estimator Career Guide**
**Process Based Cost Estimating**
**Improved Operations Cost Modeling**
**Use of COTs tools**

- NASA Headquarters Cost Analysis Division

## Ongoing NASA Cost Estimating Initiatives

**Cost Analysis Data Requirement (CADR)**
_**O**_**ne **_**N**_**ASA **_**C**_**ost **_**E**_**stimating (ONCE) Database**
**Cost Readiness Levels**
**Cost Growth Study**
**Full cost estimating methods **
**Cost risk analysis methods and best practices**
**Cost Risk Feedback Continuum**
**NASA Inflation Index**
**Cost Estimating Handbook**
**NPR 7120.5C **
**Cost Estimator Career Guide**
**Process Based Cost Estimating**
**Improved Operations Cost Modeling**
**Use of COTs tools**

**Joe Hamaker**

**David Graham + Panel + Sharon Winn**

**Jill Hardash**

**David Graham**

**Kelley Cyr**

**Dr. Chris Smart**

**Rich Greathouse, Dr. Martin Steele, Edgar Zapata et al**

**Galorath, Mainstay, Price**

- NASA Headquarters Cost Analysis Division

## Cost Analysis Data Requirement (CADR) 

## Cost estimating touches several DRs* in a typical flight hardware development project
There is considerable overlap in the cost data requirements
Especially the WBS, CARD and Life Cycle Cost, and Cost Report DRs
Also considerable integration needed between these DRs
One DR can integrate many of the cost requirements
“One stop shopping” for project cost data
Can serve PM as a valuable management tool

- XYZXYZ

- Risk Plan

- EMV

- Schedule

**Cost Data Report**

- Life Cycle Cost
- Estimate

**Cost estimating touches several DRs* in a typical flight hardware development project**
**There is considerable overlap in the cost data requirements**
**Especially the WBS, CARD and Life Cycle Cost, and Cost Report DRs**
**Also considerable integration needed between these DRs**
**One DR can integrate many of the cost requirements**
**“****One stop shopping” for project cost data**
**Can serve PM as a valuable management tool**

**Cost Analysis Data Requirement (CADR)**

- *NASA DRs = DOD CDRLs

- CARD

- WBS

- NASA Headquarters Cost Analysis Division

## Cost Analysis Data Requirement (CADR)

**Most similar to a CARD but ...**
- Provides _standardized WBS_ and WBS dictionary
- Includes not only cost estimating inputs _but cost estimates as well_ (as a separable appendix)
- Contractor cost reporting
- Project Office (full cost) cost reporting
- _Increased focus on KPPs_
- _Mandatory HQ DR on Cat 1 and high risk Cat 2 flight projects_
**The CADR must be a PM responsibility**
- Prime contractor can build most of it
- Cost shops at Centers can support by providing definitions/interpretations of what is being asked for
- “What is a SLOC?”
**CADR scope/focus**
- Basic and applied research
- Technology development
- Flight projects
- Operations
- Functional products & services

- CADR only intended as a DR
-  on these 7120.5C categories

- NASA Headquarters Cost Analysis Division

## CADR Contents

**CADR Contents**

**Standardized WBS structures and dictionaries to level 5 (major component level)**
- Earth science (bus/payload)
- Space science (bus/payload)
- Launch systems
- In-space transportation systems
**Cost Input Report**
- Data sheets for KPP cost parameters by WBS element (see next chart)
**Cost Estimate Report**
- Data sheets for cost estimates by WBS element
- Contractor cost reporting
- Project Office (full cost) cost reporting

- Cost Input Report

- Cost Estimate Report

- By WBS Element

- NASA Headquarters Cost Analysis Division

## CADR As A “Flight Data Recorder” For Management

**CADR As A “Flight Data Recorder” For Management**

**The inputs for parametric cost models include a comprehensive set of KPPs as well as programmatic, schedule and management metrics that describe a project **
- In the concise and precise language of parametric cost estimating
- Metrics are rigorously defined by the models
**CADRs are living documents **
- Snapshots at least annually
- Provides the PM with basis for updated EACs _(performed by Agency parametricians)_
- Independent of contractor’s opinion
- Provides the PM with predicted cost effects of remedial actions
- CADRs are “encased” into the One NASA Cost Estimating Data Base
- Automatic lessons learned and cost autopsy document

- Time

- SLOC Count

- NASA Headquarters Cost Analysis Division

## Cost Analysis Data Requirement (CADR)

**CADR status**
- CADR document being drafted; to be ready for review at April 6-7 CASG meeting
- Completed document to be vetted through HQ Enterprises for buy-in
**CADR is proposed as a RFP DR on Category I and high risk Category II flight projects**
- To be included in 7120.5C and the NASA Cost Estimating Handbook (CEH)
- CADR template downloadable from CEH website
- To be included in NASA FAR Supplement
- Heads up via Code H Procurement Information Circulars (PICs)
- To be reviewed at Acquisition Strategy Meetings (ASMs)

**NPR 7120.5C **
**&CEH**

**NASA FAR Sup**

**ASMs**

**PICs**

- NASA Headquarters Cost Analysis Division

## Cost Analysis Data Requirement (CADR)

**Possible extension of CADR **
- CADR could be tied to a USG cost model (NAFCOM, Price, Seer, other)
- As CADRs are submitted, the prime contractor and project office* would be required to process the CADR project description (the KPPs) through the provided cost model
- *With the help of Agency parametricians
- This would occur at end of Phase A, again at the end of Phase B/PDR
- At least annually thereafter
- _Working with the Agency parametricans, the project would be required to calibrate the provided model to match the prime contract/Project Office estimate_
- The contractor and Project Office would encourages to maintain their own indigenous estimating method
- Would provide exposure of what underlying assumptions have to be made to calibrate to the prime contractor/project office position on cost
- Up and down the WBS
- 80% code reuse?  Simpliest materials/tolerances? 25% confident schedule?   Truckloads of GFE? Lake Wobegone Team*?

- *Where all the children are above average

- NASA Headquarters Cost Analysis Division

## Once NASA Cost Engineering (ONCE) Data Base

## Once NASA Cost Engineering (ONCE) Data Base 

**Recommendation #6**
**Tools and Business Practices**
**Increase NASA-wide cross-collaboration through common tools and business practices.**
- Enhance cross-Agency collaboration by putting in place standard engineering and collaborative tools and databases, processes, and knowledge-sharing structures. These tools and methods need to enhance the ability to collaborate, reduce inefficiency, and create time and resources for collaboration.

***6d. Review Common Databases. ****Assess the current state of databases, such as the Life Sciences Data Archive, for their ability to span multiple NASA Centers. Recommend adoption of common databases as appropriate. Develop a plan to increase awareness of databases and to provide database-specific guidelines for use. *

***6f. Knowledge Management System****. Establish a formal Knowledge Management (KM) pilot project, in which KM communities of practice (COPs) can occur for a small set of engineering, research, and administrative specialized disciplines that span multiple NASA Centers—for example, thermal test engineers. Identify and train these COPs in KM principles of operation**.*

**From One NASA  Website**

## One NASA Cost Engineering (ONCE) Data Base

**A major tenet of One NASA is to share information and, where it makes sense, utilize common data bases**
**Since 1971, NASA has had the REDSTAR Agency wide cost data base and the corresponding NAFCOM data base of normalized data**
- Many other cost data bases exist in pockets across the Agency
- Often estimators are unaware of or have difficulty quickly obtaining data
- Data normalizations are not standard
**Some Center data is undoubtedly unique to that Center’s mission (e.g. presumably only KSC, MSFC and SSC are interested in rocket engines) **
- But there remain convincing reasons to share cost data
- “A battery is a battery” (i.e. not Center unique)
- Will serve to standardize arbitrary WBS usage and cost normalization processes across Centers
- Facilitates CoSTER data sharing with other USG’s active in space activities
- More data and more current data in the hands of estimators is always the direction of good

- NASA Headquarters Cost Analysis Division

**Some goals **
- A One NASA product, with data availability across Centers (and via CoSTER, across USG space organizations)
- Protect contractor proprietary data rights
- Protect NASA Center competitive issues (bids on AOs and NRAs)
- Support all Center cost methodologies and numerous models (NAFCOM, PRICE, SEER, other NASA Center Models)
- Use a common WBS and common set of cost normalization ground rules
- Which may be “re-mapped” to suit particular user needs
- Collect cost _and_ key performance parameters _and_ programmatic cost drivers in one integrated data base
- Utilize the new Code BC CADR as a data collection instrument

**One NASA Cost Engineering (ONCE) Data Base**

- NASA Headquarters Cost Analysis Division

## NASA Cost Data Base Improvements

**Some issues**
- Obtaining initial Center buy-in and long term continued support
- Protecting Center competitive stance in future AO and NRA competitions
- Devising a common data normalization procedure that is usable by all Centers
- Defining “cost data”
- Quotes versus “actuals”
- When to begin collection for a particular project (PDR? CDR? FRR?)
- Sorting out in-house versus prime contractor efforts
- How to book “wraps”?
- Are we collecting O&S costs?
- What data base software to use? What server? What is the role of our support contractors?

- NASA Headquarters Cost Analysis Division

## One NASA Cost Engineering (ONCE) Data Base Concept

**ONCE DB**

**Cost Shops at**
**NASA Centers**

**Database**
**Support**
**Contractor**

_**Historical**_** data**
**From existing Center data bases**
**MSFC – REDSTAR/NAFCOM**
**GSFC**
**JPL**
**etc.**
**Data for Completed Projects**
_**Future**_** data**
**Collected with CADR at major milestones (i.e. ATP, PDR, CDR, FRR, etc.), at least annually**

- NASA Headquarters Cost Analysis Division

## Once NASA Cost Engineering (ONCE) Data Base

**Concept**
**Data collected at WBS level 4/5**
**In early project phases, higher level OK**
**Data base resident on server at IPAO/LaRC**
**Data base update/access controlled by IPAO**
**NDAs required for non-Government users**
**WBS is a common framework for management and estimating**
**If data is to be useful (for improving estimating), a standard WBS must be implemented**

- NASA Headquarters Cost Analysis Division

## ONCE Database Level One Requirements

_**Data Attributes**_
**Compatible with NASA Full-Cost initiative**
**Include technical, schedule and cost data and narrative descriptions**
**Include cost as either $ or hours, or both**
**Data normalized prior to entry into database**
**Identify recurring and non-recurring costs**
**Identify actuals vs. estimated vs. budgeted costs**
**Record project data at periodic intervals (major milestones)**
**Identify O&S costs**
**Identify costs by project phase**
**Incorporate existing NAFCOM data**
**Incorporate cost-risk data**
**Incorporate remarks/comments**

_**Data Base Functionality**_
**Prototype database operational by end of CY2004; fully operational by mid-CY2005**
**Web-accessible, simultaneous multiple users**
**Secure access for authorized individuals (User ID and password)**
**Flexible data storage at all appropriate WBS levels**
**Database easily searched**
**Store and retrieve data by project or mission as well as equipment type**
**Allow data manipulation (e.g. change economic base year)**
**Allow for manual data entry (in addition to data derived from CADR)**
**Flexible, custom report formats**

- NASA Headquarters Cost Analysis Division

## NASA Cost Growth: A Look at Recent Performance

**Matt Schaffer**
**NASA HQ Cost Analysis Division**

## Analytic Approach

**Compared initial and final budget estimates of development costs for 45 recent projects and computed percent budget growth as a surrogate for cost growth.**
- Selected projects are predominantly Space Science projects, since these are the easiest projects for which one can develop a budget trace.
- “Recent” generally means missions flown since 1996.
- All projects were initiated in 1990 or later.
- Four-fifths of the projects were initiated in the last ten years.
- Data come mostly from historical NASA budget documents; other data come from other NASA archives and Internet documents.

- NASA Headquarters Cost Analysis Division

## Are the results truly cost growth?

**The analysis did not correct for content change.**
- Many NASA projects were and are able to constrain budget growth by eliminating science requirements.
- Thus, budget growth is not a true representation of cost growth.
- But adjusting for these changes is difficult without some knowledge of the details, and these details are not found in budget documents.
- A partial reconstruction of the projects history, through analysis of project documentation, would be necessary before one could estimate the necessary adjustments.
- Such project documentation can be difficult, if not impossible, to find, as Code BC discovered in an attempt to gather documentation from recently launched systems.
**These difficulties not withstanding, the following results are probably fairly representative of actual cost growth in recent NASA projects.**
**The analysis did not adjust for inflation.**
- Cost growth is usually calculated using same-year constant-dollar estimates and actuals.
- Budget documents do not provide constant-dollar estimates, and inflation assumptions change yearly, making suspect any attempts to adjust budget dollars to constant dollars *a posteriori*, especially the initial budget estimate.
**The study attempted to capture only implementation budgets.**
- In general, the budget data should be only for the implementation phase, although not all budget documents were clear on the content in budget lines.
- It is possible some growth represents an increase in content, e.g., including launch costs in development budgets.

- NASA Headquarters Cost Analysis Division

## Overall Statistics

**Average growth:**
- 36% arithmetic mean
- This statistic is the average of percent budget growth on individual projects.
- 45% dollar-weighted mean
- This statistic is the average of percent budget growth on individual projects weighted by final budget amount.
- Thus, some of the more expensive projects experienced considerable budget growth.
**Median growth:  26%**
**35 of 45 projects exceeded the initial budget estimate**
**Total growth:  28%**
- This statistic is the relative change from the total of the 45 initial budgets to the total of the 45 final budgets.

**Recent NASA cost growth is about 30%.**

- Schaffer study goes into much more detail

- NASA Headquarters Cost Analysis Division

## Full Cost Estimating

## The “Wethington Full Cost Factor” 

**Mike Wethington, on detail to Code BC, is developing a simplified approach for estimating the effect of full cost**
- Suitable for Pre-Phase A, Phase A, (maybe) early Phase B estimates
- Under the assumption that sometime in Phase B, definitive full cost services agreements will be negotiated between the project and the Centers providing full cost services
**Combining GRC (Sefcik/Parkey) model, internal Code B data, and Creech civil service oversight curve**

- 0

- 0.05

- 0.1

- 0.15

- 0.2

- 0.25

- 0.3

- 0.35

- 0.4

- 1

- 2

- 3

- 4

**Level of Insight**

**Ratio of CS to Contractor**

**1 to 10**

**1 to 8**

**1 to 5**

**1 to 3**

_**Level 0 - No Oversight**_
**  ****Accept performing organization’s tasks at face value  (based on assessment that no penetration is required)**
_**Level 1 - Low Oversight**_
**  ****Participate in reviews and Technical Interchange Meetings and assess only the data presented**
**  ****Perform periodic audits on pre-defined process(es)**
**  ****Chair board or serve as board member, or RID writer, at a formal review**
**  ****Participate in resolution and closure of issues**
_**Level 2 - Intermediate Oversight**_** **
**  ****Includes low penetration with addition of:**
**  ****Daily or weekly involvement to identify and resolve issues**
_**Level 3 - In-depth Oversight**_
**  ****Includes intermediate penetration with addition of:**
**  ****Methodical review of details**
**  ****Independent models to check and compare vendor data, as required **
_**Level 4 - Total Oversight**_** **
**Perform a complete and independent evaluation of each task**** **

- NASA Headquarters Cost Analysis Division

## The “Wethington Full Cost Factor”

| Cost Element | Methodology |
| --- | --- |
| Procurements | From Cost Model |
| Direct Personnel
1. Civil Service Labor

2. On-site Contractor | 
f (Complexity, Acquisition Strategy, Duration, Oversight, Mission, Prime Contractor FYE)
Factor of Civil Service labor |
| Travel | Factor of Civil Service Labor |
| Service Pools
1. Facilities & Related Services
2. Information Technology
3. Science & Engineering Services
4. Fabrication Services
5. Test Services
6. Wind Tunnel Services | Factor of direct personnel or procurement (depending on best fit) |
| Center G&A
1. Center # 1
2. Center # 2
3. Center # 3 | Published factor |
| Corporate G&A | Published factor |

- NASA Headquarters Cost Analysis Division

## Cost Analysis Steering Group

**Next meeting scheduled for Tu/We April 6-7 at NASA HQ**
**Agenda**
- Discussion of Cost Analysis Data Requirement (CADR)
- Discussion of One NASA Cost Estimating (ONCE) Data Base
- Discussion of Cost Risk Analysis
- Discussion of Cost Risk Feedback Continuum (CRIFCON)

- NASA Headquarters Cost Analysis Division

## End