# Presentation: 037381

## Cost Risk Analysis Panel

- Timothy P. Anderson
- NASA Cost Symposium
- March 9, 2004

## Predicting the Future

- A TECHNICAL REALITY:  Fundamental inability to predict the future!
- Since it is impossible to make exact predictions we have:
- Cost estimating relationships (CERs)...
- ...based on historical data...
- ...for the purpose of making statistically-based educated guesses...
- ...about the cost of systems that have yet to be built.

**Notes:**
CER	Cost Estimating Relationship

## Probabilistic Nature of Cost Estimates

- Typical CERs are based on historical cost data.

- And, typical cost estimates have probability distributions.

- Cost estimates are *really* probability distributions and *not* deterministic!
- But, prevailing practice is that cost estimators report, say, the 50th percentile of the probability distribution as *the cost estimate*, and decision-makers then choose to budget at a value that is even less than that!

**Notes:**
CER	Cost Estimating Relationship

## A Narrative

- Consider the following exchange between a decision-maker and his cost analyst:
***Mr. O’Keefe:**** So, Joe, how much will our new system cost?*
***Joe Hamaker:**** Sir (looking at the 50**th** percentile number), our estimate is $115M.*
***Mr. O’Keefe:**** Well, Joe, that’s too high.  The contractor says it will only cost $72M.  We’ll just have to manage to that number.*
***Joe Hamaker:**** Aye, aye, sir.*
- Sound familiar?

## The Consequences

- Since cost probability distributions tend to be “right-tailed,” then the 50th percentile turns out to be less than the *expected* (mean) cost estimate.
- Also, if cost is estimated at the 50th percentile, then there is a 50% chance that the actual cost will be greater than the cost estimate!
- Moreover, when decision-makers choose to budget at a value less than the 50th percentile, say the 5th percentile, then they are guaranteeing a 95% probability of a budget overrun!

## More Consequences

- Consider the following exchange between our decision-maker and his cost estimator *one year later:*
***Mr. O’Keefe:****  Joe, the contractor says our program is now projected to cost $135M!  You told me it would be $115M!  YOU’RE FIRED!*
***Joe Hamaker:****  But Sir, it’s all Carpio’s fault!*

## Anderson’s Law

- Cost estimators should explain the probabilistic nature of their estimate, and...
- Acquisition decision-makers should set budgets so that they have a reasonable probability of budgetary success.

**Notes:**
Std. Dev.	Standard Deviation

## A Different Narrative

- Here’s how that initial exchange *should* have gone:
***Mr. O’Keefe:**** So, Joe, how much will our future system cost?*
***Joe Hamaker:**** Sir, we’ve developed the following cost probability distribution (shows the distribution to the decision-maker).  This distribution has an expected value of $120M, but notice, sir, that there is a fair amount of variability in this estimate.  With a standard deviation of $35M, the cost could easily exceed $155M.*
***Mr. O’Keefe:**** Well, Joe, that’s too high.  The contractor says it will only cost $72M.  We’ll just have to manage to that number.*

- Continued...

## A Different Narrative

***Joe Hamaker:**** Yes, sir, I understand what the contractor says, but we have confidence in this cost estimate based on our knowledge of history.  The contractor might be right, but the odds are against it.  If you really want this new system, then we need to be prepared to budget more than $72M for it.  In fact, $72M falls at the 5**th** percentile of this estimate.  That means there is only a 5% chance that $72M will be enough, with a corresponding 95% chance that the actual cost will exceed $72M.*
***Mr. O’Keefe:****  Hmm...I see what you mean.  So where should we set the budget to have, say, an 80% probability of avoiding a budget overrun?*
***Joe Hamaker:****  Well, sir, you should set the budget at the 80**th** percentile of the estimate, or $146M.*
***Mr. O’Keefe:**** Okay, Joe, thanks for the insight.  I’ll see what I can do about getting the budget you recommend.*

## Better Consequences

- Suppose, because of the high priority of this system, Mr. O’Keefe was able to budget to Joe’s recommendation of $146M.
- Now consider the following exchange *one year later:*
***Mr. O’Keefe:**** Joe, the contractor says our program is now projected to cost $135M!  But because of what you told me about budget risk, I’ve got enough money in the budget to cover it.  Thanks again for explaining that cost probability distribution to me.  I’M PUTTING YOU IN CHARGE OF CRLs!*
***Joe Hamaker:****  Aye, aye, sir!*

## Tenet #7

**Tenet #7**

- NASA cost risk assessment ensures cost estimates are for “likely-to-be” programs vice “as specified” for optimum credibility.

## “Likely-To-Be” vs. “As Specified”

- “As Specified”
- No engineering issues
- No uncertainties
- This time we’ll get it right
- No weight growth
- Optimistic assumptions
- No requirements changes
- No ECPs
- No schedule creep

- “Likely-To-Be”
- Test failures
- Technical uncertainty
- As usual, we made mistakes
- Unanticipated weight growth
- We were *too* optimistic
- Some requirements change
- Some ECPs
- Some schedule creep

## All Programs Have Experienced Problems

- Listed below are all of the perfectly-planned NASA programs that had no cost increases, weight growth, schedule creep, ECPs, etc.
- We are continually seeking programs to add to this list, but haven’t found any to date.

- NOTE: This list is empty.

## Estimating “Likely-To-Be”

- History shows that *no* space acquisition goes along “as specified”
- Weight growth
- Software growth
- Schedule creep
- Cost growth
- Everyone wants NASA programs to be successful.
- But a key measure of success is getting the cost right!
- Underestimating cost with the expectation that things will go “as specified” is a recipe for disappointment.

## Tenet #6

**Tenet #6**

- ...correlation levels are based on actual cost history to the maximum extent possible.

## Correlation Based on Actual History

- I agree, and I’ll defer comments on this to Steve Book.
- However, this is not always possible...then what?
- May be necessary to *subjectively* assess *approximate* correlation between WBS elements.
- Better than nothing.

## Tenet #11

**Tenet #11**

- NASA decision-makers need to know:
- How much money is in the estimate to cover risk events:
- To which WBS elements are they allocated; and
- The likelihood of an overrun.

## NRO Cost Group Risk Process

- Step 1: Define “likely-to-be” program.
- Rather than the “optimistic” program that the contractor and program manager are planning for.
- Step 2: Quantify the probability distributions describing the modeling uncertainty of all CERs, cost factors, and other estimating methods.
- Specifically, the type of distribution (normal, triangular, lognormal, beta, etc.)
- The mean and variance of the distribution
- Step 3: Quantify the correlation between all WBS elements that are estimated using CERs and other methods.
- Correlation affects the overall cost variance

**Notes:**
CER	Cost Estimating Relationship

## NRO Cost Group Risk Process

- Step 4: Set up and run the cost estimate in a Monte Carlo framework (e.g., Crystal Ball, @RISK), resulting in a “baseline” estimate.
- This will provide a probability distribution of the cost based on cost estimating model uncertainty only.
- Report the MEAN as the *baseline* expected cost.
- Step 5: Now incorporate technical uncertainty and discrete risks.
- Step 5a: Set up a new baseline estimate which also contains any “discrete risk” events that are to be guarded against.
- Quantify appropriate modeling uncertainties and correlations, as in Steps 2 and 3, for these discrete risks.
- Step 5b: Define the probability distributions for all CER input variables.
- Also may need to quantify correlation between CER input variables.

**Notes:**
CER	Cost Estimating Relationship

## NRO Cost Group Risk Process

- Step 6: Re-run the Monte Carlo simulation with random CER input variables and discrete risk events, resulting in a final “risk-adjusted” estimate.
- Results in a new *risk-adjusted* cost probability distribution.
- Wider and shifted to the right.

- Baseline
- Estimate

- Risk-Adjusted
- Estimate

**Notes:**
CER	Cost Estimating Relationship

## NRO Cost Group Risk Process

- Step 7: Assess “risk dollars.”
- Difference between the “risk-adjusted” *mean* and the “baseline” *mean* represents the estimate of “risk dollars.”
- Risk dollars can be allocated downward to any level of WBS using a variety of simple approaches.

**Notes:**
WBS	Work Breakdown Structure
QRC	Quick Reference Card

## NRO Cost Group Risk Process

- Step 8: Assess “budget risk.”
- Area under the PDF to the right of the budget represents budget risk.

**Notes:**
PDF	Probability Density Function
QRC	Quick Reference Card

## NRO Cost Group Risk Process

- Step 9: Assess “management reserve.”
- Difference between mean of risk-adjusted estimate and budget represents management reserve.