# Presentation: 109831

***NRO Cost Group******Risk Process***

**UNCLASSIFIED**

**UNCLASSIFIED**

- Copyright © 2003, The Aerospace Corporation, All Rights Reserved

## Introduction

**Problem: NRO Cost Group (NCG) has had problems with its Monte Carlo cost risk analysis process, e.g.,**
- Negative risk!
- Traceability!!
- Inability to answer key questions!!!
**D/NCG did not want to continue using Monte Carlo simulation in its cost risk analysis.**

## Introduction

**Questions D/NCG *****really***** wanted to be able to answer:**
- How much ‘risk’ is in the estimate?
*Translation*: How many dollars are in the recommended budget to guard against ‘risky’ events happening?
- And, to which WBS elements are they allocated?
- How ‘risky’ is the recommended budget?
*Translation*: If the budget is set at any value, what is the likelihood of an overrun?
- How much ‘management reserve’ is in the budget?
*Translation*: How many dollars are in the budget for management reserve, over and above dollars expected to be needed for acquisition and risky events?
**NCG Objective:**
- Develop a risk process we understand, that is acceptable community-wide, that answers these questions, and that we can readily describe to senior decision-makers.

## ICE Scope

**DDNRO requirements for ICE**
- Provide an estimate for what the program is  “likely to be” vice the program “as specified” by the SPO and contractors
**Should include:**
- Provision for normal problems, not “sunny day” estimates
- Schedule growth
- Requirements creep
- Weight growth
- Software code growth
- Etc...

## Risk Process Flowchart

- DDSE
- Independent
- Technical
- Assessment (ITA)

_**Baseline Assessment**_
**Run Models**
**(deterministic**
**inputs)**

- KTR-completed
-  Data
- Sheets

- “Likely to be”
- Deterministic
- Model inputs

**Baseline (ITA)**
**Estimate**

**Risk-Adjusted**
**Estimate**

_**Risk Assessment**_
**Run Models**
**(probabilistic inputs**
**and discrete risks)**

- NCG

- DDSE

- KTR

**“****Likely to be”**
**Probabilistic**
**Model inputs**
**(includes discrete**
**risks)**

**Risk dollars**

## Basic Risk Principles

**Cost estimates usually involve many CERs**
- Each of these CERs has uncertainty (standard error)
- CER input variables have uncertainty (technical uncertainty)
**Must combine CER uncertainty with technical uncertainty for many CERs in an estimate**
- Usually cannot be done arithmetically; must use simulation to roll up costs derived from Monte Carlo samples
- Add and multiply probability distributions rather than numbers
- Statistically combining many uncertain, or randomly varying, numbers
- Monte Carlo simulation
- Take random sample from each CER and input parameter, add and multiply as necessary, then record total system cost as a single sample
- Repeat the procedure thousands of times to develop a frequency histogram of the total system cost samples
- This becomes the probability distribution of total system cost

## CER Results - A Cost Probability Distribution 

**   **

**$**

**Cost Driver (Weight)**

*Cost = a + bX**c*

- Input
- variable

- Cost
- Estimate

- Historical data point

- Cost estimating relationship

- Standard percent error bounds

- Technical Uncertainty

- Combined Cost Modeling and Technical Uncertainty

- Cost Modeling Uncertainty

## Probability Density Function (PDF)

***Probability Density Function (PDF)***

**Lognormal Cost Distribution**
**Mean = $120M**
**Std. Dev = $35M**

- 5th percentile
- $72M

- Mean
- $120M

- 80th percentile
- $146M

- 95th percentile
- $184M

- 50th percentile
- $115M

## Cumulative Distribution Function (CDF), or S-Curve

- 0%

- 10%

- 20%

- 30%

- 40%

- 50%

- 60%

- 70%

- 80%

- 90%

- 100%

- 55

- 65

- 75

- 85

- 95

- 105

- 115

- 125

- 135

- 145

- 155

- 165

- 175

- 185

- 195

- 205

- 215

- 225

**$M**

**Cumulative Probability**

***Cumulative Distribution Function (CDF), or S-Curve***

- 80th percentile
- $146M

- 95th percentile
- $184M

- 50% probability of cost coming in at or below $115M
- 45% probability of cost coming in between $115M and $184M
- 20% probability of cost exceeding $146M
- 5% probability of cost exceeding $184M

- Mean
- $120M

- 50th percentile
- $115M

## The NCG Risk Process

## Basic Principles

**Step 1: Define “likely-to-be” program.**
- Using deterministic inputs from the Independent Technical Assessment (ITA).
**Step 2: Quantify the probability distributions describing the modeling uncertainty of all CERs, cost factors, and other estimating methods.**
- Specifically, the type of distribution (normal, triangular, lognormal, beta, etc.)
- The mean and variance of the distribution
**Step 3: Quantify the correlation between all WBS elements that are estimated using CERs and other methods.**
- If unknown, assess whether NO correlation, MILD correlation, or HIGH correlation, for example:
- NONE: ** = 0,  MILD: ** = ±0.2,  HIGH: ** = ± 0.6
- Correlation affects the overall cost variance

## Basic Principles

**Step 4: Set up and run the cost estimate in a Monte Carlo framework (e.g., Crystal Ball, @RISK), resulting in a “baseline” estimate.**
- This will provide a probability distribution of the cost based on cost estimating model uncertainty only.
- Report the MEAN as the *baseline* expected cost.
**Step 5: Now incorporate technical uncertainty and discrete risks.**
- Step 5a: Set up a new estimate which also contains any “discrete risk” events that are to be guarded against.
- Quantify appropriate modeling uncertainties and correlations, as in Steps 2 and 3, for these discrete risks.
- Step 5b: Define the probability distributions for all CER input variables.
- Also may need to quantify correlation between CER input variables.

## Basic Principles

**Step 6: Re-run the Monte Carlo simulation with random CER input variables and discrete risk events, resulting in a final “risk-adjusted” estimate.**
- Results in a new *risk-adjusted* cost probability distribution.
- Wider and shifted to the right.

- Baseline
- Estimate

- Risk-Adjusted
- Estimate

## Basic Principles

**Step 7: Assess “risk dollars.”**
- Difference between the “risk-adjusted” *mean* and the “baseline” *mean* represents the estimate of “risk dollars.”
- Risk dollars can be allocated downward to any level of WBS using a variety of simple approaches.

## Basic Principles

**Step 8: Assess “budget risk.”**
- Area under the PDF to the right of the budget represents budget risk.

## Basic Principles

**Step 9: Assess “management reserve.”**
- Difference between mean of risk-adjusted estimate and budget represents management reserve.

## Example

**The following example demonstrates the procedure.**
**First, an estimate will be produced using the deterministic “roll-up” procedure.**
**Second, the “baseline” cost estimate resulting from the ITA will be developed using Monte Carlo simulation.**
**Third, the “risk-adjusted” cost estimate will be developed, again using Monte Carlo simulation.**

## The Deterministic Roll-Up Procedure

## The Deterministic Roll-Up Procedure

**In the deterministic roll-up procedure, we simply insert the input variables into the CER equations and evaluate the result.**
**In this example, we take neither CER modeling uncertainty nor input variable uncertainty into consideration.**

## Example Space System WBS

**WBS**

- 1.1 Bus

- 1.2 Software

- 1.3 Payload

**Total**

## Example Space System CERs

**Bus**
- Expected Cost (FY00$) = 87,450(Bus Weight)0.79
- Standard Percent Error (SPE) = 34%
**Software**
- Expected Cost (FY00$) = 435,216(KSLOC)0.83
- SPE = 46%
**Payload**
- Expected Cost (FY00$)
- = 3,568,510(Payload aperture)0.84(Number of prisms)0.95
- SPE = 25%

## Example Space System Input Variables

**Input variables may differ depending on who is providing them.  In this case the trend is...KTR<PM<ITA.**
**For this example, we will use the ITA (likely-to-be) inputs.**

## Example Deterministic Roll-Up Cost Estimate

- Note the lack of any information
- about cost uncertainty

## Monte Carlo Simulation of Baseline Cost Estimate

## Monte Carlo Simulation of ITA

**In the next example, we *****statistically***** combine the probability distributions of the CERs.**
**Since the CERs in this example produce MEANs, then the mean of the total estimate is equal to the deterministic roll-up.**
**However, the probability distribution of the total estimate reflects the uncertainty due only to cost modeling uncertainty.**

## Example CER ITA Input Assumptions

## Example CER Uncertainties

***Example CER Uncertainties***

**Bus CER evaluated at ITA input value**
- Lognormal error distribution
- Mean = 28,240,469
- Mode = 23,746,103
- SPE = 35%

## Example CER Uncertainties

***Example CER Uncertainties***

**Software CER evaluated at ITA input value**
- Lognormal error distribution
- Mean = 42,559,357
- Mode = 31,912,155
- SPE = 46%

## Example CER Uncertainties

***Example CER Uncertainties***

**Payload CER evaluated at ITA input value**
- Lognormal error distribution
- Mean = 31,804,379
- Mode = 29,039,790
- SPE = 25%

## Example CER Correlation Coefficients

**Correlation between WBS elements estimated using CERs must be quantified in order to properly assess the total cost distribution.**

## Example Space System Baseline Cost Estimate(Based on ITA)

**The Baseline cost estimate reflects CER modeling uncertainty only.  CER inputs are “most likely” values based on ITA.**
**This is the estimate for a program that has no technical uncertainty.**
**The “expected value” or MEAN is $102.6M, and the standard deviation is $29.8M.**

## Monte Carlo Simulation of Risk-Adjusted Cost Estimate

## Risk-Adjusted Cost Estimate

**In the risk-adjusted cost estimate, we now combine discrete risk events and the uncertainty of the input distributions with the uncertainty of the CERs.**
**Since the input distributions tend to be right-skewed, the expected cost tends to be larger than the baseline estimate.**
**In addition, the risk-adjusted cost distribution tends to be wider than the baseline estimate.**
**The difference between the expected cost of the risk-adjusted estimate and the expected cost of the baseline estimate is, by definition, the amount of *****RISK dollars***** included in the risk-adjusted estimate.**

## Example Risk-Adjusted Bus Input Assumptions

- Following an assessment of the bus weight, it is determined that bus weight may be modeled with a Beta distribution whose 10th percentile is 1300 lbs, most likely value is 1500 lbs, and 90th percentile is 2500 lbs.

## Example Risk-Adjusted Software Input Assumptions

- Similarly, it is determined that KSLOC may be modeled with a Beta distribution whose 10th percentile is 200 KSLOC, most likely value is 250 KSLOC, and 90th percentile is 500 KSLOC.

## Example Risk-Adjusted Prism Input Assumptions

- Likewise, it is determined that the number of prisms may be modeled with a Beta distribution whose 10th percentile is 8 prisms, most likely value is 10 prisms, and 90th percentile is 15 prisms.

- Note: Would be better to
- use a discrete distribution.

## Example Input Correlation Coefficients

**Correlation between CER input variables must also be quantified in order to properly assess the total cost distribution.**

## Example Risk-Adjusted Cost Estimate

- 0.000

- 0.200

- 0.400

- 0.600

- 0.800

- 1.000

- 0

- 100

- 200

- 300

- 400

- 0

- 100

- 200

- 300

- 400

-  5%

-  90%

- 5%

-  66.7771

-  204.5112

**The risk-adjusted cost estimate contains both CER modeling uncertainty and technical uncertainty.**
**The “expected value” or MEAN is $122.6M, and the standard deviation is $42.8M.**
**The difference between the expected value of this estimate and the baseline estimate is $20M.**
**Therefore, this estimate contains $20M “risk dollars.”**

## Example Risk-Adjusted Cost Estimate

**The expected value of the risk-adjusted cost estimate ($122.6M) is $20M higher than the expected value of the baseline cost estimate ($102.6M).**
**The interpretation is that the risk-adjusted cost estimate contains $20M risk dollars.**
**Additionally, the $20M can be allocated into each WBS element by subtracting corresponding means as follows:**
- Bus:  $6.6M risk dollars
- Software: $10.0M risk dollars
- Payload: $3.4M risk dollars
**Moreover, the standard deviation of the risk-adjusted estimate is about $43M while the baseline ITA estimate has a standard deviation of about $30M, indicating that the risk-adjusted cost estimate has a wider distribution.**

## Baseline vs. Risk-Adjusted Estimates

- Baseline:
- Mean = $102.6M
- Std Dev = $29.8M

- Risk-Adjusted:
- Mean = $122.6M
- Std Dev = $42.8M

## The “S-Curve”

- Baseline Estimate
- Mean $102.6M

- 50th percentile
- $114.7M

- Risk-adjusted
- Estimate Mean
- $122.6M

- 80th percentile
- $153.5M

## But How Much Will It Cost?

**This is impossible to answer precisely.**
**Decision-makers and cost analysts should always think of a cost estimate as a *****probability distribution*****, NOT as a *****deterministic number*****.**
**The best we can provide is the *****probability distribution***** – If we think we can be any more precise, we’re fooling ourselves.  **
***It is up to the decision-maker to decide where he/she wants to set the budget.***
**The probability distribution provides a quantitative basis for making this determination.**
- Low budget = high probability of overrun
- High budget = low probability of overrun

## A Possible Narrative

***Mr. O’Keefe:*** “So, Carpio, what’s your estimate, and how many ‘risk dollars’ are in it?”
***Rey Carpio:*** “Sir, here is the estimate (shows the probability distribution).  It has a *mean* of $122.6M. The *mean* is located at the 58th percentile, and contains $20M ‘risk dollars’ dedicated specifically for risk events over and above ‘normal’ cost modeling uncertainty.  Moreover, we’ve allocated that $20M risk dollars downward to the major WBS elements.  $6.6M is earmarked for bus weight growth, $10M is apportioned for software code growth, and $3.4M is allocated for risk in the number of prisms needed. But really, sir, the decision is up to you.  You’ve seen the cost probability distribution, where would *you* like to set the budget?”
***Mr. O’Keefe:*** “Gee, nobody’s ever explained it to me this way.  Great job Carpio! You deserve a raise!  Let’s be prudent and set our budget at, say, the 65th percentile of your estimate.”
***Rey Carpio:*** “Good thinking, sir.  Let’s see...the 65th percentile is $131.4M, therefore, in addition to $20M risk dollars, you’ll have an additional $8.8M of *management reserve* in the budget, *and *you’ll only have a 35% chance of a budget overrun.”

## Conclusions – Did We Answer The Customer’s Concerns?

**Negative risk?**
- It COULD happen!  But, using this methodology, “negative risk,” that is, occasions when the “risk-adjusted” estimate is less than the “baseline” estimate, will only occur when favorable outcomes are more likely than unfavorable outcomes.
**Traceability?**
- Risk dollars can be traced to the CER input distributions.
**Inability to answer key questions?**
- This methodology enables NCG to *quantify* and *explain*:
- Risk dollars
- Budget risk
- Management reserve

## Summary

**The NCG Risk Process.**
**Example.**
**Risk dollars.**
**Budget risk.**
**Management reserve.**
**A possible narrative.**
**Conclusions**