# Presentation: 435912

## Cost Risk Analysis “Standards”Alfred Smith, Dr. Shu-Ping Hu13 April 2005

****** ****Los Angeles    ******** Washington, D.C.     ******** Boston    ******** Chantilly    ******** Huntsville    ******** Dayton    ******** Santa Barbara**
****** ****Albuquerque    ******** Colorado Springs    ******** Columbus    ******** Ft. Meade    ******** Ft. Monmouth    ******** Montgomery    ******** Ogden    ******** Patuxent River    ******** Pensacola    ******** San Diego **
****** ****Charleston    ******** Cleveland    ******** Denver    ******** New Orleans    ******** Oklahoma City    ******** Silver Spring    ******** Warner Robins AFB    ******** Vandenberg AFB **

- 12 April 2005

## Describe a systematic simulation based cost risk analysis approach demonstrating how to model: 
CER risk (including factor relationships) 
Configuration (cost driver) risk
Correlation (Pearson product moment, not Spearman rank order)

Propose standards to characterize and present the results 
Propose what needs to be “published” to bring standardization to Cost Risk analysis independent of the tool(s) selected
Compare Crystal Ball, @Risk, ACE and FRisk to an analytically solved case study and case studies for which no analytical solution is feasible.

- CER risk (including factor relationships)
- Configuration (cost driver) risk
- Correlation (Pearson product moment, not Spearman rank order)
**Propose standards to characterize and present the results **
**Propose what needs to be “published” to bring standardization to Cost Risk analysis independent of the tool(s) selected**
**Compare Crystal Ball, @Risk, ACE and FRisk to an analytically solved case study and case studies for which no analytical solution is feasible.**

- Objective

- 12 April 2005

## Setting the Stage: Overview of Existing Guidance
Proposing a Process: A Six Step Cost Risk Analysis “Standard” Approach
Show how the NASA 12 Tenets are captured
Focus on modeling cost risk, configuration risk and correlation
Identify key decisions required to establish a standard approach
Available Risk Simulation Tools:
Crystal Ball, @Risk, and ACE RI$K all give the same results for the same problem (including correlation application).
How to ensure fair comparison across tools
Concluding Observations

**Proposing a Process: A Six Step Cost Risk Analysis “Standard” Approach**
**Show how the NASA 12 Tenets are captured**
**Focus on modeling cost risk, configuration risk and correlation**
**Identify key decisions required to establish a standard approach**
**Available Risk Simulation Tools:**
**Crystal Ball, @Risk, and ACE RI$K all give the same results for the same problem (including correlation application).**
**How to ensure fair comparison across tools**
**Concluding Observations**

- Outline

- 12 April 2005

## Setting the

- Stage

- 12 April 2005

## Cost Risk Analysis Publications 

**Risk Management Policies from DoD 5000.4-M Cost Analysis Guidance and Procedures**_**[http://acc.dau.mil/simplify/ev.php?ID=6388_201&ID2=DO_TOPIC](http://acc.dau.mil/simplify/ev.php?ID=6388_201&ID2=DO_TOPIC)**_** **
**Department of the Army Cost Analysis Manual May 2002**_**[http://](http://www.ceac.army.mil/ce/default.asp)**__**[www.ceac.army.mil/ce/default.asp](http://www.ceac.army.mil/ce/default.asp)**_** **
**(Air Force) Cost Analysis Guidance And Procedures 1 October 1997**_**[http://](http://www.saffm.hq.af.mil/afcaa/)**__**[www.saffm.hq.af.mil/afcaa](http://www.saffm.hq.af.mil/afcaa/)**__**[/](http://www.saffm.hq.af.mil/afcaa/)**_** **
**NASA Cost Estimating Handbook 2002**_**[http://www.jsc.nasa.gov/bu2/NCEH/](http://www.jsc.nasa.gov/bu2/NCEH/)**_** **_**[http://www.jsc.nasa.gov/bu2/conferences/NCAS2004/index.htm](http://www.jsc.nasa.gov/bu2/conferences/NCAS2004/index.htm)**_
**FAA Life Cycle Cost Estimating Handbook v2 03 Jun 2002 **_**[http://www.faa.gov/asd/ia-or/lccehb.htm](http://www.faa.gov/asd/ia-or/lccehb.htm)**_** **
**Parametric Estimating Initiative (PEI) Parametric Estimating Handbook Spring 1999**_**[http://www.ispa-cost.org/PEIWeb/newbook.htm](http://www.ispa-cost.org/PEIWeb/newbook.htm)**_**   **
**Recent new AFCAA study by RAND.  “Towards a Cost Risk Analysis Policy”**

- 12 April 2005

## General Guidance that isTough to Implement 

**“****Areas of cost estimating uncertainty will be identified and quantified.” **
**“****Areas of uncertainty, such as pending negotiations, concurrency, schedule risk, performance requirements that are not yet firm, appropriateness of analogous systems, level of knowledge about support concepts, critical assumptions, etc., *****should***** be presented.”**
**“****Uncertainty will be quantified by the use of probability distributions or ranges of cost.” **
**“****Detailed back-up material will be provided.” **
**“****Experts disagree on the sources of uncertainty in systems acquisition.”**

- 12 April 2005

## Sources of Cost Estimating Uncertainty 

**Uncertainty **_***commonly attempted***_** in cost risk models:**
- Cost estimating relationship (CER) risk
- Cost factors such as labor rates, labor rate burdens, etc
- Configuration risk (variation in the technical descriptions driving the CERs)
- Schedule and technical risk (in excess of that captured in the CER)
- Correlation between risk distributions
**Uncertainty **_***commonly missing***_** in cost risk models:**
- Potential for massive and frequent requirements changes
- Budget Perturbations, Congressional actions
- Re-work, and re-test phenomena
- Contractual arrangements (contract type, prime/sub relationships, etc)
- Potential for disaster (labor troubles, shuttle loss, satellite “falls over”, war, etc)
- Probability that if a discrete event occurs it will invoke a project cost
**NOT** the subject of this presentation, even though **NASA Tenet 8** requires it and most DoD organizations want/need to see it captured in the estimate

- 12 April 2005

## NASA Cost Risk Analysis Tenets  

**subset of cost estimating, supports optimum project management**
**common set of risk and uncertainty definitions**
**joint activity between subject matter experts and cost analysts**
**CER risk plus technical risk plus correlation **
**combine probabilistic and discrete technical risk assessments**
**probability distributions are justifiable, correlation levels based on actual cost history **
**cost estimates are “likely-to-be” vice “as specified” for optimum credibility**
**account for all known variance sources and include provisions for uncertainty**
**cost-risk can be an input to every cost estimate’s Cost Readiness Level (CRL);**
_**integrates **_**the **_**quantification of cost-risk and schedule risk**_
**decision makers need to know: **
- How much money is in the estimate to cover risk events;
- To which WBS elements are they allocated; and,
- The confidence level of the estimate;
**tons of stuff to be stored in the One NASA Cost Estimating (ONCE) database. **
**Index of most recent NASA cost risk papers: **_**[http://www.jsc.nasa.gov/bu2/conferences/NCAS2004/index.htm](http://www.jsc.nasa.gov/bu2/conferences/NCAS2004/index.htm)**_
**Description of NASA cost risk tenets: **_**[http://www.jsc.nasa.gov/bu2/conferences/NCAS2004/presentations/2](http://www.jsc.nasa.gov/bu2/conferences/NCAS2004/presentations/2)**_

- 12 April 2005

## Common Cost Risk Analyst Observations

**Analysts want to have...**
**Clear guidance on how to conduct cost risk analysis**
**Standard expectations for quality and completeness**
**Consistent approaches for:**
- Interpreting the point estimate CER (mean?, median? mode?, other?)
- Sensitivity analysis vs. stochastic analysis?
- Selecting a distribution and its bounds?  Are there defaults?
- Defining dispersion and/or correlation
- Adjusting risk for schedule/technical concerns?
- Planned growth (i.e., weight, power, operational profile, etc margins).
- Risk allocation
- How to sum costs with differing confidence levels (think software + hardware)
- What/how to present to managers (including BY vs. TY)
**	****Analysts want to improve the quality of their risk adjusted cost estimates in a more productive/repeatable way.**

- 12 April 2005

## Six Step Cost Risk 

- Six Step Cost Risk
- Analysis Approach

- 12 April 2005

## Cost Risk Analysis Approach

***Step 1***: _***Complete***_ the Point Estimate

- 12 April 2005

## Step 1:  The Point Estimate 

- Step 1:  The Point Estimate

_**Elements of a Point Estimate:**_
- R&D, Procurement, and O&S
- Software, Hardware & Personnel
- Inherent levels of indenture
- Combination of methods:
- Engineering build-ups
- Linear/non-linear CERs
- Pass-throughs, etc.
- CERs derived from historical data
- CERs (Judgmental)
- Inflation, learning, fee/overhead
- Phased & non-phased variables
- BY & TY phased results

**Decision Required:**  Define what should be addressed in a _**risk analysis**_ (vs. sensitivity analysis). (**NASA Tenet 5**.)

- 12 April 2005

## Step 2.a: Cost Estimating Risk:Picking a Distribution Shape and Bounds

**Objective Distribution Selection**
- OLS CERs – produce the “**mean**” (also the mode/median), error is **normally** distributed.
- Log Space OLS CERs - produce the “**median**”, error** **is **log-normal** in unit space.
- MUPE CERs approximates the “**mean**”, where the error is **normally** distributed.
**Subjective Distribution Selection**
- Analysts will often declare that risk will be non-symmetrical about the CER result.
- Risk on non-parametric CERs (analogy, build-up, through-puts) are almost always subjective.
- Log-normal, weibull, or beta are popular to avoid a sharp peakness around the mode with at least some probability of a large overrun.
**Bounds**
- Statistical analysis (objective)
- Expert Opinion (subjective)

**Suggestion** (**NASA Tenet 6**)**:**
- Publish the objective distribution shape for each regression technique.
- Define how to interpret the CER (mean or median).
- Provide guidance on what to pick if there is a basis to depart from the objective shapes.

- 12 April 2005

**Notes:**
Instructor should open the training utility and demonstrate.  Instructor should also note that this is available to all students in their training CD.  There is one picture in the book to show pictures from the utility.

Use this slide to talk about does the equation make sense ask how much would the missile cost if the payload wgt is “0”. Ans 92.

Unit Space CERs - Error term is additive and the CER produces the mode which is also the mean and the median of a normal distribution
Log Space CERs – Error term is multiplicative and the CER produces the median unless MUPE is used.  All MUPE CERs have normal distributions an produce the mode.
MUPE can be run in CO$TAT.
R^2 Adj is the dispersion of the CER.  T stats need to be above 90% to indicate probability they are not zero.  MAD needs to be less than 25%

## Step 2.a: “Standard” Distribution Shapes and Bounds

**Plots compare different distribution shapes based on similar dispersion (CoV)**

**Suggestion:**
- Publish “standard” distribution shapes and bounds.
- Develop tables for different distribution shapes by commodity. (**Support for NASA Tenet 6**)

***CoV – Coefficient of Variation***
***= standard deviation/mean***
***For symmetric distributions: standard deviation/point estimate***

- 12 April 2005

**Notes:**
Use the RI$K Distribution feature of the utility to demonstrate this.

A main message here is that it is not picking the distribution that is the big issue it is rather the bounds of the distribution.

The difference between a “central” skewed triangular, beta and normal distributions is minimal.  
You can not skew a normal distribution.
The difference between a skewed triangular or skewed beta distribution is more noticeable and therefore is a more important decision.

## Step 2.a: Use Basic or Advanced Wizards to set Shapes and Bounds

- 12 April 2005

**Notes:**
Use the RI$K Distribution feature of the utility to demonstrate this.

A main message here is that it is not picking the distribution that is the big issue it is rather the bounds of the distribution.

The difference between a “central” skewed triangular, beta and normal distributions is minimal.  
You can not skew a normal distribution.
The difference between a skewed triangular or skewed beta distribution is more noticeable and therefore is a more important decision.

## Step 2a: Some Potential Display Standards

**Point estimate reflects “median” for lognormal, “mode” for all others.**
**Right click to choose distribution and “default” spread/skew**
**Permit dispersion to be specified such that distributions scale with sensitivity analysis....ie bounds that are a % of the point estimate, log SE for log-normal or CoV**
**Bracketed numbers in Baseline column reports point estimate confidence level**

- 12 April 2005

## Step 2.b: Configuration Risk

**Focus is now on the cost drivers (risk or sensitivity analysis?)**
**Frequent sources of cost risk: learning slope, lines of code count, weight, composite labor rates, etc. assumptions**
**Modeling considerations:**
- Do CER inputs represent design goals or include allowable margin?
- Do CER inputs represent the mode/mean/median (normal error) or median (log-normal error) or some other percentile value?
- Are only discrete sets of CER inputs permissible (i.e. is it inappropriate to model them with continuous risk distributions)?
- Can CER inputs be functionally linked?  For instance, can airframe weight be estimated from the engine weight?

**Suggestion: ****NAFCOM permits analysts to assign distributions to the inputs.****  **Publish “default” input variable interpretation, distribution shapes, and bounds based upon commodity type.

- 12 April 2005

## Step 2.c: Schedule/Technical Considerations  

**Estimating methods capture some “nominal” schedule/technical cost impact (contributes to regression error term?).  **
- Compare the project you are estimating to the CER source data.
- Realistically assess the degree to which the schedule and technical considerations compare to the CER source.
**CERs, estimating methods, analogy and expert opinion estimating processes are influenced by past, real projects.**
**Difficult to isolate **_**schedule from technical**_** cost impacts.  Many approaches assess the impact together.**
**Subjective assessment.**

**Decision Required: **
- 	Develop a default method for adjusting risk distributions to capture schedule and technical considerations:
- Parametric approach - penalty factor, additional distribution, etc
- Employ schedule and EVM experts to explicitly model the schedule risk**(NASA Tenet 10).**

- 12 April 2005

## Step 2.d Review for Consistency

- Bounds expressed as % of point estimate _**or CoV**_ (unitless):
- Scale with changes to the point estimate
- Provide a consistent basis for comparison

- 12 April 2005

## Step 3a: Run the Simulation

**Simulation tool results are influenced by:**
- Interpretation of point estimate
- Truncation assumption (do you allow cost, weight, etc risk to go negative?)
- Number of iterations
- If using Latin Hypercube [LHC], the number of partitions
- Random seed (impossible to be consistent between tools.   Some tools at least provides for consistency across different machines and different versions)
**When the above assumptions are consistent (as far as possible), Crystal Ball, @Risk, ACE and FRisk all produce similar results.  **

**Decision Required:****  **
- Identify acceptable risk simulation tools
- Provide guidance on how they should be applied
- Periodically publish “common errors” as new versions are released

- 12 April 2005

## Step 3b: Correlation

_***Measure***_** the correlation **_***already present***_** due to modeling relationships to determine if correlation needs to be adjusted**
**Modeling considerations often overlooked when trying to assess the correlation **_***already present***_** in the cost model**
- Functional relationships between the input variables
- Functional relationships between WBS elements
- More than one CER sharing same risk-adjusted input variable. (example: same risk adjusted learning slope variable driving more than one WBS element)
- Simulation tool bias (i.e. how random seeds are generated).
**Input variable functional relationships can be simulated using correlation (i.e.: cause structure weight to “move with” payload weight)**

- 12 April 2005

## Measure Correlation Present in The Cost Model

- Pearson Product moment correlation measured by capturing results from every iteration
- (Excel CORREL function can be used to validate)

- Define intention when “injecting” correlation.
- Correlation after layering an additional 20% across all elements

- 12 April 2005

## Step 4: View and Interpret Results

**Risk analysis will give context to the point estimate**
**CoV (Stdev/Mean), confidence of the point estimate (PEcl) and quartile range are useful measures of the overall risk in the cost model ****(*****Tenet 9*****)****.**
**Observations in DoD Estimates:**
- Estimates rich in parametric CERs: 15%<CoV<45%, and 5%<PEcl<30%
- Estimates rich in build-up methods:  5%<CoV<15%, and 30%<PEcl<45%

**Suggestion:****  **Identify reasonable, commodity-based metrics the analyst can use to assess the completeness and possibly the quality of the risk analysis as it is being developed.  **NASA has done so with the CRL concept.**

- 12 April 2005

## Step 5: Allocate Risk

**Confidence level results *****do not add*****  **
- Mathematicians are quite happy with this result, budget folks are not.
**Results must:**
- Be phased in both BY (constant year) and TY$ (real dollars?)
- Add up
**Selection of level from which to allocate risk has a significant impact on total**
**Many issues must be resolved to define a phased, risk allocation method that yields consistent BY and TY results**
**Problem is very much exacerbated if policy requires some elements to be at one confidence level (i.e. 80%) and others at another (i.e. 50%)**
**Phasing assumptions will have **_***significant impact***_** on TY risk results.**

**Decision Required:****  **
- Choose a “default” risk allocation approach, including how the cost risk dollars should be phased along with acceptable alternatives
- Define how to deal with elements that are at different CLs
- Cost models should be flexible enough to phase the risk dollars consistent with the program managers risk mitigation plans

- 12 April 2005

## Consequence of an Allocated Risk Report

**In this example, risk funds managed from the 2****nd**** level (70%)**
**Total project dollars required are greater than 70% CL overall**
**All numbers “add”**

- 12 April 2005

## Step 6:  Charts and Tables

**Decision Required:****  **
- Identify the standard charts and their contents to be presented to management.
- Ensure consistent x and y-axis arrangements.
- Determine “if” a TY S-curve should be presented and if so, define the process to be used (Sep 04, Army funded such a study and a solution is at hand).**  **

- 12 April 2005

## Benefits of Clear Guidance

**Default positions could establish minimum guidance & expectations for cost risk analysis – not a cookbook**
**No need to “over specify” the guidance**
**Advanced analysts will still develop sophisticated models to deal with exceptional circumstances**
**Establishing a “standard process” will:**
- Focus analyst’s attention on “building” the risk adjusted estimate rather than determining “how” to build it
- Enable more risk analysis practitioners to “do” cost risk analysis _***with confidence***_

- 12 April 2005

## Cost Risk Tools

- 12 April 2005

## Compare Cost Risk Tool Results

**What are the risk tools and which should I choose?**
**Crystal Ball, @Risk, ACE RI$K and FRisk **_***results***_** are compared.... Not their usability or suitability. **
**Two case studies examined ****(SCEA paper has three)****: **
- Published, simple and analytically solved case study (SCEA paper June 04, Reference 5).
- Second example is based upon a more “realistic” cost model that cannot be solved analytically (Reference 7).
**If handled properly, **_***all tools produce similar total cost distribution results***_** even when correlation is applied.**

- 12 April 2005

## A “Realistic” Model

- 12 April 2005

## USCM 7 Comparison

- More than 30 linear, non-linear, throughput CERs and 30 input values
- Compared total cost result at the 95th percentile based upon a systematic layering of correlation assumptions
- All three tools produce remarkably similar results in each scenario.

- Point Estimate: $480k

**CoV**
**18%**

**CoV**
**20%**

**CoV**
**22%**

**CoV**
**35%**

- 12 April 2005

## Comparing Risk Tools

**If you are consistent with:**
- How to interpret the point estimate
- Number of iterations.
- If using Latin Hypercube [LHC], the number of partitions.
- Inflation, learning, and other modeled adjustments.
- How functional correlations are modeled
- Distribution shape and bound assumptions.
- Truncation assumptions.
**If you follow the tool developer’s recommendation for inputting correlation:**
**	****Crystal Ball, @Risk and ACE will give results well within the simulation tool error band. **

- 12 April 2005

## Backup Slides

- 12 April 2005

## ACEIT Is Structured to Automate the Estimating Environment

- 12 April 2005

## Need for Correlation Wizard

- Ability to layer additional correlation across selected WBS or cost driver elements (or a blend of both)

**Decisions Required:**   Define how correlation should be applied.  Decide if you should allow the user to **“turn off”** functional correlation.

- 12 April 2005

## Case Study Page CE V – 80 SCEA Training Manual

**Combination of throughput and factor relationships**
**No risk applied to the factors**
**PMP drives about 70% of the model result, so 70% of the risk is modeled with a normal distribution making it reasonable that the total cost is likely to be normally distributed.**
**Sys Test & Eval has an additive risk which is unusual in cost risk analysis.  We generally assume the risk scales with the estimate. **

- 12 April 2005

## All Tools Perform Well

- 12 April 2005

## Unintentional Correlation?

- _Unintentional_ Correlation?

- 12 April 2005

## Removing Unintentional Correlation

- Need separate slope variable for the missile.

**Decisions Required:**
- Define Correlation Strength
- Strong (.9?)
- Moderate (.6?)
- Weak (.2?)
- When to apply?

- 12 April 2005

## Impact on on Total Cost by Layering Risk Assumptions

- In this model, the impact of correlating the Gimbal elements is insignificant.   Applying 20% across all remaining WBS elements and inputs increases the cost result at 80% by 12%.   The CoV of the final result is 35%.

- Applying risk to the CERs and inputs in ACE, before layering correlation, captures most of the risk.  Forcing an additional 20% correlation across all WBS elements (other than the Gimbal) does have a significant impact in this model.
- Although the CoV of the final result is 35%, it might be excessive.  To force even a 20% correlation across all elements is contrary to correlation studies on some datasets.

- 12 April 2005