# Presentation: 280343

## Cost-Risk Connections Between CCRM & CRM

- David R. Graham
- OCFO/Cost Analysis Division
- HQs NASA

**Presentation to **
**NASA COST ANALYSIS SYMPOSIUM**
**14 April 2005**

## Purpose

- Describe and illustrate how CRM and CCRM are complementary
- Describe to cost community how CRM procedures and resource specialists can assist CCRM in determining cost-risk
- Describe to risk management community how CCRM cost-risk quantification can assist in assessing risk mitigation costs

## Focus of Briefing

- How CCRM’s cost-risk quantification can augment CRM’s risk mitigation costing
- Not going to focus on:
- Early cost/performance trades (e.g., CAIV) even though both CRM and CCRM emphasize early assessment of risk in early Analysis of Alternatives trades
- Is really a sub-set of general cost-risk quantification
- Nor on cost-risk data collection/cost-risk database
- Will concentrate on risk identification, prioritization, assessment, quantification and management commonalities between CRM and CCRM

## Continuous Cost-Risk ManagementA System of Cost Systems linked together in sequence by the same risks 

**Perform Cost/**
**Benefit Trades**
**(CAIV is a **
**Subset)**

**Build Reqmts/**
**Function /**
**WBS Matrix**
**(CADRe)**

**Develop Ref**
**Point Cost &**
**Schedule**
**Estimate**

**Assess WBS**
**Element Risk**

**Review Cost**
**Proposal**
**During Source**
**Selection**

**Participate**
**In Post-Contract**
**Award Meeting**

**5**

**4**

**3**

**2**

**1**

**12**

**9**

**11**

**10**

**Update LCCE**
**(CADRe) &**
**Cost / **
**Sched-Risk**
**Assessment**
** **

**Assess Data **
**For Model & **
**Database Updates**
** ****(CADRe) & (ONCE)**
** **

**(Key**
**Bridging**
**Event)**

**RISK**

***Incorporated in ***
***NPR 7120.5C***

**Develop**
**EVM, **
**Risk & CADRe**
**RFP DRDs**

** ****Translate Risk**
** ****into**
**Cost /Schedule**
**Impacts**

**Do EVM,**
**“****S”-curve,**
**schedule risk **
**critical path **
**analyses, **
**etc.**

- Compile
- End-of-Contract
-  Cost-Risk Data
-  for Evaluation
- & Analysis

_***Preparation***_
- Cost-Risk Feedback:
- Steps 1-5

_***Development***_
- Cost-Risk
- Feedback:
- Steps 6-8

_***Application***_
- Cost-Risk
- Feedback:
- Steps 9-12

**Notes:**
	In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational.  The new focus for cost management at NASA will be Continuous Cost-Risk Management (CCRM).  Additionally, the days of implementing cost management as a set of “stovepipe” activities are over.  In reality, cost management is a series of related cost activities and involves three main Stages that are linked together through a shared set of project risks.  Cost management, in effect, IS the management of cost-risk and can be characterized as continuous feedback on cost-risk.  Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on medium and high-risk drivers alert the project manager that a negative cost trend has been identified and requires action.  The three Stages of Continuous Cost-Risk Management: Preparing for cost-risk feedback; Developing cost-risk feedback; and, Applying cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a continuum of activities interconnected through risk. This CCRM repeats in most acquisition phases.

	The first Stage in Continuous Cost-Risk Management, Preparing for cost-risk feedback, involves NASA project teams performing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., part of the Cost Analysis Data Requirements (CADRe)) and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or “S”-curve).  CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for earliest implementation. These cost/performance trades are the first opportunity for representing the potential cost impacts due to risk.  The CADRe will contain the definition of the project (analogous to the DoD Cost Analysis Requirements Description or CARD) for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation, technical and correlation risk.  Participants in the Preparing for cost-risk feedback Stage of the CCRM are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included.

	The second Stage in Continuous Cost-Risk Management is Developing the feedback to manage the cost-risks.  Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office.  They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, as part of the CADRe, they will be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones (at least annually) as part of the contractual effort.  The “S”-curve products of these requirements will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk. The CADRe will also require that initial key technical parameters, and changes to them over time, be documented along with actual costs associated with all WBS elements. These data will eventually populate the One NASA Cost Engineering (ONCE) database, keeping a record of project cost behavior for updating NASA cost models and available for cost analysis.  Requirements to monitor, manage, and report monthly on the top medium and high-risk WBS elements identified during the Preparing Stage will be included. When required, a monthly earned value report will also be in the RFP, requiring performance measurement, variance analysis and corrective action reporting on all WBS elements, with a special focus on medium and high-risk WBS elements. These reports will require monthly Estimates at Completion (EAC) on all elements with a special focus on medium and high-risk WBS elements. Electronic access to these data will also be required.  Other sources for monitoring and managing the top medium and high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Probability Risk Assessment reports. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods.  If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR).  Participants in Developing cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists.

	The third Stage in cost-risk management is Applying the cost-risk feedback for managing costs.  If the first two Stages in cost-risk management, Preparing for cost-risk feedback and Developing cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will give the project manager the highest quality information possible for managing those WBS elements most likely to drive cost growth. The focus for reporting, analysis and action will be on medium and high-risk WBS elements since they were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal. If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the Preparing for cost-risk feedback Stage above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs.  CADRe-required updates to the initial LCCE estimate at significant milestones (at least annually) can be analyzed for changes (hopefully reductions) in “S”-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to the ONCE database for follow-on contractor performance analysis, cost-risk methodology calibration and updating cost models in order to better cost estimate future projects. Participants in Applying cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating “S”-curves, databases and cost models.

## Continuous Cost-Risk Management (CCRM)

- A cost management architecture providing:
***Identification*** of medium and high risk WBS elements, their assessment & translation of risk into cost-risk in LCCEs
- Supports adequate budget for project
***Communication*** of identified medium and high risk WBS elements to project managers (contractor or NASA)
***Post-cost estimate tracking*** of medium and high risk WBS element cost and schedule performance Application of EVM system
- Produces early warning of potential cost and schedule problems
- Enables actionable intelligence for timely mitigation/management
***Updates*** of technical and cost data (including annual LCCEs)
***History*** of cost and technical data for use in updating cost models

## Continuous Risk Management

## CRM Definition (7120.5C)

**Continuous Risk Management (CRM).  **The process that *identifies* risk; *analyzes* their impact and prioritizes them; develops and carries out plans for risk mitigation or acceptance; *tracks* risk and the implementation of plans; supports informed, timely, and effective decisions to control risks and mitigation plans; and assures that risk information is *communicated* and *documented*.

## Cost-Risk Identification & Assessment

- CRM

## CRM Risk Identification

*1. Search for and locate risks before they become problems*

*2. State risks as, “If ‘*_*event’*_* happens due to *
_*‘*__*cause’*_*, then ‘*_*undesirable consequence’*_* *
*may occur.”*

*4. Draw from expert *
*facilitation and opinion, *
*lessons learned and *
*brainstorming*

*3. Question the consequence until it truly *
*affects the most important requirements*

## Analyze

- Convert risk data into useable information for determining priorities and making decisions

## Likelihood of Occurrence

| Adjective Rating | Numerical Equivalent | Description |
| --- | --- | --- |
| Very High Likelihood | 5 | Occurrence is almost certain, and may not be controlled by following existing processes, procedures, and plans. |
| High Likelihood | 4 | Occurrence is very likely, and may not be entirely controlled by following existing processes, procedures, and plans. |
| Moderate Likelihood | 3 | Occurrence is possible, and may not be entirely controlled by following existing processes, procedures, and plans. |
| Low Likelihood | 2 | Occurrence is unlikely, and may not be entirely controlled by following existing processes, procedures, and plans. |
| Very Low Likelihood | 1 | Occurrence is very unlikely, and is generally controlled by following existing processes, procedures, and plans. |

## Categorizing Risks

**  ****1           2           3         4           5**

** ****Likelihood (Probability)**

**Impact**
**(Consequence, severity)**

**5**
**4**
**3**
**2**
**1**

- Primary
- RISKS

- Examine each risk to isolate the
- cause and determine the
- likelihood and consequence

**Notes:**
Primary Purpose of Slide: 

Key Points:	

## Cost-Risk Identification & Assessment

- CCRM

## Slide 14

## Cost-Risk Assessment in CCRM

- Assessment areas
***Cost model uncertainty***
- Cost estimators handle this
***Input parameter uncertainty***
- Engineering and CRM assessment needed
***Indigenous/Programmatic uncertainty***
- Engineering and CRM assessment needed for application of Relative Risk Weighting (RRW)
- 3 WBS element risk profiles (pessimistic, optimistic & reference) in terms of their Key Engineering Performance Parameters (KEPPs)
***Correlation uncertainty***
- Engineering/CRM/cost estimator assessment needed

## KEPPs as Discrete Risks

- A KEPP is a technical or operational parameter that can be described as a requirement
- An advantage to defining any risk profile in terms of KEPPs is that a more discrete picture of a WBS risk emerges
- e.g., *The **radiation resistance **for a power subsystem’s ASIC** has never been designed for this level of tolerance so there’s likely to be engineering challenges*
- This discrete picture is more intuitively attractive to a project manager than vague statistical notions of risk
*These profiles document the statistical representations of the risks behind the “S”-curve*

## Key Engineering Performance Parameters1 (KEPP) Examples

- KEPPs for new electronic component for a S/C
- Dynamic load resistance
- Operating voltage
- Power regulation
- Radiation resistance
- Emissivity
- Component mass
- Operating temperature range
- Operating efficiency
- KEPPs for a Laser/Amplifier Transmitter
- Wave front sensing
- Wave generation
- Mirror coatings and gratings
- Autonomous resonator alignment
- Bore sighting
- Electrical power generation

***1******The Technology Puzzle: ***
***Quantitative Methods for Developing Advanced ***
***Aerospace Technology; Liam Sarsfield (RAND)***

## CORRELATION

- What is Correlation?2
- A measure of association between two variables
- It measures how strongly the variables are related, or change, with each other
- Engineers and CRM specialists can assist cost estimators in identifying and quantifying correlation between WBS elements

**2**** **_[www.statlets.com/usermanual/glossary.htm](http://www.statlets.com/usermanual/glossary.htm)_

**Notes:**
Correlation should be understood by all cost analysts performing quantitative cost risk analysis.  

So what is correlation? First, it is a measure of association between two variables.  It measures how strongly these variables are related, or change, with each other.  If two variables tend to move up or down together, they are said to be positively correlated.  If they tend to move in opposite directions, they are said to be negatively correlated.  The most common statistic for measuring Association is the Pearson correlation coefficient.  

## Cost-Risk Quantification

- CRM

## CRM Cost-Risk Quantification

- Limited to grassroots/bottoms-up valuation of specific, discrete risk mitigation costs
- Advantage is that defining discrete risks and identifying probabilities for their occurrence is intuitively attractive to the project manager
- Can understand in more concrete terms what she will be getting for her risk dollars
- What her risk dollars are being spent on
- Similar to risk profiles in RRW

## Cost-Risk Quantification

- CCRM

## Cost Model and Input Parameter Uncertainty Cost Quantification 

**Cost Model and Input Parameter Uncertainty Cost Quantification**** **

**   **

**$**

**Cost Driver (Weight)**

*Cost = a + bX**c*

- Input
- variable

- Cost
- Estimate

- Historical data point

- Cost estimating relationship

- Standard percent error bounds

- TECHNICAL RISK

- COMBINED COST MODELING AND TECHNICAL RISK

- COST MODELING UNCERTAINTY

- CER

##  Indigenous/Programmatic Uncertainty Quantification Using RRW

*** *****Indigenous/Programmatic Uncertainty Quantification Using RRW**

***THESE FACTORS ARE THEN APPLIED TO THE RPE***
***     ******TO OBTAIN THE “LOW AND HIGH END” COSTS***

-     RPE

-        2.0 * RPE
- (“HIGH END” COST)

**1.  PESSIMISTIC PROFILE        ****	****HIGH****	****VERY HIGH****	****VERY HIGH            HIGH          ****	****5.9**
**2.  REFERENCE PROFILE****	****MOD****	****MOD****	****MOD****	****MOD ****	****2.9**
**3.  OPTIMISTIC PROFILE****	****LOW****	****MOD LOW****	****MOD ****	****MOD****	****2.0**

**WBS ELEMENT PROFILES     ****	**** 0.35****	****0.25****	****    0.2****	****  0.2****	****  1.0**

**    ****	****	****TECHNLGY****	****DES/ENG                COMPLEXITY       SCHEDULE**

- TOTAL

-      REFERENCE “SCORE”

- =

- 5.9

- 2.9

- =

- 2.0

- =

- LOW END RISK FACTOR
- FOR** S/C**

-                   0.7 * RPE
-      (“LOW END” COST)

-  PESSIMISTIC “SCORE”

- Risk Rating
- Matrix

- Translation
- To Cost

- For Indigenous/Programmatic Risk

***Cost-risk categories***
***(could be more)***

**Notes:**
Since, in most cases, the best information we have is low, most likely and high estimates (or can credibly develop), a triangular distribution is used (bottom of chart).  The development of the low and high ends of the triangle are the result of factors applied to the government point cost estimate (GPE).  A risk category matrix (top of chart) is developed utilizing risk categories and weighted Applying the Analytical Hierarchy Process (AHP), a purely mathematical technique for generating valid, ratio-level (vice ordinal-level) weights.   The AHP is also used to weight the Very Low to Very High scales.  Three profiles of the WBS element are rated against the weighted risk categories Applying the weighted scales and three risk ‘scores’ are generated representing how risky each is perceived to be by the raters, mostly engineers.  Two ratios are developed from these three risk scores and used as factors on the point cost estimate (assumed to be the most likely in the triangle) to generate the low and high ends of the triangular distribution.  
Since the CARD specifications are rated for the Reference Profile and result in the Reference Profile risk score and the CARD specifications result in the Reference Point cost estimate for that WBS element, there is an implied equivalency between the Reference Profile risk score and the Reference Point cost estimate.  In other  words, the Reference Profile risk score represents the WBS in qualitative, technical risk form. The Reference Point cost estimate represents the WBS in cost form.   It is this equivalency that justifies the application of the ratios to the point cost estimate that generates the upper and lower bounds of the cost-risk triangle.

## Correlation

- Dr. Stephen Book (MCR) plotted the theoretical underestimation of percent total cost standard deviation (y-axis) when correlation (x-axis) is assumed to be zero rather than its true value, **.
- In cost estimates we would underestimate % SD ~60%-80% @ 0.2 actual correlation

- From: 1999 Cost
- Risk Analysis Seminar,
- Manhattan Beach, CA

- % Underestimated
- Theoretical
- Total
- Cost Standard Deviation

**Notes:**
This chart shows the percent underestimation of total cost sigma when correlation is ignored, rather than Applying its true value.  Note that there is a knee in the curve for each line.  Each line represents a different number of WBS elements being summed. As this number increases, the actual correlation value where the knee in the curve appears becomes smaller.

## CONVOLVE ALL COST-RISK DISTRIBUTIONS IN MONTE CARLO

**CONFIDENCE**
**     ****LEVEL**

**+**

** ****RPE**

**PROBABILITY**
**    ****DENSITY**

**COST**

**=**

_**SUMMARY COST DISTRIBUTIONS**_**:**

_***CORRELATED***__*** ***__**SUBSYSTEM COST **_
_**DISTRIBUTIONS**_**:**

**COST**

**100**

**50**

**20**

_** **__**BELL CURVE**_

_**“**__**S”-CURVE**_

**RPE**

**+**

**70**

**85**

- RPE

- RPE

***EPS***

***(Power Subsystem ASIC)***

**+**

**RPE**

**CER, **
**Parameter Input,**
**and RRW **
**Cost-Risk**
**Distributions on **
**Lower Level **
**WBS Elements**

**Notes:**
The monte carlo simulation process results in a summary Probability Density Function (PDF) and a summary Cumulative Distribution Function (CDF) from which costs are easily associated with confidence levels visually.  It is these “S”-curves that are generated at the proposal, mid-phase and end-of-phase by the contractors to indicate progress (hopefully) in increasing confidence around the ‘Aggressive Cost Objective’.  If the government has done an analysis itself, then during source selection its “S”-curve can be used to compare with the offerors’.

## Cost-Risk Management in CRM

**Risk management??**

**In simplest terms, Risk Management is an organized process within project management to uncover risks, their likelihood and severity, and deal with them upfront**
**Managing**** ****Risk leads to successful projects**

**Cost-Risk Management??**

**In simplest terms, Cost-Risk Management is an organized process within project management to uncover cost-risks and severity, their likelihood, and deal with them upfront**
**Managing Cost-Risk leads to successful projects**

## Track

- Acquire Data
- Determine what information is required
- Obtain the data
- Compile Data
- Organize data into understandable information
- Develop trends
- Format data in a form that is consistent with what is being used on other projects, and that the reviewers are use to seeing
- Report Data
- Communicate the compiled data to the Risk Management Board
- Make recommendations on the status of each risk and any modifications that may be required

## Control

- Make decisions based on effectiveness of mitigation plans & determine best course of action

**Control**

**• ****evaluate**

**• ****decide**

**• ****execute**

- Project Data

## Cost-Risk Management

- CCRM

## CPR Data Requirements Description

- For cost-risk feedback, the contractor or performing organization needs to be informed in the RFP/Project Plan about:
- Medium and high-risk systems, subsystems and/or WBS elements identified initially in the cost estimate
- EVM performance measurement requirements against these specific risky WBS elements
- e.g., WBS element reporting levels (NPR 7120.5C)
- An EVM CPR DRD template is available on the Cost Estimating Handbook website
-  www.ceh.nasa.gov

## Example of Earned Value DRD InstructionsParagraph 1:  High Risk WBS List & Reporting Criteria

**	****1.  Earned value insight (BCWS, BCWP, ACWP on Format 1 and narrative status on Format 5) ****for the following high risk WBS elements shall be provided every month regardless of variance percentage levels until the system program office (SPO) informs the contractor otherwise****: **
**    **_***Power  Subsystem ASIC***_**; Solar Power Converter; Pointing & Control System**
**    ****Laser Amplifier/Transmitter; Laser Transmit Antenna; Microwave Receive Antenna; Laser Receive Antenna; Tracking & Control System; Laser Conditioning Receiver; Laser Rectifier/Converter; Flywheel Storage System**
**	**
**	****If WBS elements, other than those identified here, begin to experience variances exceeding 10% at one or two levels above the control account (source of risk) for two consecutive months in current month performance measurement, the contractor/performing organization will inform the Project Manager and a consensus reached on adding them to the group of high risk WBS elements identified for monthly cost performance reporting and analysis purposes.   **
**	****All other WBS elements shall have earned value (BCWS, BCWP, ACWP) reported at level 3 of the WBS to satisfy observing and monitoring requirements**

**Example of Earned Value DRD Instructions****Paragraph 1:  High Risk WBS List & Reporting Criteria**

**Notes:**
ACWP	Actual Cost of Work Performed
BCWS	Budgeted Cost of Work Scheduled
BCWP	Budgeted Cost of Work Performed
CDRL	Contract Data Requirements List
SPO	System Program Office
WBS	Work Breakdown Structure

## Traditional Level 3 Reporting10% Variance Reporting

- Traditional Level 3 Reporting***10% Variance Reporting***

- Power Subsystem
- (CWBS Level 3)
- NORMAL REPORTING LEVEL
- ($50M Budget)

- Optical Rectifier
- (CWBS
- Level 4)

_***Optical***_
_***Electronics***_
_***Level 5***__*** ***_

***Normally, if a 10% variance threshold is breached at Level 3 (a $5M problem) a drill-down to lower WBS levels commences to locate the source of the risk for subsequent scrutiny eventually developing to the high-risk ASICs in the <$1M Level 6 CA***

_***Electronics ***_
_***Assembly***_
_***Level 4***_

- Space Solar Power
- Project (CWBS Level 1)

- WBS Drill-Down
- Level

- OBS Drill-Down
- Level

_***ASICs***__*** ***_

***Control ***
***Account*** (Level 6)

***Drill-down***
***from level 3***
***thru level 6 to***
***Level 7 ASICs***
***only ***_***AFTER***_
***problem ***

**Power**
**Subsystem**

**Notes:**
ACWP	Actual Cost of Work Performed
ADACS	Attitude Determination and Control System
AKM	Apogee Kick Motor
BAC	Budget at Completion
BCWP	Budgeted Cost for Work Performed
BCWS	Budgeted Cost for Work Scheduled
EAC	Estimate at Completion
EPS	Electrical Power System
OBS	Organization Breakdown Structure
RWA	Reaction Wheel Assembly
TT&C	Telemetry, Tracking and Control
VP/GM	Vice President/General Manager
WBS	Work Breakdown Structure

Control Account is the logical intersection of the organizational breakdown stricture (who can do the job) with the work breakdown structure (what needs to be done).

## CCRM Med/High Risk ReportingHigh-Risk No-Threshold Variance Reporting3

- CCRM Med/High Risk Reporting***High-Risk No-Threshold Variance Reporting******3***

- Power Subsystem
- (CWBS Level 3)
- NORMAL REPORTING LEVEL
- ($50M Budget)

- Optical Rectifier
- (CWBS
- Level 4)

_***Optical***_
_***Electronics***_
_***Level 5***_
_***($5M Budget)***__*** ***_

***A reasonably low enough reporting level is identified, e.g., Level 5, which must be sensitive enough to indicate the Level 6 ******CA ******cost and schedule problems where the high-risk ASIC effort is located***

_***Electronics ***_
_***Assembly***_
_***Level 4***_

- Space Solar Power
- Project (CWBS Level 1)

- WBS Reporting
- Level

- OBS Reporting
- Level

_***ASICs***__*** ***_

***Control ***
***Account*** (Level 6)

***Report up***
***from level 5***
***to capture***
***level 7 ASICs***
_***BEFORE***__*** big***_
***problem***

**Power**
**Subsystem**

***3******Until risk is no longer a threat or is retired***

**Notes:**
ACWP	Actual Cost of Work Performed
ADACS	Attitude Determination and Control System
AKM	Apogee Kick Motor
BAC	Budget at Completion
BCWP	Budgeted Cost for Work Performed
BCWS	Budgeted Cost for Work Scheduled
EAC	Estimate at Completion
EPS	Electrical Power System
OBS	Organization Breakdown Structure
RWA	Reaction Wheel Assembly
TT&C	Telemetry, Tracking and Control
VP/GM	Vice President/General Manager
WBS	Work Breakdown Structure

Control Account is the logical intersection of the organizational breakdown stricture (who can do the job) with the work breakdown structure (what needs to be done).

## CCRM Cost-Risk Management 

- CPR will deliver *key performance measurement* data on *medium and high risk WBS* elements monthly
- This will enable the project managers to *determine actions to take* to mitigate potential problems
- The *NASA project manager* works closely with the *in-house NASA control account managers (CAMs) and contractor CAMs* to determine what mitigation actions to take
- The *NASA project manager* works with both *in-house and contactor CAMs* to determine if *performance measurement can be dropped* on previously risky WBS elements due to risk retirement

## Conclusion

- Focused only on cost-risk between CRM and CCRM
- There are other areas CRM and CCRM have in common for future discussions
- Cost/performance trades; requirements allocation to functions and WBS elements; risk data collection; risk databases
- Bottom line: CRM and CCRM are complementary processes
- CCRM needs CRM input
- CCRM can provide CRM with valuable cost-risk quantification

## BACKUPS

## CRM Risk Identification

- Clearly define objectives (everyone understands)
- Brainstorm issues/concerns to meeting objectives (what can go wrong)
- Decide which issues/concerns are real
- Develop risk statement from each issue/concern
- Assign attributes to the risk
- Brainstorm potential mitigations
- Be sure to follow rules for brainstorming

## CRM Risk Identification Tools

-  Some tools to help identify and manage Risk:
- Brainstorming (identifying issues)
- Decision Analysis (mapping decision paths)
- Lessons Learned
- Personal knowledge and experience (gut feeling)
- Expert Interviews and Independent Assessments
- Questionnaires. Topic or taxonomy based
- WBS, EVM, budgets and schedules
- FTA, FMEA’s, PRA, Monte Carlo ( reliability tools)
- Safety / Hazard analysis
-  These tools also help identify problems

**Notes:**
Tools help identify risks as well as problems 
For further information on these and other risk identification tools, please refer to the Continuous Risk Management Guidebook, NASA System Engineering handbook (sp-6105 section 4), and the S.E tool box   ******

## Evaluate Risks at the Appropriate Level

## Build a Risk Rating Matrix

**1.  PESSIMISTIC PROFILE           HIGH****	****        VERY HIGH    VERY HIGH           HIGH          ****	****5.9**
**2.  REFERENCE PROFILE            MOD            MOD****	****                MOD                 MOD ****	****2.9**
**3.  OPTIMISTIC PROFILE             LOW****	****        MOD LOW            MOD                 MOD****	****2.0**

**WBS ELEMENT PROFILES      0.35         0.25                 0.2****	****  0.2****	****  1.0**

**    ****	****      ****TRL****	****DES/ENG     COMPLEXITY     SCHEDULE**

- TOTALS

_*Build a Risk Rating Matrix*_

- Indigenous/Programmatic Risk

***Example risk influences***
***(could be more)***

***Need definitions of: pessimistic, optimistic & reference profiles***

## CORRELATION

- What is Correlation?2
- A measure of association between two variables.
- It measures how strongly the variables are related, or change, with each other.
- If two variables tend to move up or down together, they are said to be positively correlated. If they tend to move in opposite directions, they are said to be negatively correlated.
- The most common statistic for measuring association is the Pearson correlation coefficient, P.
- Engineers and CRM specialists can assist cost estimators in identifying and quantifying correlation between WBS elements

**2**** **_[www.statlets.com/usermanual/glossary.htm](http://www.statlets.com/usermanual/glossary.htm)_

**Notes:**
Correlation should be understood by all cost analysts performing quantitative cost risk analysis.  

So what is correlation? First, it is a measure of association between two variables.  It measures how strongly these variables are related, or change, with each other.  If two variables tend to move up or down together, they are said to be positively correlated.  If they tend to move in opposite directions, they are said to be negatively correlated.  The most common statistic for measuring Association is the Pearson correlation coefficient.  

## Indigenous/Programmatic Uncertainty QuantificationRelative Risk Weighting

*Task 1:  Identify and prioritize cost-risk drivers and intensity scales*
*Task 2:  Define pessimistic, reference, and optimistic risk profiles for each WBS element and “score” them against the prioritized cost-risk drivers Applying the intensity scales*
- Task 3:  Create two ratios {pessimistic/reference} and {optimistic/reference} from profile score ratios to scale the reference point _*cost*_ estimate to create pessimistic and optimistic cost estimates
- Task 4: Use results as cost-risk triangular distribution endpoints

**Notes:**
RRW	Relative Risk Weighting
WBS	Work Breakdown Structure

## How the RRW Works

- Reference Point
- Estimate (RPE)

-   Pessimistic*
-    Profile Score

- Optimistic*
-     Profile Score

- Reference*
-    Profile Score

-   Pess/Ref
- RRW Ratio

-   Opt/Ref
- RRW Ratio

- Pessimistic Cost
- Estimate

- Optimistic Cost
- Estimate

- Risk Dimension

- Cost Dimension

- Equivalent Cases

-    *Cost Analysis Data Requirement (CADRe)

- Ratio
- Made

- Ratio
- Made

-  Ratio Applied
- To RPE

-  Ratio Applied
- to RPE

**Notes:**
CARD	Cost Analysis Requirements Description
RPE	Reference Point Estimate
RRW	Relative Risk Weighting

## Track

- Acquire, compile,
- & report data on watched & mitigated risks

***Systematically track and evaluate the progress and effective-***
***ness of risk-handling actions***
***    ******Integrate tracking with performance measurement***
***     ******system    ***

***Select appropriate ***
***metrics to monitor risk***

***Track overall risk environment and specific risks***

## Communicating Cost & Schedule RequirementsCCRM Step 6 

**Develop RFP**
**EVM & CADRe**
**DRDs**

**Communicating Cost & Schedule Requirements****CCRM Step 6 **

**Requirements-to-Mission Allocation**
**Organize estimate:  ****requirements / functions / ****WBS allocation**
**Develop cost and****schedule estimate**
**Analyze risk**
**Translate risk into****cost and schedule ****impacts**
**Communicate cost and****schedule requirements**
**Choose contractor**
**Cement risk management****plans and chosen****contractor**
**Monitor cost / schedule ****performance**
**Update cost, schedule,****and risk**
**Use past data for better****future estimates**
**Perform V&V**

**Perform Cost**
**Benefits Trade**

**Build System**
**Function /**
**WBS Matrix**
**(CADRe)**

**Develop Ref**
**Point Cost &**
**Schedule**
**Estimate**

**Assess WBS**
**Element Risk**

**Assess Cost /**
**Schedule**
**Risk Impact**

**Review Cost**
**Proposal**
**During Source**
**Selection**

**Participate**
**In IBR**
**(Integrated****Baseline**
**Review)**

**5**

**4**

**3**

**2**

**1**

**12**

**9**

**11**

**10**

**Analyze**
**CPR & **
**“****S”-curve**

**Update LCCE**
** ****(CADRe)**
**& Cost / **
**Sched-Risk**
**Assessment**
** **

**Assist in**
**End-of-Contract**
**Data**
**Analysis**
** **

**Assess Data**
**For Model &**
**Database Updates**
**ONCE & CADRe**
** **

**Key**
**Bridging**
**Event**

**6**

- 3

**Notes:**
CARD	Cost Analysis Requirements Description
CPR	Cost Performance Report
EVM	Earned Value Management
IBR	Integrated Baseline Review
LCCE	Life Cycle Cost Estimate
RFP	Request for Proposal
SPO	System Program Office
V&V	Verification and Validation
WBS	Work Breakdown Structure

## Continuous Cost-Risk ManagementSystem of Cost Systems linked together in sequence by the same risks 

**Review Cost**
**Proposal**
**During Source**
**Selection**

- _Continuous Cost-Risk Management_***System of Cost Systems*** linked together in sequence by the same risks

**Perform Cost/**
**Benefit Trades**
**(CAIV is a **
**Subset)**

**Build Reqmts/**
**Function /**
**WBS Matrix**
**(CADRe)**

**Develop Ref**
**Point Cost &**
**Schedule**
**Estimate**

**Assess WBS**
**Element Risk**

**Participate**
**In Post-Contract**
**Award Meeting**

**5**

**4**

**2**

**1**

**12**

**9**

**11**

**10**

**Update LCCE**
**(CADRe) &**
**Cost / **
**Sched-Risk**
**Assessment**
** **

**Assess Data **
**For Model & **
**Database Updates**
** ****(CADRe) & (ONCE)**
** **

**(Key**
**Bridging**
**Event)**

**RISK**

***Incorporated in ***
***NPG 7120.5C***

**Develop**
**EVM, **
**Risk & CADRe**
**RFP DRDs**

** ****Translate Risk**
** ****into**
**Cost /Schedule**
**Impacts**

**Do EVM,**
**“****S”-curve,**
**schedule risk **
**critical path **
**analyses, **
**etc.**

- Compile
- End-of-Contract
-  Cost-Risk Data
-  for Evaluation
- & Analysis

_***Set Up***_
- Cost-Risk Feedback:
- Steps 1-5

_***Get***_
- Cost-Risk
- Feedback:
- Steps 6-8

_***Use***_
- Cost-Risk
- Feedback:
- Steps 9-12

**Notes:**
	In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational.  The new focus for cost management at NASA will be Continuous Cost-Risk Management (CCRM).  Additionally, the days of implementing cost management as a set of “stovepipe” activities are over.  In reality, cost management is a series of related cost activities and involves three main Stages that are linked together through a shared set of project risks.  Cost management, in effect, IS the management of cost-risk and can be characterized as continuous feedback on cost-risk.  Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on medium and high-risk drivers alert the project manager that a negative cost trend has been identified and requires action.  The three Stages of Continuous Cost-Risk Management: Preparing for cost-risk feedback; Developing cost-risk feedback; and, Applying cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a continuum of activities interconnected through risk. This CCRM repeats in most acquisition phases.

	The first Stage in Continuous Cost-Risk Management, Preparing for cost-risk feedback, involves NASA project teams performing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., part of the Cost Analysis Data Requirements (CADRe)) and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or “S”-curve).  CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for earliest implementation. These cost/performance trades are the first opportunity for representing the potential cost impacts due to risk.  The CADRe will contain the definition of the project (analogous to the DoD Cost Analysis Requirements Description or CARD) for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation, technical and correlation risk.  Participants in the Preparing for cost-risk feedback Stage of the CCRM are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included.

	The second Stage in Continuous Cost-Risk Management is Developing the feedback to manage the cost-risks.  Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office.  They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, as part of the CADRe, they will be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones (at least annually) as part of the contractual effort.  The “S”-curve products of these requirements will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk. The CADRe will also require that initial key technical parameters, and changes to them over time, be documented along with actual costs associated with all WBS elements. These data will eventually populate the One NASA Cost Engineering (ONCE) database, keeping a record of project cost behavior for updating NASA cost models and available for cost analysis.  Requirements to monitor, manage, and report monthly on the top medium and high-risk WBS elements identified during the Preparing Stage will be included. When required, a monthly earned value report will also be in the RFP, requiring performance measurement, variance analysis and corrective action reporting on all WBS elements, with a special focus on medium and high-risk WBS elements. These reports will require monthly Estimates at Completion (EAC) on all elements with a special focus on medium and high-risk WBS elements. Electronic access to these data will also be required.  Other sources for monitoring and managing the top medium and high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Probability Risk Assessment reports. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods.  If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR).  Participants in Developing cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists.

	The third Stage in cost-risk management is Applying the cost-risk feedback for managing costs.  If the first two Stages in cost-risk management, Preparing for cost-risk feedback and Developing cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will give the project manager the highest quality information possible for managing those WBS elements most likely to drive cost growth. The focus for reporting, analysis and action will be on medium and high-risk WBS elements since they were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal. If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the Preparing for cost-risk feedback Stage above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs.  CADRe-required updates to the initial LCCE estimate at significant milestones (at least annually) can be analyzed for changes (hopefully reductions) in “S”-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to the ONCE database for follow-on contractor performance analysis, cost-risk methodology calibration and updating cost models in order to better cost estimate future projects. Participants in Applying cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating “S”-curves, databases and cost models.

## Cost-Risk Assessment

- CRM

## Cost-Risk Assessment

- CCRM

## Cost Model Uncertainty

- Due to the “spread” of the underlying CER or analogy data, estimating error must be accounted for:
- Type of distribution is dependent on statistical data available

- CER or
- Analogy
- Estimate

- Analogy
- Estimate

## Input Parameter Uncertainty

- Due to uncertainty about the deterministic nature of an estimated CER-driving parameter value, its indeterminate nature must be accounted for:
- e.g., mass, power, volume, S/W productivity, etc.
- Type of distribution is dependent on statistical data available

- CER
- Estimate

- Analogy
- Estimate

## Define WBS Element Risk Profiles(In Writing)

- Define WBS Element Risk Profiles(In Writing)

- ‘Pessimistic’
- A situation surrounding the development of the WBS element’s Key Engineering Performance Parameters (KEPPs) that assumes the realization of the worst conditions under each category of risk affecting the element in meeting the WBS performance expectations documented in Parts 1 & 2 of the CADRe
- ‘Optimistic’
- A situation surrounding the development of the WBS element’s KEPPs that assumes the realization of the best conditions under each category of risk affecting the element in meeting the WBS performance expectations documented in Parts 1 & 2 of the CADRe
- ‘Reference’
- A situation surrounding the development of the WBS element’s KEPPs that assumes the realization of the most likely conditions under each category of risk in meeting the WBS performance expectations as documented in Parts 1 & 2 of the CADRe

**Notes:**
CARD 	Cost Analysis Requirements Description
RRW	Relative Risk Weighting
WBS 	Work Breakdown Structure

## Indigenous/Programmatic Uncertainty

- May need a unique technique for processing this type of uncertainty
- e.g., Relative Risk Weighting
- A process whereby WBS element-appropriate cost-impacting influences are taken into account in a subjective yet quantitative manner
- Examples of influences;
- TRL; Design/Engineering; Schedule; Integration; Requirements Stability; Complexity; Manufacturing, Supportability, etc.

**Notes:**
RRW	Relative Risk Weighting
WBS	Work Breakdown Structure

## Indigenous/Programmatic Risk

***         ***
***	******	******           ***_TRL_      _DES/ENG_      _COMPLEXITY_       _SCHEDULE _       _TOTAL RISK_
- 		         (0.35)       (0.25)	       (0.2)	     (0.2)           	_SCORE_
- Pessimistic        High*	** *VH	        VH	                   HIGH
-    Profile 		(5*) **	**(7.5)**	**         (6)**	**                    (5.3)*	              	*5.9*
- Reference          MOD       MOD	       MOD	    MOD
-    Profile 		(3*)**	**(3.5)**	**       (2.7)**	**    (2.2)*	              	*2.9*
- Optimistic         LOW         ML	       MOD	    MOD
***  ***Profile		(*1)**	**(2.5)**	**       (2.7)**	**    (2.2)*	    	*2.0*
***Ref Profile Calc: (0.35)(3) + (0.25)(3.5) + (0.2)(2.7) + (0.2)(2.2) = 2.9***

## CCRM Risk Assessment

- Estimators know the uncertainty in cost methodology
- Talk to those involved in performing Continuous Risk Management (CRM)
- Talk to engineers on the project
- They understand the risks in:
- CER input parameters values (e.g., weight)
- Correlations between input parameters and between WBS elements
- Technology state of the art (TRL)
- Designs that use the technologies
- Engineering necessary to implement the technologies used in the designs
- Adequacy of the schedule to design and implement the technologies
- Integration involved at the box, component, subsystem and system levels