# Presentation: 223368

##  NASA Cost Symposium Application of the Cost-Risk Feedback Continuum at NASA1  March 10, 2004JSC

- David R. Graham
- NASA Headquarters
- 300 E Street SW
- Washington, DC 20546-0001
- Code BC
- David.graham-1@nasa.gov

- 1The approach is presently being reviewed as the Cost, Risk and Performance Management
- Integration section for 7120.5C for NASA policy towards cost-risk management.

## Outline

- NASA’s New Emphasis
- NPG 7120.5C: NASA Project Management
- The Cost-Risk Management Problem
- 2nd Law of Thermodynamics (Entropy)
- Evolution and living systems
- The Cost-Risk Management Solution
- Cost-Risk Feedback Continuum
- Implementation through benchmarking

## NASA’s New Emphasis:Improved Budgetary Control

- Need better project management
- Need better cost management
- Sen. John McCain: “When we look back on the past cost of programs at NASA, there has been one constant, and that’s been that the costs have exceeded the initial estimates.  What’s Different?”
- 1 Feb 04 Washington Post following Administrator O’Keefe’s Senate Commerce, Science and Transportation Committee hearing the previous week

## More Bad Press

- Sen. Ernest F. Hollings: “If we’ve learned one thing during the past year, it is that we need a vision that relies on credible certainty, not just technical possibility.” (1 Feb Washington Post)
- “The long series of NASA program over-runs, followed by cancellations, have eroded NASA’s credibility – explanations “don’t wash” and don’t jibe with NPG 7120.5” (House Staff Director, 2003)2
- 7120.5 is NASA Program and Project Management Guidance

- 2 NASA Program and Project Management Processes and
- Responsibilities, NPG: 7120.5”C”); Buschman, Sherry; 10 April 2003

## 7120.5 “C”: Program and Project Management Guidance

- Guidance covers all aspects of program and project management at NASA, including....
- 		“_***Cost, Risk and Performance Management Integration***_”
- Context is the Cost-Risk Feedback Continuum3
- A cost discipline architecture designed for maximizing the quality of cost management information for the Project Manager that:
-  Is a “system of cost systems”
-  Removes the “stove piped” nature of cost disciplines
-  Links cost disciplines together by common risk management   threads
- Manifests the interconnectedness of cost and schedule disciplines in *setting up, getting, and using *cost and schedule risk feedback for successful project and risk management

- 3 “Setting Up, Getting and Using Cost-Risk Feedback”; David R. Graham; May 2002 Risk Management Symposium; Aerospace Corp.

## Metaphorical Illustration of the Cost-Risk Environment using the 2nd Law of Thermodynamics, Living Systems, and the Cumulative Distribution Function 

**The Cost-Risk Management Problem**

**Notes:**
	Before describing the Cost-Risk Feedback Continuum in any real detail, I think it would be instructive to view the cost-risk management environment in terms of the overall problem for which the Continuum is the author’s proposed solution. I will borrow from the field of physics the 2nd Law of Thermodynamics (physicists please cut this cost analyst some slack) and from the field of evolutionary biology (same request from the biologists) to develop a metaphor for the conditions under which cost-risk management must exist. 

## 2nd Law of Thermodynamics (Entropy)

- Entropy?
- What Entropy is (roughly speaking):
***“******FOR A CLOSED SYSTEM, ***
***ORDER TENDS TOWARDS DISORDER”***

**Notes:**
	From the 2nd Law of Thermodynamics I would like to characterize the general nature of project management in terms of entropy.  My working definition for entropy is simply, ‘for a closed system, order tends towards disorder’.  So, without some intervening action, chaos eventually will rule project management. Perhaps a project familiar to the reader resembles this remark?

## Living Systems

- Living systems behave differently than entropic systems
- They evolve
- Evolution is a counterforce to entropy
- Evolution involves processing ***information*** in the form of ***feedback ***to initiate and sustain progress
- ...feedback is the ***fuel*** of evolution
- ...feedback is ***negative ***entropy4

- 4  “The Overview Effect”: Space Exploration and Human Evolution;
-    Frank White; 1987

**Notes:**
	As a counterforce to entropy nature has produced living systems.  Living systems behave differently than entropic systems, they evolve.  Evolution is a counterforce to entropy.  Evolution involves processing information in the form of feedback to initiate and sustain progress.  Feedback is crucial to evolutionary development.  In fact, feedback can be seen as being the fuel of evolution producing the counterforce to entropy, negative entropy.  Here, obviously for evolution to occur, negative entropy is a good thing.

## 4 General Phases of Evolution

- Pre-Equilibrium
- Positive entropy >> negative entropy
- Equilibrium
- Positive and negative entropy are equal
- Change
- Original system is recognizable but is different
- Transformation
- Living system has adapted to new environment for survival
- Less recognizable

**Notes:**
	To continue my metaphor I need to describe the four general phases of evolution.  The first phase is “pre-equilibrium” and exists whenever positive entropy is greater than negative entropy.  The second phase is, you guessed it, “equilibrium”.  In this phase positive and negative entropy appear to be equal.  I say “appear” to be equal because, if evolution is truly occurring, this is only a temporary phase because the living system is learning from feedback in order to adapt to any new conditions and the first step is to halt the tendency to disorder.  Kind of like treading water.  The third phase of evolution is “change”.  The living system has changed slightly in adapting to the new conditions but still is recognizable but something is different.  The fourth and final phase of evolution is “transformation” where the living system has adapted to the new environment for survival and, in this phase, is much less recognizable from its earlier appearance.

## What’s Any of This Got To Do With Cost-Risk Management?

- A good cost-risk management system
- _IS_
- a living system!
- and....
- It is fueled by _***feedback***_ in order to improve

**Notes:**
	What’s any of this got to do with cost-risk management?  Here’s the punch line: a good cost-risk management system is a living system which is fueled by feedback in order to improve. 

## Cost-Risk Management Systems Phases of Evolution 

- Pre-Equilibrium
- Equilibrium
- Change
- Transformation
- ....using “S”-curves as the metric

**Notes:**
	Let me illustrate the four phases of a cost-risk management system’s evolution, from pre-equilibrium, equilibrium and change through transformation using the cumulative distribution function as the metric.

## Phase: PRE-EQUILIBRIUM 1st Contract Cost “S”-Curve (Positive Entropy >> Negative Entropy)

**Phase: PRE-EQUILIBRIUM ****1****st**** Contract Cost “S”-Curve**** ****(Positive Entropy >> Negative Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***ProjectBudget***

**Notes:**
	In the beginning of the Pre-Equilibrium phase, the government’s estimate (in reality a project/ICE reconciled CDF) as illustrated in Fig X, indicates the selected confidence level dollar value (in Fig X the 50% confidence level).  This CDF represents the best result from a cost estimate and cost-risk assessment as to the cost of a particular project. 

## Evolution Phase: PRE-EQUILIBRIUM 2nd Contract Cost “S”-Curve (Positive Entropy >> Negative Entropy)

**Evolution Phase: PRE-EQUILIBRIUM ****2****nd**** Contract Cost “S”-Curve**** ****(Positive Entropy >> Negative Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***Project Budget***

**Mid-Contract Cost-**
**Risk Distribution**

**Notes:**
	As the project proceeds, as illustrated in Fig X1,  it becomes clear that control over the risk impacts to cost is not successful as the CDF shifts to the right and becomes less vertical.  This behavior is indicative of risks not only not being retired but increasing their negative cost effects.

## Evolution Phase: PRE-EQUILIBRIUM 3rd Contract Cost “S”-Curve(Positive Entropy >> Negative Entropy)

**Evolution Phase: PRE-EQUILIBRIUM ****3****rd**** Contract Cost “S”-Curve****(Positive Entropy >> Negative Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***ProjectBudget***

**Mid-Contract Cost-**
**Risk Distribution**

**Near-End of Contract**
**Cost-Risk Distribution**

**Notes:**
	At the third LCCE update in Fig X2, it is apparent that the risks have not been retired and, in fact, are continuing to have a deleterious effect on the costs. The CDF has shifted further to the right and has even gotten ‘flatter’ indicating that not only were the basic assumptions of the 50% cost estimate wrong but the assessments of the risks were also understated. If this looks like any programs you are familiar with it is strictly coincidental, however, cost behavior such as that illustrated is consistent with a Pre-Equilibrium evolutionary state where there really is no cost-risk management system learning from potential feedback. 

## Phase: EQUILIBRIUM1st  Contract “S”-Curve (Positive Entropy = Negative Entropy)

**Phase: EQUILIBRIUM****1****st****  Contract “S”-Curve**** ****(Positive Entropy = Negative Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***ProjectBudget***

**Notes:**
	If the underlying evolutionary situation is the second phase, that is, Equilibrium, the CDF behavior illustrated in Fig X3 is expected.  In this case, the government’s CDF estimate begins in the same position as in the Pre-Equilibrium case.  

## Phase: EQUILIBRIUM1st & 2nd  Contract “S”-Curves (Positive Entropy = Negative Entropy)

**Phase: EQUILIBRIUM****1****st**** & 2****nd****  Contract “S”-Curves**** ****(Positive Entropy = Negative Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***Project Budget***

**Mid-Contract Cost-**
**Risk Distribution**

**Notes:**
However, the 2nd and 3rd updates to the CDF produce a mirror image of the CDF indicating that a stabilization of cost-risk management has occurred as illustrated in Figures X4 & X5.

## Phase: EQUILIBRIUM1st, 2nd, & 3rd  Contract “S”-Curves (Positive Entropy = Negative Entropy)

**Phase: EQUILIBRIUM****1****st****, 2****nd****, & 3****rd****  Contract “S”-Curves**** ****(Positive Entropy = Negative Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***ProjectBudget***

**Mid-Contract Cost-**
**Risk Distribution**

**Near-End of Contract**
**Cost-Risk Distribution**

**Notes:**
	 Cost behavior such as that illustrated in Fig X5 is consistent with an evolutionary state in Equilibrium where the cost-risk management system is beginning to learn from potential feedback

## Phase: CHANGE 1st  Contract “S”-Curve (Negative Entropy > Positive Entropy)

**Phase: CHANGE ****1****st****  Contract “S”-Curve**** ****(Negative Entropy > Positive Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***Project Budget***

**Notes:**
	In the third evolutionary phase, Change, the government’s initial CDF again starts in the same location as illustrated in Fig X6 but in Change, negative entropy is greater than positive entropy.   

## Phase: CHANGE 1st & 2nd  Contract “S”-Curves (Negative Entropy > Positive Entropy)

**Phase: CHANGE ****1****st**** & 2****nd****  Contract “S”-Curves**** ****(Negative Entropy > Positive Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***Project Budget***

**Mid-Contract Cost-**
**Risk Distribution**

**Notes:**
	The first CDF update indicates a reduction in variance has occurred as illustrated in Fig X7.  This is identified by the change in relative steepness in the updated CDF relative to the initial CDF.  This indicates that the risks and cost-risks are being retired.  The CDF is the metric that indicates this by a vertical shift in its position relative to the initial range represented by the initial CDF.  

## Phase: CHANGE 1st, 2nd, & 3rd  Contract “S”-Curves (Negative Entropy > Positive Entropy)

**Phase: CHANGE ****1****st****, 2****nd****, & 3****rd****  Contract “S”-Curves**** ****(Negative Entropy > Positive Entropy)**

**100**

**70**

**25**

**Govt’s Initial Cost-Risk**
**Distribution**

**Confidence**
** ****Level**

**Cost **

***   ******Contract ***
***    ******Target***
***      ******Cost***

***Project Budget***

**Mid-Contract Cost-**
**Risk Distribution**

**Near-End of Contract**
**Cost-Risk Distribution**

**Notes:**
	The second CDF update, illustrated in Fig X8, shows a continuation of the increases in steepness of the CDF, indicating a reduction in the possible range of probable costs (variance reduction) due to the success of the cost-risk management system.  The cost-risk management system is utilizing feedback about the effects of the risks creating the opportunity for project management to utilize its information content to react to risk challenges successfully.  Even though the initial 50% estimate has not changed through two updates, the variance has changed indicating success in reducing the negative effects of risk on cost.    

## Evolution Phase: TRANSFORMATIONAL1st  Contract Cost “S”-Curve (Negative Entropy >> Positive Entropy)

**100**

**70**

**25**

**Confidence**
** ****Level**

**Cost**

***   ******Contract ***
***    ******Target***
***      ******Cost***

***Project Budget***

**Evolution Phase: TRANSFORMATIONAL****1****st****  Contract Cost “S”-Curve**** ****(Negative Entropy >> Positive Entropy)**

**Govt’s Initial Cost-Risk**
**Distribution**

**Notes:**
	In the final example of evolutionary development, Transformational, we start again with the same initial CDF as illustrated in Fig. X8.

## Evolution Phase: TRANSFORMATIONAL1st & 2nd  Contract Cost “S”-Curves (Negative Entropy >> Positive Entropy)

**100**

**70**

**25**

**Confidence**
** ****Level**

**Cost**

**Mid-Contract Cost-**
**Risk Distribution**

***Project Budget***

**Evolution Phase: TRANSFORMATIONAL****1****st**** & 2****nd****  Contract Cost “S”-Curves**** ****(Negative Entropy >> Positive Entropy)**

**Govt’s Initial Cost-Risk**
**Distribution**

***   ******Contract Target Cost***

**Notes:**
	By the time of the first update, as depicted in Fig. X9, not only has the CDF gotten steeper relative to the initial CDF, it has shifted to the left, indicating so much success in retiring risk that not only has the range of possible costs been reduced, the 50% estimate has now been reduced.  

## Evolution Phase: TRANSFORMATIONAL1st, 2nd & 3rd Contract Cost “S”-Curves (Negative Entropy >> Positive Entropy)

**100**

**70**

**25**

**Confidence**
** ****Level**

**Cost**

***   ******Contract ***
***    ******Target***
***      ******Cost***

**Mid-Contract Cost-**
**Risk Distribution**

**Near-End of Contract**
**Cost-Risk Distribution**

***Govt Budget***

**Evolution Phase: TRANSFORMATIONAL****1****st****, 2****nd ****& 3****rd**** Contract Cost “S”-Curves**** (Negative Entropy >> Positive Entropy)**

**Govt’s Initial Cost-Risk**
**Distribution**

**Notes:**
	As illustrated in Fig. 12, the second CDF update continues this trend of ever-steepening CDFs and shifting to the left even more.  The cost-risk management system is producing an optimum level of actionable information from the feedback mechanisms in place such that project management can react so quickly to the possible negative trends that they never materialize.  Such insight is being produced that actions taken are effective in exactly the right places at exactly the right times to make these positive changes occur.  

## Evolution Phase: TRANSFORMATIONAL1st, 2nd & 3rd Contract Cost “S”-Curves (Negative Entropy >> Positive Entropy)

**100**

**70**

**25**

**Confidence**
** ****Level**

**Cost**

***   ******Contract ***
***    ******Target***
***      ******Cost***

**Mid-Contract Cost-**
**Risk Distribution**

**Near-End of Contract**
**Cost-Risk Distribution**

***Govt Budget***

**Evolution Phase: TRANSFORMATIONAL****1****st****, 2****nd ****& 3****rd**** Contract Cost “S”-Curves**** (Negative Entropy >> Positive Entropy)**

**Govt’s Initial Cost-Risk**
**Distribution**

***Actual***

**End of Contract**
**Cost-Risk Distribution**

**Notes:**
	As illustrated in Fig. 12, the second CDF update continues this trend of ever-steepening CDFs and shifting to the left even more.  The cost-risk management system is producing an optimum level of actionable information from the feedback mechanisms in place such that project management can react so quickly to the possible negative trends that they never materialize.  Such insight is being produced that actions taken are effective in exactly the right places at exactly the right times to make these positive changes occur.  

## How to Get from Pre-Equilibrium to Transformational?

- Cost-Risk Feedback Continuum

**Notes:**
	So, how do we get from a situation of Pre-Equilibrium to Transformational as I have described them?  NASA project and program management is tentatively exploring the implementation of the Cost-Risk Feedback Continuum to evolve its acquisition management results to a situation that more closely resembles the Transformational than the Pre-Equilibrium state.

## The Cost-Risk Feedback ContinuumA System of Cost Systems linked together in sequence by the same risks 

_**The Cost-Risk Feedback Continuum**_**A ***System of Cost Systems*** linked together in sequence by the same risks**** **

**Perform Cost/**
**Benefit Trades**
**(CAIV)**

**Build Reqmts/**
**Function /**
**WBS Matrix**
**(CARD)**

**Develop Ref**
**Point Cost &**
**Schedule**
**Estimate**

**Assess WBS**
**Element Risk**

**Review Cost**
**Proposal**
**During Source**
**Selection**

**Participate**
**In Post-Contract**
**Award Meeting**

**5**

**4**

**3**

**2**

**1**

**12**

**9**

**11**

**10**

**Update Cost**
**Estimate**
**& Cost / **
**Sched-Risk**
**Assessment**
** **

**ONCE DB;**
**Assist in**
**End-of-Contract**
** ****Data Analysis; **
**BCC & ECC**
**Calibrations**
** **

**Evaluate Cost**
**& Schedule**
**Models**
** **

**(Key**
**Bridging**
**Event)**

- 12 cost activities,
- traditionally treated as
- stovepipes,
_**NOT**_
- stovepipes
- in the Continuum

***Incorporated in ***
***NPG 7120.5C***

**EVM Analyses **
**& **
**Cost Estimating **
**Data **

**Develop RFP**
**EVM, LCCE, etc.**
**DRDs**

** ****Translate Risk**
** ****into**
**Cost /Schedule**
**Impacts**

**Notes:**
	In order to meet the space project cost challenges for the next decade and beyond, NASA cost management processes must evolve from traditional methods to modes that are truly transformational.  The new focus for cost management at NASA will be cost-risk management.  Additionally, the days of implementing cost management as a set of “stovepipe” activities are over.  In reality, cost management is a continuum of related cost activities and involves three main steps that are linked together through a shared set of project risks.  Cost management, in effect, IS the management of cost-risk and can be characterized as a cost-risk feedback “continuum”.  Feedback is essential to the transformation of cost management into a dynamic, continually reacting system where focused reporting of metrics on high-risk drivers alert the project manager that a negative cost trend has been identified and requires action.  The three steps of this cost-risk feedback continuum: setting up for cost-risk feedback; getting cost-risk feedback; and, using cost-risk feedback, occur at different points in time during an acquisition phase and involve the collaboration among cost estimators, project engineers, project managers, procurement analysts and Earned Value Management (EVM) specialists in managing the challenges presented by the risks. Cost management is NOT a grouping of unrelated stove-piped cost activities but is a "system of cost systems" based on viewing 12 cost activities normally treated as stovepipes as a "continuum" of activities interconnected through risk. This continuum repeats in most acquisition phases.

	The first step in the cost management continuum, setting up for cost-risk feedback, involves NASA project teams doing three main activities: cost/performance trades (e.g., Cost as an Independent Variable (CAIV)); developing a definition of the program (e.g., Cost Analysis Requirements Description (CARD)); and, producing a range of possible costs (e.g., probability density function (PDF) and cumulative distribution function (CDF) or “S”-curve).  CAIV trades flow out of a well-defined Concept of Operations (CONOPS) and demonstrate a commitment to evolutionary acquisition by being initiated in pre-Phase A for maximum results.  The CARD will contain the definition of the project for use by cost estimators where the traceability from Work Breakdown Structure (WBS) element, through functions, to initial requirements will be clearly identifiable. The cost range production exemplified in a PDF and CDF involves developing a reference point cost estimate from a cost model (e.g., NAFCOM, PRICE, SEER, etc.), and incorporating cost model estimation risk, technical risk identification/assessment and dependency (i.e., correlation) risk.  Participants in the setting up for cost-risk feedback step of the continuum are mainly cost estimators, project engineers and project managers. This represents the starting point for cost-risk management. From this point forward the challenge will be in managing to the cost level chosen, no matter what cost-risk margin has been included.

	The second step in cost management is getting the feedback to manage the cost-risks.  Since hardware contractors are selected to develop NASA systems, they must be informed about the potential cost-risk impacts identified by the NASA cost teams for their attention, monitoring, management, and reporting to the NASA project office.  They must be informed in the Request for Proposal (RFP) Data Requests (DR) to produce multiple products that reflect the status and trends of these potential cost-risks. For example, they should be required to produce a Life Cycle Cost Estimate (LCCE) for the proposal and LCCE updates at significant contract milestones as part of the contractual effort.  These products will enable the measurement of variance changes in the cost-risk distribution over time reflecting the management of risk and cost-risk.  Requirements to monitor, manage, and report monthly on the top high-risk WBS elements identified during the setting up for cost-risk feedback step should be included.  Sources for monitoring and managing the top high-risk WBS elements can come from Technical Performance Measure (TPM), Risk Management, Technical Interchange Meeting (TIM), Integrated Product Team (IPT) and Earned Value Management (EVM) reports.  The requirement for the production of a monthly report on WBS element cost and schedule performance should also be in the RFP, describing variances from plans with explanations as to what is driving them from a technical or other perspective along with what the contractor is doing about them. These reports should require monthly Estimates at Completion (EAC) on all elements with a special focus on high-risk WBS elements. Based in part on the way bidders address these RFP requirements in the cost proposals submitted, NASA will select a winning bidder and set up a post-award meeting with the selected contractor to verify the proposed cost-risk management methods.  If EVM is required on the effort, the meeting at which this discussion takes place (along with baseline validation) is called an Integrated Baseline Review (IBR).  Participants in getting cost-risk feedback are the cost estimators, project engineers, project managers, procurement analysts and EVM specialists.

	The third step in cost-risk management is using the cost-risk feedback for managing costs.  If the first two steps in cost-risk management, setting up for cost-risk feedback and getting cost-risk feedback, have been properly accomplished, the cost-risk feedback from the EVM (or similar) system, supported by TPM reports, Risk Management Reports, TIM minutes, IPT meeting minutes, etc., will contain the highest quality information possible for managing risk and cost-risk. The focus for reporting will be on high-risk WBS elements since the high-risk WBS elements were identified for specific reporting in the RFP and addressed by the winning contractor in his proposal.  If cost and schedule performance analysis indicates problems, a decision to reiterate a cost/performance trade (part of the setting up for cost-risk feedback step above) may have to be made, for a possible adjustment to a system requirement. EVM performance analysis, focused on risk impacts to cost and schedule, will enable development of monthly EACs providing the project manager crucial feedback on the potential cost effects of the risks. This information provides the project manager with focused insight into the cost-risk in order to better manage his/her costs.  Required updates to the initial LCCE estimate at significant milestones can be analyzed for changes (hopefully reductions) in “S”-curve variances, indicating progress in managing risks and cost-risks. At the end of the effort a volume of high-quality cost, risk, and cost-risk information will have been collected that can be added to existing cost and cost-risk databases for updating cost models in order to better cost estimate future projects. Participants in using cost-risk feedback are primarily project engineers, project managers and EVM specialists with cost estimator involvement during cost/performance trades (if required), and updating “S”-curves, databases and cost models.

## DRDs Modified 

- Earned Value Management System Description
- Cost Performance Report
- Risk Management Plan & Reports
- PRA Plan and Reports
- Financial Management Reports (533M&Q)
- Project Integrated Master Schedule
- Cost Analysis Data Requirement (CADR)
- Standardized Work Breakdown Structure (WBS)
- Cost Analysis Requirements Description (CARD)
- Life Cycle Cost Estimate (LCCE)
- Cost Estimating Data Report (CEDR)

## The Cost-Risk Feedback ContinuumA System of Cost Systems linked together in sequence by the same risks 

**Perform Cost/**
**Benefit Trades**
**(CAIV)**

**Build Reqmts/**
**Function /**
**WBS Matrix**
**(CARD)**

**Develop Ref**
**Point Cost &**
**Schedule**
**Estimate**

**Assess WBS**
**Technical**
**Risk**

** ****Translate Risk into**
**Cost /Schedule**
**Impacts**

**Develop RFP**
**EVM, LCCE, etc.**
**DRDs**

**Review Cost**
**Proposal**
**During Source**
**Selection**

**5**

**4**

**3**

**2**

**1**

**12**

**9**

**11**

**10**

**EVM Analyses **
**& **
**Cost Estimating **
**Data **

**Update Cost**
**Estimate**
**& Cost / **
**Sched-Risk**
**Assessment**
** **

**Assist in**
**End-of-Contract**
** ****Data**
**Analysis**
** **

**Evaluate Cost**
**& Schedule**
**Models**
** **

**(Key**
**Bridging**
**Event)**

_**Feedback is Key**_
**(Electronic approach is proposed)**

**The Continuum **
**environment provides feedback for cost-risk management **
**to evolve**

**Participate**
**In Post-Contract**
**Award Meeting**

**Notes:**
	The flow of cost-risk feedback information not only flows in a simple sequence from step to step in the implementation of the Continuum, but there can and should be a dynamic flow of information between many Continuum steps during, after and before subsequent efforts as illustrated in Fig. 17 below.  	

Examples of feedback between Continuum steps:

	- CAIV trades inform RPE, tech risk, and tech risk-to-cost impact steps;
	- CPR step may require CAIV trade, which may prompt an update to CARD due to requirement or performance level changes,
           which may require an adjustment to the basic RPE
	- Cost estimator gets feedback from end-of-contract CPR cost data to update his model for improved future estimates
	- Cost estimator needs latest information on risk of a WBS element he estimated 9 months ago (for example) and contacts 
	  the CPR analyst monitoring the cost performance of that WBS element on an ongoing contract.  CPR analyst, who has 
	  electronic access to contractor’s EVM system’s work package that has that high risk WBS element in it, gives the cost 
           estimator electronic access to view current cost performance of that WBS element giving the cost estimator new insight on
           its risk for translation into cost impact;
	- Insight for translating risk into cost impacts also comes from step 11 (technical risk-driven and external programmatic-
           driven risk) and 12, final end-of-contract actual cost CPR data on cost-risk performance of high and medium risk WBS
           elements; 
	-  Step 11, end-of-contract technical risk-driven and external programmatic-driven risk, informs the team developing the
	   narratives for Data Requirements for RFP solicitation documents;
	- Step 11, end-of-contract technical risk-driven and external programmatic-driven risk, informs the team reviewing the
	  bidder cost proposals for bidder past performance credibility relative to cost proposal claims;
	- Step 5, translation of risk into cost impacts informs Step 8, IBR visit, by ability to present to contractor control account
 	  managers the risks identified by government team in the development of the cost-risk for contractor acknowledgement
	  of responsibility to manage those risks;
	- Step 9 informs Steps 11 & 12 with cost performance cost-risk WBS element data, technical risk-driven and external 
	  programmatic-driven cost growth causes which can be used to create calibration factors for use during source selection. 
	- Step 9 data from CPRs can also be useful to cost estimators updating their cost models. 
	- Step 10, updates to RPE and cost-risk informs Steps 3,4,& 5 with an updated cost and cost-risk assessment;
	- Step 10 also informs the CPR Format 5 since any changes to initial cost-risks tracked in EVM system will be identified on
	  Format 5.  These changes could represent an elimination of tracking of some medium or high risk WBS element due to
 	  successfully meeting the risk challenges and/or adding a new medium or high risk WBS element for special tracking due
	  to unexpected poor performance discovered during the effort;

## NASA Project Cycle Acquisition Phases

- IOC

- Conceptual Definition

-  Phase A

-  Conceptual Design

- Pre-Phase A

-  Phase B

- Preliminary Design

- NAR

- Design, Development,
- Test & Evaluation

- Phase C/D

- Preliminary
- Design
- Review

- Critical
- Design
- Review

- ATP

**NASA Project Cycle Acquisition Phases**

- Preliminary CARD

- Full CARD

- CARD Updates as Necessary

- Pre-NAR

**Notes:**
Only the cost/performance (CAIV) trades step of the Continuum is expected to be applied during Pre-Phase A.  However, in all subsequent phases, the entire Continuum’s 12 steps can be applied during the project’s management/

## Total Cost Growth for Two Space Programs5

- 5 “The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale
- Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)

- Quantitative Framework

- Addressed by Continuum: 70%

**Notes:**
The point of this chart is to show that the Continuum is designed to identify, monitor and measure the behavior of cost-risk in the very areas pointed out by the Booz, Allen, Hamilton (BAH) study to be driving 70% of space cost growth. BAH has, over the past few years, mounted a concerted effort to identify the drivers of space cost growth.  Using two space cost programs they identified where their cost growth came from as a percentage of the total cost growth.  The areas addressed by the cost-risk feedback continuum are identified on the chart by the arrows and are requirements generation and translation, cost estimation, underestimation of risk and schedule slips.  Each of these areas has an underpinning of risk that drives the cost growth.  In the requirements area it is the extra risks in trying to achieve objective performance capability values rather than threshold values and requirements growth.  These risks express themselves as cost impacts (i.e., cost-risk) and result in cost growth.  The Continuum is designed to constantly address and re-address the cost/performance trades (i.e., CAIV trades) triggered by cost trends monitored and then acted upon during the effort. Cost growth is also driven in the requirement area by the risk of letting new requirements “creep” into the program over time.  If no extra budget is allocated then it not only is cost growth but there is also a potential for overruns.  Cost estimation problems revolved around software estimating, outdated data and inaccurate assumptions.  Risk in cost estimation manifests itself in three ways: First, there is error in the cost estimation model itself and so must be accounted for when the estimate is made; second, all other areas of “known unknown” risk must be accounted for in the cost estimating process, that is, technology maturity, design/engineering risk, integration risk, complexity, manufacturing risk, etc.; and third, the dependencies between the WBS elements being estimated, i.e., correlation risk.  Absent these considerations of risk the cost estimate will understate the cost.  The assessment of the risk itself and its translation into cost and schedule impacts is the second and third areas the Continuum addresses.  There needs to be a structured analysis of the risks and a very careful translation from that assessment into cost impacts for the cost estimate to be accurate (or at least credible).  The Continuum promotes a strong risk analysis process so that the cost impacts derived from it can accurately portray the cost and schedule impacts.  The fourth area of space cost growth that the Continuum addresses is schedule slips, driven by the government and contractor.  Beginning with writing the risk tracking requirements into the RFP, the Continuum sets up the ability to receive feedback via Earned Value Management (EVM) reporting, for example, on what is driving the schedules (and who).  Often the EVM information on schedule slips can trigger attention on the network schedule where potential or actual schedule slips can be analyzed for impact and trigger cost/schedule/performance trades to utilize this feedback and adjust. Schedule slips cause cost growth due to the carrying of “standing armies” of contractors.   The total value of these areas with respect to total space cost growth is 70%.  This chart is not to be interpreted as that if the Continuum were applied there would be a 70% reduction in space cost growth.  It should be interpreted as that if the Continuum were applied there would be logical expectations, due to focused attention on the risks and cost-risks, that a reduction to some degree in the magnitude of space cost growth can be achieved.  

## Program Management Capabilities1

- “As one would expect in an era of TSPR2, our study showed ***higher*** program management scores for the ***contractors*** associated with the examined programs than for their government partners”
- “Our experience and research have shown us that program management ***benchmarking ***and ***follow-on corrective actions*** to develop and maintain program management processes are the best ways to improve program management capabilities”

- 1The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale Systems in an Increasingly Complex Environment (Booz|Allen|Hamilton)
-  2Total System Performance Responsibility – granted to Air Force contractors during Acquisition Reform

## Implementation Challenge: Achieving Continuum Optimality

- Benchmarking the Project Offices
- A knowledgeable entity with the requisite authority (e.g., IPA*, NAR*) applies a ***Cost-Risk Feedback Continuum (CRIFCON) Criteria*** to the project office to establish their level of program management capability/competency
- Rate program/project management on all continuum steps and work to achieve all “5” ratings over time on all 12 steps (1-to-5 scale)

- * Proposed

**Notes:**
	Implementation of the Cost-Risk Feedback Continuum throughout the NASA project and program management discipline will not happen by osmosis.  A proven strategy of injecting new modes of organizational behavior is the benchmarking approach.  This approach is suited to the implementation challenge of what I would term ‘achieving Continuum optimality’.  

	The proposal at the present time within NASA project management is to identify exactly where each project management team is relative to optimal implementation of the Continuum.  The first step needed is to develop a set of criteria against which each project management team can be objectively measured that indicates how well it is implementing the step of the Cost-Risk Feedback Continuum.  In this approach, each Continuum step has a number of ‘key elements’ that, if practiced by the project management team, will indicate its level of optimal Continuum implementation for that step.  If there are five key elements for a particular Continuum step, and each element is equally weighted, then a ‘score’ of 5 indicates optimal implementation of that step of the Continuum.  If a scale of 1 to 5 is used for all Continuum steps, 1 being lowest and 5 being highest, then an average of all scores for all 12 Continuum steps would give a measure of how close to optimal a particular project management team is in implementing the Cost-Risk Feedback Continuum.  An acronym might be useful at this point to designate a quick way to relatively place a project management team on such a scale such as “CRIFCON 4”, CRIFCON being a shortened version of Cost-RIsk Feedback CONtinuum, indicating that over all 12 Continuum steps that team received an average score of “4” out of 5 possible.  A project management team with a score of “4” may well represent the best project management team at NASA and hence would become the benchmark for all other teams to strive to meet or beat. 

	An obvious question arises as to who will have the authority to rate the teams?  NASA is looking at having a knowledgeable entity with the requisite authority such as the Independent Program Assessment (IPA) review team, the Non-Advocate Review team, an ad hoc ‘greybeard’ project management team set up by the Chief Engineer’s Office to rate project management teams, or more of a self-assessment approach somehow supervised by the Chief Engineer’s Office. 

## Challenge: Implementation Achieving Continuum Optimality

- Hypothesis:
- Those project offices closer to being rated at a “5” level for every Continuum step have a more likely chance to minimize the difference between final actual costs and beginning target costs
- CRIFCON Levels
- A program/project with a “CRIFCON 2” level does not have as mature a program management capability as a program/project with a “CRIFCON 4” level

**Notes:**
	The underlying hypothesis for Continuum implementation is that those project offices closer to being rated a “5” level for every Continuum step have a more likely chance to minimize the difference between final actual costs and beginning target costs.

## CRIFCON Criteria

- 	Key Elements
- -         All missions identified and prioritized
- -         Multiple levels of each requirement identified
- -         Each requirements levels’ effectiveness rated against missions
- -         Multiple requirement levels costed with risk impacts included
- -         Analysis to determine optimum, cost-constrained requirements levels “bundle”
- -         e.g., knee-in-the-curve; incremental benefit/cost
- Scoring
- -     “1” = Accomplishing one key element
- -         “2” = Accomplishing two key elements
- -         “3” = Accomplishing three key elements
- -         “4” = Accomplishing four key elements
- -         “5” = Accomplishing five key elements

*Example of criteria for Step One of the Continuum*

**Notes:**
This chart contains an example of Step One of the CRIFCON’s 12 Steps being used as criteria for rating the capability of a project office’s management team.  Each criteria has a number of “Key Elements” derived from the activities representing the successful implementation of that particular CRIFCON Step.  A linkage is made from these Key Elements to a rating scale that reflects the project management team’s Key Element accomplishment.  The more Key Elements the team is accomplishing, the higher the team’s rating.

## Implementation Challenge: Achieving Continuum Optimality(Suggested Implementation Plan)

- Apply on new developments (e.g., Code T)
- Corporate NASA Team works with programs/projects to educate, rate, and strive over time for project achievement of Continuum optimality (i.e., CRIFCON “5”)
- Develop an electronically-based collaborative networking environment to facilitate interaction between personnel involved in implementing Continuum steps
-  Improves education for ultimate CRIFCON level-5 achievement

**Notes:**
	Since the number of project management teams at NASA is quite large, a complete CRIFCON rating of all these teams is probably impractical.  The target project management teams NASA is considering for CRIFCON rating are those with the responsibility for new development projects and programs.  Priority will be given to those teams whose projects have unusually large cost targets, are highly visible and have a high degree of risk.  In accomplishing the CRIFCON ratings, Code BC (Cost Division), Code D (Chief Engineer), Enterprises, IPAO and Center SMOs will work with projects/programs to educate, rate, and strive over time for project management achievement of Continuum optimality (i.e., CRIFCON “5”).

	To further enhance Continuum implementation, Code BC is working to develop an electronically-based collaborative networking environment to facilitate interaction between personnel involved in implementing Continuum steps. A system such as this will accelerate Continuum education for ultimate CRIFCON level-5 achievement.  It can be viewed as part of OMB and Bush administration’s R&D initiative for improving networking and IT R&D.

## Conclusion

- NASA hopes to improve budget credibility through more efficient project and cost management
- 7120.5 “C” guidance revision
- Cost-Risk Feedback Continuum approach
- Focus on cost-risk management
- Achieve at least “equilibrium” or “change” cost-risk management evolutionary state
- Implement by benchmarking to top “CRIFCON” project teams