# Presentation: 235225

## 2004 NASA Cost Analysis Symposium “Cost Analysis:  Launch Pad for One NASA”Johnson Space Center (JSC), Houston Texas March 10, 2004

- Richard W. Fox, Manager
- Assessments, Cost Estimating and Schedules Office International Space Station Program Planning and Control Office

- Space Station Assessment Implementation

## Goals of Presentation

- Provide insight into the implementation and operation of the International Space Station Program Assessments, Cost Estimating, and Schedules Office (ACES)
- Background on the office formation
- Overview of office processes, products, and tools
- Description and examples of office products and tools

## Office Background

- Assessment Team was put in place in Mid-July of 2001
- Primary goal was, and still is, to provide the ISS Management with the information needed to make good and timely decisions
- Secondary goal was, and still is, to always provide ISS Management with options, proposals, or recommendations with any assessment
- Recommendation was to make this office independent so that its assessments could span the entire program without bias
- An initial goal was to  “keep our eye on the ball” as we worked our way through the challenges facing the Program based on the $4 billion budget problem – as documented by the Young Committee
- Since that time, other initiatives have been undertaken --  including several that tie back  to the Young Committee recommendations

## Office Background

- Assessment Team was put in place in Mid-July of 2001 - continued
- The office was started with a manager and 6 support contractors who had been working ISS assessments under the Resource Management Office
- The office was established within the Program Control and Planning Office (previously Business Management Office)
- Even though this may not appear “independent”, the office has been allowed by the Program Manager and the PP&C Manager to operate in an “independent manner” since it’s inception
- The office has been augmented with both civil service and support contractor personnel to facilitate these activities
- Recently the schedules group, which had been reporting to the Program Integration Office, has been transferred into the ACES

## Cross Trained

**Program Planning and Control Office**

**Assessments, Cost Estimating,  and Schedules Office**

- Assessments/Cost Estimating

**Cost Estimates of Changes**
**Cost Estimates of Threats**
**Quantitative Risk Analysis of Threats/Budget**
**Documentation of Office Processes and Procedures**
**Utilization of Cost Estimating Models**
**Government Estimates for New Contracts**
**Cost Benefit Analysis**

**Assessments**

**Schedules**

**Schedules**

**Early Warning System – Monthly Ass’mts & Reports**
**Early Warning System – Quarterly Previews & Reports**
**Program Metrics**
**Program Performance Measurement System**
**Program Assessments – Near term/Run-out**
**Assessments of Program planning, contracting, and processes**

- Assessments/Schedules

**Cost Estimating**

## Initial Assessment Team Initiatives 

**Initial Assessment Team Initiatives **

- Early Warning System Monthly Reporting

- Cost Estimating of Changes, Risks, Threats, and Budgets

- Performance Measurement System Development and Implementation

- Quantitative Risk Analysis of Program Threats

- Assessments of Program Requirements and Status

- Quarterly Review Program Manager Previews and Post Meeting Reporting

- Unsolicited Assessments of Program planning, contracting, and processes

- Enhancement and New Development of Modeling Capabilities

- Recommendations for ISSP Interim Management Information System (IMIS)

**From August 2001**

## Initial Status

- Initial Assessment:  “Toolbox”
- Are in the process of completing an initial EWS Assessment using these tools and methodologies:
- Functional Area Assessment Tool (FAAT) Analysis
- Includes Functional One Pager Analyses
- Creepback Analysis (using 1-Page Trace)
- This Year/Next Year Analysis
- Analysis of Changes / Liens / Threats / Risks (PRAB)
- Integrated Program Schedule (IPS) Analysis
- Provider / Manager Inputs

**From August 2001**

## Office Processes and Products

- Since the Mid-July 2001 timeframe, the office has continued to evolve it’s processes and products as more assessment and cost estimating tools have been developed.  Primary processes and products include:
- Early Warning System Monthly Reporting Process
- EWS Report
- Pulses Program from many different views
- Merges cost impacts with technical and schedule issues and identifying trends
- Makes program control recommendations to the Program Manager
- Performance Measurement Process
- Earned Value Report
- Integrates PMS across both “Prime” and “Non-Prime” Program areas
- Expanding current  system to PRISM, a COTS earned value management system
- Performance Indicator / Performance Indicator Metrics Process
- Integrated Monthly Program Review – Meeting and Report
- Measure via a “stoplight approach” with a focus on Key Program Performance Indicators -- backed by Manager Level Pis to keep an eye on details

## Office Processes and Products

- MIS/IMPR
- Disseminates all “tools” and processes to all levels of management in the ISSP
- Source for HQ and Agency insight into Program Status and Health
- Integrated Monthly Program Review (IMPR) allows once a month update of metrics, schedules, and other MIS information
- Quantitative Risk Analyses (QRA) Process
- Provides probabilistic estimates of the magnitude of consequences for identified budget threats
- Independent Assessment Process
- Independent Assessment Presentations and Reports
- Examples:  Quarterly Preview and Post-Quarterly Reviews
- Raises potential issues and performance implications to Program Manager
- Cost Estimates and Change Evaluations Process
- Produce independent government estimates on changes and new work
- Schedules Assessments Process (recent office addition)
- Individual Schedule Assessments

## Office Assessments “Tool Box”

- The office assessment “Tool Box” has changed dramatically since July 2001 and has been key to the development of the office processes and products
- Assessment “tools”
- Program wide Earned Value
- Quantitative Risk Analysis (QRA)
- Cost Estimating capabilities
- OTS Models – SEER, PRICE, COCOMO
- IN-House Models – ISSAC
- Performance Indicator Metrics
- Independent Assessment capabilities
- Assessment Scorecard
- Schedule Analysis

## During the Last 12 months the following has been accomplished
Development and presentation of 12 EWS Reports
Development and presentation of 4 Quarterly Previews
Development and presentation of 4 Quarterly Reviews
Development and presentation of 12 Earned Value Reports
FY04 Assessment
FY03 EOY Assessment
POP Preview
POP Assessment
Approximately 15-20 Cost Estimates 
Major Program Cost Estimate on “Greater than 3 Crew” capability costs
Columbia Assessment(s) – Bottoms up and Model generated
ISS Threats Realization Study
POP Process Re-engineering Initiative
BOE Reviews

- Development and presentation of 12 EWS Reports
- Development and presentation of 4 Quarterly Previews
- Development and presentation of 4 Quarterly Reviews
- Development and presentation of 12 Earned Value Reports
- FY04 Assessment
- FY03 EOY Assessment
- POP Preview
- POP Assessment
- Approximately 15-20 Cost Estimates
- Major Program Cost Estimate on “Greater than 3 Crew” capability costs
- Columbia Assessment(s) – Bottoms up and Model generated
- ISS Threats Realization Study
- POP Process Re-engineering Initiative
- BOE Reviews

**Products completed during past 12 Months**

## Early Warning System

## Provides a summary assessment which results from reviewing ISSP data utilizing the following tools:

Cost Plan vs. Actual Analysis
Assessment Scorecard
Performance Indicator Metrics
Performance Measurement Analysis
Threats and QRA Assessment
Program Schedules
Other resources including monthly reviews, contractor reports, quarterly reviews, and participation in various ISSP forums, and ACES assessments

- Cost Plan vs. Actual Analysis
- Assessment Scorecard
- Performance Indicator Metrics
- Performance Measurement Analysis
- Threats and QRA Assessment
- Program Schedules
- Other resources including monthly reviews, contractor reports, quarterly reviews, and participation in various ISSP forums, and ACES assessments

**Section 1:  Executive Summary**

## Executive Summary

**Overall Summary**
- Cost reporting thru December indicates the ISS Program is underspending the baseline operating plan by 33m
- The underspend is higher when EVA actuals are normalized for ISS costs only (~43m underspend)
- The ACES Assessment completed in October FY04 indicates that the program is likely to underspend the initial operating plan by at least 105m
- The monthly message has not changed since January although the 200m Congressional Mark-Up was realized since the last report -- which has greatly impacted the ISS Program’s reserve posture in FY04 and FY05..
- The Program is left in the position of having known threats in excess of available reserves
- A projection of FY04 end of year posture including projected underspend and threats realization indicates the Program could leave the year with only ~150m (75 of which is fenced for forward funding requirements)
- Further assessments are underway and FY04 threats will be discussed in detail during the February Quarterly Review
- Additionally, significant threats continue to jeopardize the program’s ability to achieve US Core Complete and move toward a crew greater than three
- Contract Reconciliation impacts (still being assessed)
- Additional headquarters/external driven impacts
- Continued Node 3/Regen ECLSS development issues
- Large number of known threats in Program PRAB database

## Executive Summary

**Cost Plan vs. Actual**
- The FY04 ISS Program cost plan is 1,365m
- The ISS Program cum actual thru December is 292m
- The program is under spending the cum thru December Operating Plan by 33m (~43m for normalized EVA actuals)
**Assessment Scorecards**
- The FY04 ISS Assessment is 1,260m, this is 105m below the plan
- Boeing - 526m, Remaining Program - 734m
- The ISS Program is under spending the ACES cum thru December Assessment by 16m
- The underspend relative to the assessment is small (5%)
- Thru December there are no areas of the program that are of a concern not already factored in the Assessment.
- Overall there are 2 _*Red*_ (Boeing CM & Regenerative ECLSS) and 2 _*Yellow*_ (Boeing LP/SM & Boeing Carriers) areas vs. the Operating Plan on the Assessment Scorecards
**Performance Indicators**
- Overall there are 6 _*Reds*_ and 12 _*Yellows*_ out of the 42 Performance Indicators
- The Red indicators are for Research KPPI (upmass/planned investigation issues), SSP Overall Program Status for ISS KPPI (Shuttle impact to Station), Regen ECLSS KPPI (cost/schedule issues), Program Budget KPPI (reserves posture), Research Resources MLPI (crew time/upmass limitations), and JAXA schedule status MLPI (CAM schedule risk)
- Increased number of reds and yellows since January of last year (from 12 to 18) does not appear to indicate any specific trend since all are due to varied reasons

## Executive Summary

- SPI = **0.984**, CPI = **1.026**; status:  **GREEN**; SPI trend       (Nov.:  0.980), CPI trend        (Nov.:  1.004)
- CAPs’ FY04 plans to be adjusted as rephasing to new RTF schedule is completed
- IFMP FY04 startup data availability issues corrected this month; affects primarily CAP 13/CPO, used as repository for “balancer” cost data
- Due to delayed PMSR data, CAPs 1-3/Prime are estimated from CSTR data
- Performance index trend assessment:
- SPI increase driven by CAP 3/Prime Spares & CAP 9/Vehicle (recovery from last month drop); in general, ISS continues to hold non-flight-dependent schedules (e.g., H/W deliveries)
- CPI increase driven by CAPs 1-3/Prime & CAP 13/CPO (corrected IFMP issues)
- EOY projection continues to indicate will underspend FY04
**Threats / QRA Assessment**
- The current (2/6/04) Threats List contains $1079m (up $22m) for all Threats, FY04-08
- QRA results ~$800m (up $60m) expected impact at 80% confidence
- Much activity this month:
- Pre-Scrub rephasing of FY04 threats into FY05
- Many offsetting Threat movements (add, delete, etc) with little effect to Grand Total, but by-year impact is noticeable (FY04 decrease, FY05-08 increase)
- Level 1 Threats increased $102m, driving QRA
- Threats Scrub scheduled Feb 6, QRA result will appear in IMPR Budget charts, Quarterly Preview

**Executive Summary**

## Executive Summary

- Overall there are 2 _*Reds*_ and 3 _*Yellows*_ out of 18 on the ISS Program Schedule Health Chart
- The 2 _*Red*_s are the CAM schedule and Regen ECLSS
- The 3 _*Yellows*_ are Node 2, ATV and Node 3 schedules
- No new issues since the last report
- Regen ECLSS Racks:  ORU milestones have continuously slipped since re-plan in 6/2003 due to manufacturing and test problems
- Continued serial impacts to system tests and rack integration and test
- MSFC Project Office schedule reassessment completed.  Review scheduled for 2/13/04.
- ACES Office is currently working with the MSFC Project Office on an assessment of the Regen ECLSS program
**Quarterly Preview**
- A quarterly preview will be provided to ISS Program Management prior to the Quarterly Reviews on February 17th & 18th
- Quarterly Review Goals
- Re-phase additional content to FY05 and out to help build FY04 reserves and smooth transition for FY05
- Status ISS Program Full Cost Accounting performance thru the first quarter & validate current Civil Servant staffing plan
- Review Program Threats List for potential additions/deletions during the POP process
- Create a good baseline for the FY04 POP

**Executive Summary**

## Non Boeing Assessment Score Card

## ISS Program Schedule Health 

- Chart Owner: K. Martindale   x48196

**Purpose:****	**To provide a summary assessment of changes in ISS Program Schedules identifying significant impacts and threats to major milestones
**Sources:  **Detail schedules updated twice monthly with inputs from ISSP Participants & summarized in the IPSP review process, correlated with reporting of the KSC Integrated Assessment and Launch Package Manager’s DSSR reports

**G = Green:   ****Low Risk**
**Y = Yellow:  ****Medium Risk**
**R = Red:**       **High Risk**
**C = Blue:      ****Complete**

## EVM/PMS Executive Summary

## ISS Mid-program Implementation of Earned-Value

**Issue:  **Lack of across-the-board Earned-Value Measurement (EVM) noted by IMCE team as ISS weakness
- Had formal EVM reporting from Prime Contractor & several others, comprising about 40% of the ISS budget
- Remainder of Program had never reported EVM; needed to be captured under new Perf. Measurement Sys. (PMS)
**Challenge:  **Implement “sensible” EVM system, to answer IMCE criticism & be useful to mgmt. team
- Varying start-points for existing, EVM reporting CAPs; starts-from-scratch for new CAPs... how to synchronize?
- ISS largely in operations phase of life cycle... how to devise meaningful EVM metrics?
- ISS operates on moving 5-year budget window... how to define “completion” for BAC/EAC/VAC data?
- How best to project performance, considering ISS advanced stage of development?
- How to do all the above in as unobtrusive a way as possible (practical as well as political issue)?

## ISS Mid-program Implementation of Earned-Value- continued

**Approach:  **Use available data; do minimum needed to capture remaining variable, high-risk content
- Use existing contractor-supplied EVM data as-is; use cumulative data at PMS start as set-point
- Develop new CAPs along functional lines – responsible organization CAMs are PMS CAMs as well
- Use existing schedules – back-filled to be resource-loaded as required – as basis for CAP PMBs
- Start indices at 1.0, variances at 0.0 in absence of prior cumulative data
- Measure discretely only work content that was development, variable, or high-risk, & was tied to significant $
**Timeframe:  **18 weeks, including system/CAP development, ISS PMS/EVM Tool (IPET) development & implementation, & completion of 3 months of dry runs

## Typical CAP – Program Integration, CAP 12

**Content:**
-  Multi-increment
-    planning, system &
-    segment integration,
-    Intl. Partner element
-    integration, & external
-    carrier integration
-  Very little DDT&E
**Characteristics:**
-  Measures 41% of
-    content discretely
-  Remainder is “CPO”
-    (Cost Perf. Only)
-  Representative of
-    “earned-value like”
-    implementation
**Note:****  **CAP summary
- of content, earned-
- value metrics/methods,
- etc. documented in IPET

## Bullseye CAP Scatter Plot – December 2003

**Bullseye CAP Scatter Plot – ****December ****2003**

**Message:**
-   ISS aggregate CPI (see “x”) > TCPI (see green dashed line) indicates reported FY03-08 EACs are achievable

*<footer>**OG/Michael Jansen, X36614; michael.c.jansen1@jsc.nasa.gov*

## Synopsis of ISS PMS/EVM Results – December 2003

**Message:**
-  1 CAP degraded; 2 improved; 10 remained the
-      same (+ Research... remained the same)
-  RED:  None

*<footer>**OG/Michael Jansen, X36614; michael.c.jansen1@jsc.nasa.gov*

## Performance Indicators

## A metrics hierarchy creates a common framework by which the Program can understand its performance by looking at key indicators

**Performance Indicators**

**Performance Indicators **
**Metrics**

**A metrics hierarchy creates a common framework by which the Program can understand its performance by looking at key indicators**

**Key Program Performance Indicators**

- -------------------------------- Control Account Manager Metrics – Lower levels  -------------------------------------

**Performance Indicators**

## The Program Metrics Dashboard provides drill-down capabilities for Sr. Management to assess specific Program performance details

**The Program Metrics Dashboard provides drill-down capabilities for Sr. Management to assess specific Program performance details**

**PROGRAM METRICS DASHBOARD**

**METRICS DESCRIPTOR**

**PERFORMANCE INDICATOR METRICS**

## Slide 29

## Slide 30

## Performance Indicator Summary

-  Overall there are 6 _*Reds*_ and 12 _*yellows*_ out of 42 Indicators

**Performance Indicator Summary**

## QRA

## Quantitative Risk Analysis (QRA) Overview

- Provides ISS Program Management with a probabilistic estimate of the magnitude of consequences for identified budget threats.  Designed in conjunction with Futron.
- Threats are categorized into “Levels” based on the likelihood of their occurrence:
- Level 1 will most likely occur (>50%)
- Level 2 may or may not occur (50/50)
- Level 3 are not likely to occur (<50%)
- Uncertainty Factors are applied to each threat using probability distributions based on OPR input and/or the type of threat (Design & Development, Process, or Management).
- A Monte Carlo Simulation is run to produce a probabilistic forecast of the ISS Budget Threats and their impact to Management Reserves.

**QRA Estimate = (Level Distribution) X (Uncertainty Distribution) X (Threat $)**

## QRA Level Distributions

- Levels:   Futron made the following assumption based on NASA threat level definitions:
- Level 1: 84% Likely (range 67% to 100%)
- Level 2: 50% Likely (range 34% to 66%)
- Level 3: 17% Likely (range 0% to 33%)
- Uncertainty Factors: The Uncertainty Factors are defined as:  Design & Development, Process, Management
- Historical spacecraft design, development and operations cost data were collected from 1964 to 1993 in 347 programs and used to determine the uncertainty factors:
- Shuttle: 19 Flights
- Solar System Exploration: 51
- Manned (I.e. Apollo, Skylab and Spacelab): 29
- Planetary Landers: 41
- Normal (a catch-all for all others and the experiment packages onboard): 207
- Current efforts are underway to “tune” these distributions to better reflect the actual impact to Reserves based on Program Threats.
- QRA-2 will allow ability to tune outputs yearly to tie to realized ISS Program actuals

## Threats and Related Analysis – Detail

**Threats and Related Analysis – Detail**

**Quantitative Risk Assessment (QRA) analysis**The Program can be 80% confident that the actual impact to Reserves will be less than or equal to $800m

## Cost Estimating and Models

## ISSP Cost Estimating Tool Box

- SEER SEM Software Estimation &
- SEER H Hardware
-  PRICE S Software Model & H Hardware Model
- NASA/Air Force Cost Model

- Parametric Models

- Flight, protoflight, qual, and engineering unit hardware development
- Ground and flight operations software
- Ground facility operations

- @Risk
- “Ad-In” to Microsoft Excel

- Risk Analysis

- Quantitative Risk Assessments of ISS Program threats

- International Space Station Analytical Cost (ISSAC) Model
- NASA/Air Force Cost Model

- Program-Level Cost Models

- Total program costs based on scenario variables
- Scenario combinations
- Contractor Estimates

- Revised Intermediate Constructive Cost Model (COCOMO) and Revised Intermediate COCOMO (REVIC)

- Software Models

- Ground Trainers
- Ground and on-orbit operations software
- Functional testing

## ISSAC Main Portal Overview

- The International Space Station Analytical Cost (ISSAC) Model was developed with the intent to provide the ISS Program with greater capability and flexibility in performing cost estimation  analyses and project cost assessments.

## Activity Overview

- Activities in the ISSAC Model are defined as a task or job that is performed by an Organization in support of the ISS Program.
- Activities are aligned with the Products/Services in the ISS CARD.
- Examples of some of the activities identified in the ISSAC Model are: Assembly Validation, Development Activity, Flight Element Processing, Planning, and Facilities Support.

## Cost Driver Overview

- Total Dollars: The total dollars per year for the selected activity
- Number of EP: The total number of EP per year for that activity
- Step Function: Allows Users to assign dollars or EPs to a pre-determined unit range.  This allows an incremental change in costs if the number of units changes from one range to another.
- Dollars per Unit: The Dollars Per Unit for an Activity including:  Documents, Anomaly, Procedures, Flights, EVAs, Elements, Increment, Crew Member, Lbs, Kg, Hours, Shuttle Flight, Soyuz Flight, Progress Flight, ATV/HTV Flight, Pressurized Vehicle, SpaceHab, MPLM, Racks, KSLOCS, and PRs. The Cost type (Fixed or Variable) is also identified
- EP per Unit: the EP per Unit for an Activity.  The lists of units are the same as found in the Dollars per Unit field above.  The cost type (Fixed or Variable) is also identified

- Cost drivers determine the cost of an Activity within an Organization.  There are currently five types of cost drivers identified in the ISSAC Model:

## Reports and Outputs Overview

- Administrators, Advanced Users, and Users have the capability to run the predefined _[Reports and Outputs](file:///Fox_Space_Station_Assessment_Implementation.ppt)__.  ISSAC allows the User to define reports to be generated by specifying the following:_

- Scenario combinations (Flight Schedule, Activity Status, and Inflation)
- Compare Scenario combinations
- Government Fiscal Year (GFY) range
- All Contractors or selected contractors (Administrator Only) and All Contracts or selected contracts
- Scope of Reports (Entire ISS Program, Organization, Branch or Activity)
- Report Type, Output in dollars and/or EP, chart type
- Rounding
- Other Options

## MIS / IMPRFlow chart and Demo

## The ISS Program utilizes the NMIS as the primary location for key Program Metrics and Schedule Status

**Performance**
** ****Indicator Metrics**

**Schedules**
**and Issues**

- The Monthly Program Review provides a focus for ISS Management to review both Cost and Technical Metrics and Flight and Increment Schedules status.

- The Program data is uploaded to the NMIS within minutes of the completion of the IMPR ensuring most current data availability.

- The ISS Program Manager uses the ISS MIS and NMIS as the primary location of Key Program data – allowing real time status capability to all within the Agency.

- Performance Indicator Metrics are updated and presented monthly

- Flight and increment schedules are updated with status and issues presented monthly

**The ISS Program utilizes the NMIS as the primary location for key Program Metrics and Schedule Status**

**NASA Management Information System**