# Presentation: 106763

## TMCO RISK ANALYSIS

**Phase I Proposal Risk Assessment:**
-   Selection based primarily on Science
-   TMCO risk assessment high level on a *preliminary concept* with some benefit of
-     doubt given to proposers.
-   Cost analysis done without proposer feedback and integrated into overall risk.
-   Goal:  No High Risk proposals but OK Med-Low Risk if science is compelling.
-   Concern:  Phase II might find proposals too risky.
**Phase II Concept Study Risk Assessment:**
-   Science selection has been completed.
-   Risk of Implementation assessment now done at a lower level and on an *advanced*
*   **concept* with some proposer feedback (all major concerns addressed by proposers
-    at oral briefings during a site visit)
-   Cost risk analysis integrated into total risk assessment but is also highlighted
-    separately for consideration.
-   Goal:  Give selection officials best possible assessment of overall risk and in
-    particular provide some indication of possible cost concerns.
-   Concern:  Even *advanced concepts* can fail as they proceed through design
**Assumption:  **TMCO process is structured, objective, and aimed at highlighting
-       foreseeable problems, however, _unforeseeable_ problems can always emerge
-       during design and development.

## Earth and Space Science Mission Risk

**Risks**
**For **
**Earth and Space Science Missions**

**Inherent **
**Risks**
**of **
**Missions**

**Implementation**
** ****Risks of Missions**
**(Evaluated by TMCO)**

**Programmatic Risks **
**of **
**Missions**

- Risks that are unavoidable
- to do the mission:
-   Launch environments
-   Space environments
-   Mission durations
-   Technologies or technology
-     extensions
-   Unknowns
-   etc.

- Risks that are uncertainties
- due to matters beyond project
- control:
-   Environmental Assessment
-     approvals
-   Budgetary uncertainties
-   Political impacts
-   Late/non-delivery of NASA
-     provided project elements
-   etc.

- Risks that are associated with implementing the mission:
- Adequacy of planning
- Adequacy of management
- Adequacy of development approach
- Adequacy of schedule
- Adequacy of funding
- Adequacy of Risk Management (planning for the known and unknown)

## TMC Evaluation Purpose

- The TMC evaluation is to determine, for each proposal, the level of risk of accomplishing the scientific objectives of the mission, as proposed, on time and within cost
- Three levels of risk are typically defined:  **Low Risk, Medium Risk and High Risk**
**Low Risk:**  No problems that can not normally be overcome within the time and cost proposed.  Problems not of sufficient magnitude to doubt the proposers likelihood to accomplish the Mission. “Envelope more than adequate”
**Medium Risk:**  Problems that make us somewhat uncomfortable, but are not sufficiently bad that the proposers can not overcome them with good management and application of engineering resources.  Technology may not be sufficiently ready, but we think it can be made ready with the time and money available  Complexity is inherently risky but not too risky. Resources are tight but possible.  “Envelope Tight”
**High Risk:**  Major problems which make us expect failure.  Insufficient resources to overcome the problems.  “Does not fit in the Envelope”

## Envelope Concept

**Envelope Concept**

**Envelope:  **All resources (TMC) available to handle known and unknown development problems.
- Includes schedule and funding reserves; reserves and margins on physical resources such as mass, power,
- & data; descope options; and fallback plans.
**Low Risk:  **Required resources fit well within the available resources.
- 					Available (Technical, Management, Cost Resources)
**Medium Risk:  **Required resources fit, but just barely inside the available resources.  Tight but likely doable.
- 					Available (Technical, Management, Cost Resources)
**High Risk**:  Required resources DO NOT fit inside available resources.  **Expect the project to fail.**

- Required

- Required

- Available (Technical, Management, Cost Resources)

- Required

## Evaluation Approach:  Validate the Proposals

**•****	****TMCO could never achieve the level of knowledge that the PI’s and their teams have**
**•****	****We look for ways of checking that the overall plans for implementation are good ones**
**	****-****	****Does everything add up and make sense?**
**	****-****	****Is everything included?**
**	****-****	****Are there ample contingency reserves and margins?**
**	****-****	****Does the plan jibe with past experience?**
**	****-****	****Are known problems dealt with effectively?**
**•****	****How did **_**you**_** check your plans?  Tell us, so we can check it that way, too.**
**	****-****	****E.g. How did you check your grass-roots, bottoms-up cost estimate?**

## Slide 6

## Options for TMC Reviews

**Things we’ve tried**
**•****	****Full science and TMC evaluation of proposal but no further competition**
**	****-****	****Seems to lead to a lot of overruns**
**•****	**** ****Full science and TMC evaluation of proposal followed by a TMC evaluation of mission concept study**
**	****-****	****The current system for MIDEX ‘01**
**•****	**** ****Full science evaluation but only TMC-lite evaluation of proposal followed by a TMC evaluation of mission concept study**
**	****-****	****Experiment underway with SMEX ‘99**
**•****	**** ****Full science evaluation of proposal followed by a TMC evaluation of proposals getting a high science rating**
**	****-****	****UNEX**
**•****	**** ****Full science evaluation but only TMC-lite evaluation of Step 1 proposal followed by a TMC evaluation of Step 2 proposal and followed later still by a competitive downselection at PDR/Confirmation**
**	****-****	****Experiment underway with ESSP-3**

## Report from the SMEX ‘00 TMC-Lite Experiment

_**What NASA was hoping to accomplish**_
**•****	****Allow proposers to concentrate on science**
**•****	****Reduce need to find industrial partners**
**•****	****Reduce effort to prepare initial proposals**
_**Comments from proposers**_
**•****	****2 liked it, 4 thought it was OK or so-so, 2 did not like it. **
**	****-****	****Selected and non-selected were in each group**
**•****	****“****I think the system is great!”**
**•****	****“****Process is seriously flawed because it leads to proposals will have to be descoped...”**
**•****	****“****Avoiding the full TMC evaluation at the early stage saves time and money on all sides”**
**•****	****“****Full study of implementation was necessary anyway in order to avoid ‘mission impossible’.”**
**•****	****“****Universities are not prepared to submit proposals with ‘cost guesses’.”**
**•****	****“****Review process still seems to favor proposals which better prove realism.  We included information not required and received favorable comments.”**
**•****	****“****Rejected, but only negative comments were for implementation issues.  Wish for more room to explain the implementation.”**