# Presentation: 482079

## ARRT  Advanced Risk Reduction Tool

- Presentation to the 1st Annual NASAOffice of Safety and Mission Assurance (OSMA)Software Assurance Symposium (SAS)
- Dr. Martin S. Feather
- ARRT Center Initiative Lead*
- Jet Propulsion Laboratory
- California Institute of Technology
- Martin.S.Feather@Jpl.Nasa.Govhttp://eis.jpl.nasa.gov/~mfeather
- *Initiative began in 1999 with Dr. John Kelly as Lead

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT Heritage & Contributors

- ARRT is inspired by, and based onJPLer Steve Cornford’s Defect Detection and Prevention (DDP)and JPLer Tim Larson’s Risk Balancing Profiles (RBP).

- John Kelly
- Burt Sigal
- James Eddingfield
- Steve Cornford
- Phil Daggett
- Julia Dunphy
- Roger Klemm

- Jim Kiper (U. Miami, Ohio)
- William Evanco (Drexel)
- Steve Fickas (U. Oregon)
- Martha Wetherholt (NASA Glenn)
- Richard Hutchinson (Wofford, SC)
**primary collaborators**
- Tim Menzies (U. British Columbia)Tim Kurtz (NASA Glenn)Hoh In (Texas A&M)

**contributors (JPL)**

**contributors**

**funding, management****& guidance**
- NASA Code Q, NASA Goddard IV&V Facility
- Siamak Yassini, Ken McGill, Marcus Fisher

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## The Universe of ARRT Customers

**optimists**

**pragmatists**

**pessimists**

**“****Hello, I’m from Software Quality Assurance / IV&V****and I’m here to help you”**

**“****Got Risk?”**
- “Too much...”“Too little...”“Don’t know...”

**“****Plan the best use****of Software Quality Assurance / IV&V”**

**“****How?”**

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## TheOptimists

**“****Hello, I’m from Software Quality Assurance / IV&V and I’m here to help you”**

- Many attendees of this symposium are likely to already believe in the net value of assurance activities, but optimism alone is not sufficiently contagious!

- What is needed is the means to *quantitatively* assess the cost/benefit of assurance activities applied to specific projects. This will:
-  be more convincing
-  determine best use of limited resources
-  identify alternatives (e.g., requirements to discard)

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## The Optimists

- ARRT performs quantitative cost/benefit calculation for *suite* of assurance activities applied to a specific project

- Cost/benefit data & reasoning has been applied to:
**Individual activities**, e.g., Regression testing [Graves et al, 1998].
**Pairwise comparisons**, e.g., “Peer reviews are more effective than function testing for faults of omission and incorrect specification” [Basili & Boehm, 2000].

**Lifecycle process improvement**, e.g., Quality, productivity and estimation gains from CMM-like process improvement [McGarry et al, 1998].

- Gap!

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT’s Quantitative Cost/Benefit Model

**Risk mitigations** subdivided into
**Preventions** – prevent problems from appearing in the first place	e.g., training programmers **** fewer coding errors	**cost** = performing prevention	**benefit** = reduction of risk likelihood
**Detections** – detect problems so that they can be corrected	e.g., unit testing **** detects internal coding errors	**cost** = performing detection +		performing the repair (cost depends on when!)	**benefit** = reduction of risk likelihood
**Alleviations** – applied to decrease the severity of problems	e.g., robust coding **** tolerant of out-of-bound input values	**cost** = performing alleviation	**benefit** = reduction of risk severity

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Cost/Benefit – Simple Scenario

**System tests,**** ****observed by spacecraft engineers****($$)**

**Mission loss** **due to misinterpretation of requirements**

**Reimplement misinterpreted requirements**** ($$$)**

**Misinterpret ambiguous requirements**

**Poorly written requirements**

**Programming errors**

**Correct programming errors**** ($$)**

**Mission loss due to programming errors**

- Implementationphase

- assurancechoices

- mistakeshappen

**Use ARM to do Requirements Analysis**** ****($)**

**Correct ambiguous requirements**** ($)**

- Testphase

- Operationsphase

- Requirementsphase

- Low costs to analyze with ARM & correct flaws now

**RISKS**

*High* cost to reimplement requirements this late in development

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Cost/Benefit – Simple Scenario (cont.)

**Use ARM to do Requirements Inspection**** ****($)**

**System tests,**** ****observed by spacecraft engineers ****($$)**

**0 + 0 + $$ + $$$ + $$ = $$$$$$$**

**Correct ambiguous requirements**** ($)**

**Reimplement misinterpreted requirements**** (-/$$$)**

**Correct programming errors**** ($$)**

**$ + $ + 0   + 0     + 0   = $$**

**$ + $ + $$ + 0     + $$ = $$$$$$**

**0 + 0 + 0   + 0  +    0   = 0**

**risk of mission loss**

- Lowest risk, but NOT highest cost – savings from correcting problems early

- risk decreases

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Return On Investment of Assurance & IV&V

**Return On Investment (ROI) calculation**	ROI = benefit of risk reduction / cost of assurance
*Conservative* basis for ROI: benefit = 	Mission cost * (Risk reduction due to Assurance & IV&V)
-  E.g., Mars Polar Lander + Mars Climate Orbiter missions cost = $183,000,000
*Aggressive* basis for ROI: benefit = 	(Value of attaining mission requirements) *	(Risk reduction due to Assurance & IV&V)
-  What is the value of discovering water on Mars?** **
** **What is the value of returning a Mars sample to Earth?

**$ + $ + $$ + 0     + $$ = $$$$$$**

**risk of **
**Mission****loss**

**$ + $ + 0   + 0     + 0   = $$**

- Is it worth paying **$$$$** to save this much risk?

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT’s Quantitative Cost/Benefit Model

- Cost/benefit computations in ARRT
-  Automatic
-  Handle *suite* of assurance activities
-  Permit data to be changed if we know better than standard estimates
-  Distinguish development phases (requirements, design, ...)
- Distinguish preventions, detections and alleviations
-  Combine with underlying risk computation model (see next section)

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## The Pessimists

- GOTRISK?

*TOO MUCH* – use ARRT to planhow to **reduce risk in a cost-effective manner**.
*TOO LITTLE* – use ARRT to plan how to **accept more risk in exchange for reduced cost and schedule, more functionality**, etc.
*JUST RIGHT* – use ARRT to **maintain a desired risk profile** through the lifetime of the project.
*DON’T KNOW* – use ARRT to **assess risk status.**

**“****Risk as a Resource” – Dr. Michael Greenfield****[Greenfield, 1998]**

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT’s treatment of Risk– DDP & RBP concepts, specifically populated with software data

- DDP is a *process* [Cornford et al, 2001]supported by a custom *tool*  [Feather et al, 2000a] for *quantitative* risk management.
- RBP is a *qualitative* risk management tool populated with risk and risk mitigation data.
- DDP & RBP merged [Feather et al, 2000b] into DDP
- ARRT uses this merged combination of DDP & RBP

- ARRT is inspired by, and based onJPLer Steve Cornford’s Defect Detection and Prevention (DDP)and JPLer Tim Larson’s Risk Balancing Profiles (RBP).
- In particular, ARRT inherits DDP’s Risk Model.

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT inherits DDP’s Risk Model

**Impact**** of a given FM on a particular requirement**

***Impacts***

- Failure Modes/Risk Elements

- Mission
- Requirements

- 

- 

**Effectiveness**** of a given PACT to detect, prevent or alleviate a particular FM**

***Effects***

*Weighted* Failure Modes/Risk Elements

- PACTs*

-  

-  

- 		DDP utilizes three trees of key concepts:
- Requirements (what you want)
- Failure Modes / Risk Elements (what can get in the way of requirements)
- PACTs (what can mitigate risk)
- 		and two matrices that connect those concepts:
- Impacts (how much Requirement loss is caused by a FM)
- Effectivenesses (how much a PACT mitigates a FM)

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT/DDP Computations & Visualizations

-  Information is derived from user-provided data via built-in computations, e.g.,
- FM’s cumulative impact = FM.Likelihood * ( (R  Requirements) R.Weight * Impact(R, FM))
- Information presented via cogent visualizations
- Bar charts
- Risk Region chart
- Stem-and-leaf plots
- Detailed view of properties of individual element

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT/DDP Trees

*Taxonomies* of Software Requirements / Risks / Risk Mitigations

- ContractedExpanded
- SelectedDeselected
- Number:Title

- Autonumbering: *linear* 1,2,... or *tree* 1, 1.1, 1.2, 1.2.1, ...

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT/DDP Matrices

**Effects** (Mitigation x Risk)

**Impacts** (Requirement x Risk):proportion of Requirement loss if Risk occurs

- proportion of Risk reduced by Mitigation

- numbers supplied by experts and/or based on accumulated metrics

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT/DDP Visualizations - Bar Charts

- Unsorted – order matches leaf elements in Risk tree

- Sorted – in decreasing order of remaining risk

**Requirements bar chart** – how much each is impacted
**Mitigations bar chart** – how much impact each is saving

- Green: of this Risk’s total Impact on Requirements, that *saved* by Mitigations

- Red: of this Risks’s total Impact on Requirements, that *remaining* despite Mitigations

**Risks bar chart**

- Item number in tree

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT/DDP Visualizations – Risk Region – “InChart”

- User defines risk levels demarking red/yellow/green/(tiny) risk regions

- Log/Log scale: diagonal boundaries = risk contour lines

- Conventional measure of riskas impact (severity) x likelihood.

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT/DDP Visualizations – stem-and-leaf(*) charts

- (*) Tufte attributes these to John W. Tukey, “Some Graphical and Semigraphic Displays”Their usage was introduced into RBP by D. Howard, extended further by us in DDP.

- Compact visualization of DDP’s sparse matrices

**Risks** – red width  log outstanding  impact

**E.g., Risks & their Mitigations**

**Mitigations**** – **turquoise width  effect

- selected

- unselected

- item number in Risk tree

- item number in Mitigation tree

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## The Pragmatists

**Objective: Plan the best use of****Software Quality Assurance & IV&V**
- “Has it been used?”
- “Where does the data come from?”
- “How does it combine with software estimation & planning?”
- “What about...?”

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Focused study data: Software Assessment Exercise 

- Steve Cornford, JPL + others
- Focus: code generation by [product name deliberately hidden]
- Flight code of modest experiment
- Flight code for future missions
- 15+ experts in 4 x 4-hour sessions, Sept 2000
- [product] experts
- Mission experts
- Software experts (SQA, coders, ...)
- Large information set
- 47 Requirements (unprioritized)
- 76 Risks (near-term mission-specific & futuristic)
- 303 Mitigations (pre-populated with large set)
- 107 Impacts
- 223 Effects

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Software Assessment Exercise – extract

*Portions of the Requirements tree and bar chart*

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Software Engineering Community Data 

- Risks:  Software Risk Taxonomy (SEI)
- Mitigations: two datasets:
- JPL’s Risk Balance Profile of SQA actions
- Assurance activities from Ask Pete (NASA Glenn tool)
- Effects: cross-linkings of the above (Jim Kiper)
- Expert’s best estimates of yes/no (Prof. J. Kiper)
- Experts’ 1000+ best estimates of quantified effectiveness (Prof. J. Kiper & J. Eddingfield)
- Note: Requirements are PROJECT SPECIFIC

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Software Estimation & Planning data: ARRT – Ask Pete collaboration

- Tim Kurtz,  Tim.Kurtz@grc.nasa.gov
- SAIC/NASA Glenn Research Center
- http://tkurtz.grc.nasa.gov/pete
- Principal Investigator  Martha Wetherholt

- see companion presentation in this symposium

- Ask Pete  runs to gather project characteristics, make first cut at suggested selection of risk mitigations.*Mitigation selection passed to ****ARRT***
**ARRT**** **runs to allow user to assess risk, provide costs, **customize to project **(add/remove risks, refine effect values, etc.), tune selection accordingly.*Revised mitigation selection returned to **Ask Pete*
- Ask Pete runs to generate final reports

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT - Tim Menzies collaboration

- Prof. Tim Menzies, U. British Columbia
- Optimization – automated search for (near) optimal mitigations suites
- Least risk for given cost
- Least cost for given risk
- Sensitivity analysis
- On which data values do the results hinge?
- Scrutinize these values further
- Identify points of leverage (e.g., problematic requirements; make-or-break decisions)
- Retain human involvement
- Extend reasoning to more complex data
- Interactions: mitigations that induce risk(e.g., code changes to correct one bug may introduce other bugs)
- Ranges / distributions of values (e.g., [0.1 – 0.3])

- tim@menzies.com

- Benefits to ARRT of collaboration

- see companion presentation in this symposium

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## ARRT – Hoh In et al collaboration: IEESIM

_***I***__*ntegrated*_ views (data schema) from local tool views
_***E***__*xchangeable*_ format based on XML
_***E***__*xtendable*_ interfaces for additional tools
_***S***__*hared*_ _***I***__*nformation*_ _***M***__*ediator*_

**INTERMEDIARY**

- IEESIM

**DDP**

**Other Tools (e.g., VCR)**

**Shared Database**

**IEESIM Server**

**IEESIM Client**

**IEESIM Client**

- Web Browser

**IEESIM Client**

**ASK PETE**

- http://www.cs.tamu.edu/faculty/hohin/

- Repository ofproject data
- Insert & classify,Search,Retrieve,Delete
- Accessibility via the web

**Prof. Hoh In, Texas A&M University**

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Hoh In et al –Visualized Conflict Resolution (VCR)

- Sophisticated Visualization
- Intuitive graphical presentationsof consensus, conflict trends.
- Scalable and multi-dimension visualization.
- Powerful Analysis Support
- Identify non-trivial interrelationships (Clustering).
- Discover stakeholder decision rationales (Profiles).
- Benefit-cost tradeoff analysis
- XML adopted as standard medium of data exchange
- Status:examples of both kindsof data transferred & visualized

- Hoh In et al –Visualized Conflict Resolution (VCR)

- see Friday’s demo at this symposium

- Hoh’s visualization work motivated inclusion of the green/yellow/red Risk chart capability into ARRT – slide 18

- [Hoh & Roy, 2001]

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Hoh In et al – Visualized Conflict Resolution (VCR)

- Shows
- clusters per criterion,
- mean, max,min
- values

- Shows
-  issues, criteria of
- evaluation

- Shows
- clusters spanning all
- criteria of an issue

- Shows
- individual stakeholder
- perceptions/votes,
- group perceptions

- Shows
-  the degree of consensus
- in form of ellipse

- Hoh In et al – Visualized Conflict Resolution (VCR)

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## Concluding Remarkseven this talk maps to ARRT/DDP’s concepts!

**optimists**

**pragmatists**

**pessimists**

**Requirements**:
- what ARRT will help you achieve

**Mitigations**:
- what it takes to apply ARRT

**Risks**:
- what ARRT will help you avoid

- see Friday’s demo at this symposium

**http://eis.jpl.nasa.gov/~mfeather**

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## References 

- [Basili & Boehm, 2000] V. Basili & B.Boehm "CeBaSE: The Center for Empirically based Software Engineering" *NASA Goddard 25**th** Annual Software Engineering Workshop*, 2000.
- [Cornford et al, 2001] S.L. Cornford, M.S. Feather & K.A. Hicks. “DDP – A tool for life-cycle risk management”, *IEEE Aerospace Conference*, Big Sky, Montana, Mar 2001, pp. 441-451.
- [Feather et al, 2000a] M.S. Feather, S.L. Cornford & M. Gibbel. “Scalable Mechanisms for Requirements Interaction Management”, *4th IEEE International Conference on Requirements Engineering*, Schaumburg, Illinois: 119-129, June 2000.
- [Feather et al, 2000b] M.S. Feather, S.L. Cornford & T.W. Larson. “Combining the Best Attributes of Qualitative and Quantitative Risk Management Tool Support”, *15th IEEE International Conference on Automated Software Engineering*, Grenoble, France: 309-312, September 2000.

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001

## References

- [Graves et al, 1998] T. Graves, M. Harrold, J. Kim, A. Porter and G. Rothermel. “An Empirical Study of Regression Test Selection Techniques”. *20**th** Int. Conference on Software Engineering*, 1998, pp. 267-273.
- [Greenfield, 1998] M.A. Greenfield “Risk Management ‘Risk As A Resource’ ” *http://www.hq.nasa.gov/office/codeq/risk/*
- [Hoh & Roy, 2001] H. In & S. Roy “Visualization Issues for Software Requirements Negotiation” *25**th** Annual International Computer Software and Applications Conference*, Chicago, IL, Oct. 2001.
- [McGarry et al, 1998] F. McGarry, S. Burke & B. Decker. Measuring the impacts individual process maturity attributes have on software products., *5**th** International Software Metrics Symposium*, 1998, pp. 52-60

- Advanced Risk Reduction Tool - M.S. Feather

- NASA OSMA SAS 2001