# Presentation: 524425

## Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making

**FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP)**
**for the**
**NASA Independent Verification and Validation Facility**** **
**COTR: Kenneth McGill**
**PI: Nancy Eickelmann**
**Contract #S-54493-G**
**September 4, 2002**

**Notes:**
Phd in Computer Science at University of California Irvine
MS CS
MBA Information and Decision Systems
Bachelor’s in Finance

I’ve been in WV for a year working on the IV&V ROI project, just prior to that I was with Advanced Research Program at MCC after spending 4 years at Hughes Research Labs in Malibu CA.

## Agenda

- Why we need ASK IVEY
- Consequences and Likelihood of Failure
- IV&V Yield
- Probability of IV&V Yield: Min, Max, Most Likely
- ROI and Magnitude of Return of IV&V
- What ASK IVEY can do

## Why we need ASK IVEY

- NASA program managers are asked to quantify the ROI and evaluate the cost/benefit of applying IV&V technologies.
- This is a prediction of future events based on decisions and actions taken in the present.
- A point estimate is likely to be inaccurate, whereas a probability of yield has a history of providing a scope of potential yield and an extent of likelihood of expected yield.

## Calculating ROI a Financial Analysis Prompt Map

## Consequences of Failure

- NPG 2820 IV&V Criteria

## Likelihood of Failure

## IV&V YIELD

- Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V.
- If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit.
- Should the resources saved be less than the resources consumed, we have a net cost.

## Cost of Poor Quality

- Defect Leakage
- If discovered internally
- defect management
- rework
- retesting
- If discovered externally
- technical support
- complaint investigation
- defect notification

## Stephen Knox“Modeling the Cost of Software Quality,”Digital Technical Journal,  (Fall 1993) 

## Raytheon Cost of Poor QualityHaley and Dion

## How Process Maturity Levels Affect IV&V

**LEVEL 1**
**INITIAL**
- UNPREDICTABLE &
- POORLY CONTROLLED

**LEVEL 2**
**REPEATABLE**
- CAN REPEAT
- PREVIOUSLY
- MASTERED TASKS

**LEVEL 3**
**DEFINED**
- PROCESS
- CHARACTERIZED,
- FAIRLY WELL
- UNDERSTOOD

**LEVEL 4**
**MANAGED**
- PROCESS MEASURED
- AND CONTROLLED

**LEVEL 5**
**OPTIMIZED**
- FOCUS ON PROCESS
- IMPROVEMENT

- KEY PROBLEMS:
- CONFIG. MGMT.
- PROJECT MGMT.
- SOFTWARE QA
- PROJECT PLANNING
- EASTIMATING
- COST
- SCHEDULE

- KEY PROBLEMS:
- PROPER TRAINING
- DEVELOPMENT OF PRACTICE & PROCEDURES
- STANDARDS
- ORGANIZATION

- KEY PROBLEMS:
- ACCURATE PROJECT MEASUREMENT
- OBJECTIVE PROCESS ANALYSIS
- QUANTITATIVE QUALITY PLANS
- PRIORITIES:
-  - DECIDING WHAT TO IMPROVE FIRST
-  - BALANCING THE PROCESS AND THE ORGANIZATION

- KEY PROBLEMS:
- CHANGING TECHNOLOGY
- PROBLEM ANALYSIS
- PROBLEM PREVENTION
- ORGANIZATIONAL OPTIMIZATION

- KEY PROBLEMS:
- STILL HUMAN INTENSIVE PROCESS
- DIFFICULT TO MAINTAIN OPTIMUM ORGANIZATION
- DIFFICULT TO MAINTAIN TOOLS & PRACTICES AT STATE OF THE ART

- IV&V
- SOMEWHAT
- UNPREDICTABLE

- UNABLE  TO ESTIMATE
- NON-TECH %

**IV&V**

**IV&V**

**IV&V**

**IV&V**

**NON-TECH UP TO 25%**

**NON-TECH UP TO 15%**

**NON-TECH UP TO 6-8%**

**NON-TECH UP TO 3-4%**

**MATURITY**

## Cost of Leakage GrowsOver Time 

- Relative cost of fixing a problem found in design/coding, testing, or after release are:
- 1:20:82 (Remus, 1983)
- 1:13:92 (Kan, 1989)
- 10:100:1000 (Coyle, 1999)

## Cost of Rework in Each Phase

- Rework*product design* =
- leakage*requirements* * cost-to-fix*nominal* * 10
- Rework*programming* =
- leakage*requirements* * cost-to-fix*nominal* * 100
- leakage*design* * cost-to-fix*nominal* * 10
- Rework*integration* =
- leakage*requirements* * cost-to-fix*nominal* * 1000
- leakage*design* * cost-to-fix*nominal* * 100
- leakage*programming* * cost-to-fix*nominal* * 10
- Rework*deployment* =
- leakage*requirements* * cost-to-fix*nominal* * 10000
- leakage*design* * cost-to-fix*nominal* * 1000
- leakage*programming* * cost-to-fix*nominal* * 100

## Rework at Deployment

- Tremendous cost
- rework plus
- product recall
- technical support
- field visits
- cost factor may be over 10,000

## Leakage: An Example

- Rework*product design* =
- 49r * cost-to-fix*nominal* * 10 = 490
- Rework*programming* =
- 39r * cost-to-fix*nominal* * 100 = 3,900
- 113d * cost-to-fix*nominal* * 10 = 1,130
- Rework*integration* =
- 26r * cost-to-fix*nominal* * 1000 = 26,000
- 49d * cost-to-fix*nominal* * 100 = 4,900
- 418p * cost-to-fix*nominal* * 10 = 4,180
- Rework*deployment* =
- 8r * cost-to-fix*nominal* * 10000 = 80,000
- 16d * cost-to-fix*nominal* * 1000 = 16,000
- 56p * cost-to-fix*nominal* * 100 = 5,600		**142,200**

## CMM Maturity and Leakage

- There is some evidence to suggest organizations with increased maturity have reduced rework costs
- Knox: Percent of Budget to Rework:
- Level 1: 55%
- Level 2: 45%
- Level 3: 35%
- Level 4: 20%
- Level 5: 6%

## IV&V and Defect Leakage

- Application of IV&V can reduce leakage to subsequent phases
- The goal of the financial model is to propose a range of potential savings
- Specific parameters will need to be established empirically

## Timing of benefits for IV&V

- Full In-Phase IV&V
- prevention of errors starting at requirements - can potentially bar any errors from leaking through
- Partial IV&V
- prevention of errors at point of insertion - no errors from this phase will leak
- Endgame IV&V
- discovery of errors at the end of development - can potentially bar any errors from leaking to deployment
- Audit Level IV&V

## Rework and Return from IV&V By Maturity Level

## Components to Return on Investment

- Cost of IV&V
- Expected Return
- cost savings - measured as hours of rework
- Likelihood of Returns
- how effective is the organization at minimizing rework?
- how effective will IV&V be?

## Independence...

- An organization *independent* from the developers study the artifacts of software production [IEEE Std. 1012-1998].
- This requires:
***Technical independence****. M*embers of the IV&V team may not be personnel involved in the development of the software.
- .***Managerial independence****. *The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software.*** ***
***Financial independence****.* Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software.
- IV&V is often perceived as testing the code after the development is completed .....NASA IV&V is _*full life cycle*_ activities

## IV&V is NOT SQA

- IV&V is a full life cycle set of acivities that are applied to defect prevention, defect detection, and certification. NASA IV&V conforms to IEEE Standard 1012-1998.
- IV&V and Software Quality Assurance (SQA) are not redundant activities. SQA as defined by DOD-Std 2168 defines 10 activities of SQA that are complemented by IV&V activities. There are 32 types of activities conducted by IV&V, of these 32, 22 are unique to IV&V and 10 are complemented by SQA.

## Ask Ivey Prototype

- What ASK IVEY can do...

## Ask Ivey Input Screen

## Ask Ivey Pull Down Menu

## Ask Ivey Numeric Entry 

## Ask Ivey On-Line Report 

## Ask Ivey Printed Report

## Ask Ivey On-Line Help

## Questions?

- Ask Ivey...