# Presentation: 929792

## Direct Return on Investment of Software Independent Verification and Validation: Methodology and Initial Case Studies

James B. Dabney and Gary Barber
Assurance Technology Symposium
5 June 2003

- James B. Dabney and Gary Barber
- Assurance Technology Symposium
- 5 June 2003

## Overview

- Motivation
- Methodology
- Results
- Summary and Conclusions

## Motivation

- Direct benefit of IV&V
- Direct cost reduction via phase containment
- Indirect benefits of IV&V
- Maintenance
- Safety
- Mission success
- Management insight
- Process IV&V

## Slide 4

## Return on Investment

- Ratio of cost savings to cost of IV&V
- Metric to guide IV&V decision makers
- Component of future tool to tailor IV&V process
- How much IV&V is appropriate
- Phasing of IV&V

## Objectives

- Develop analytical method to compute IV&V ROI
- Baseline IV&V process (criticality and risk-driven)
- Inputs are standard lifecycle project metrics
- Staffing for developer and IV&V
- Issue databases
- Activity reports
- Cost to fix data
- Develop predictive ROI tool
- Baseline IV&V process
- Inputs are early lifecycle metrics
- Provide a measurement methodology for future IV&V project optimization

## ROI Study Plan Overview

- Phase I – Case studies
- Four projects from different domains
- Direct ROI only
- Phase II – Prototype predictive model
- Define indirect benefits and method for quantification
- Develop and test model
- Phase III – Production predictive model
- End result: model to estimate ROI for candidate project using
- Measureable project characteristics
- Baseline IV&V process
- Future extension
- Lifecycle model allowing varying levels of IV&V
- Permit optimization of IV&V effort

## Case Studies

- Four IV&V projects from different domains
- Unmanned flight software
- Mission critical ground software
- Manned flight software
- Launch vehicle software
- Direct ROI
- Consider only direct development cost savings from early defect detection and correction
- Project data for defect identification (IV&V and Developer)
- Cost estimates from project data or COCOMO-II
- Published data for phase-based cost-to-fix scale factors

## Assumptions

- Cost-to-fix ratios conservatively based on published data
- Assume that developer would find all issues identified by IV&V in a phase later than IV&V found them
- Discovery distribution same as other issues identified by developer when we have that information
- Issues found in test would have required same EP to find as expended by IV&V
- All found in current release cycle (no leakage to operations)

## Defect Removal With IV&V

**Requirements **
**Phase**

**Design**
**Phase**

**Code **
**Phase**

**Test**
**Phase**

**Operations**

**Number of Defects Remaining**

## Defect Removal Without IV&V

**Requirements **
**Phase**

**Design**
**Phase**

**Code **
**Phase**

**Test**
**Phase**

**Operations**

**Number of Defects Remaining**

## Relative Cost-to-Fix Ratios

| Issue Type | Phase Issue Found |  |  |  |
| --- | --- | --- | --- | --- |
|  | Requirements | Design | Code | Test |
| Requirements | 1 | 5 | 10 | 50 |
| Design |  | 1 | 2 | 10 |
| Code |  |  | 1 | 5 |
| Test |  |  |  | 1 |

- These estimates are conservatively based on
- published results.

## Methodology – Rework Cost Available

- Actual cost to fix each issue (developer and IV&V) from project database
- Use project defect database to compute developer probability of detection for each error type and phase
- Assume developer would have located issues in test phase by expending additional effort  equal to IV&V effort

## ROI Computation

- Example – Requirements issue discovered by IV&V in design phase
- Assumptions
- Developer would have found in either code or test phase
- Probability of discovery in each phase same as similar issues actually discovered by developer
- W/O IV&V Cost = With IV&V Cost *(10 * Pc + 50*Pt)/5
- where Pc + Pt = 1

## ROI Computation

- ROI = W/O IV&V + IV&V test phase* – W/IV&V      -----------------------------------------------------
-                                 Total IV&V
- *    Assumes developer would have expended same effort in test phase to find issues as IV&V actually expended

## Methodology – Rework Cost not Available

- COCOMO-II cost estimating tool
- Measure of task size is source line of code (SLOC)
- Calibration provides conversion from SLOC to effort (EP-month)
- BRAK is the SLOC corresponding to the effort to fix defects
- BRAK based on Function Points (FP)
- Baseline is requirements error found in test phase – complete rework
- FP determined through analysis of defect data
- Early error detection reduces cost to fix

## BRAK Computation With IV&V

- BRAK = FP * *k**w* * *k**L* * *k**s*
*k**w* : Weighted Factor, applied per function point type and complexity
*k**L* : Language Conversion Factor, convert adjusted FP to SLOC
*k**s* : SLOC Reduction Factor, adjusts SLOC effort based on issue type and phase found
- Project independent
- For calculation of cost to fix IV&V issues

## With-IV&V SLOC Reduction Factors

| Issue type | Phase found | ks |
| --- | --- | --- |
| Requirements | Requirements | 0.02 |
| Requirements | Design | 0.1 |
| Requirements | Code | 0.2 |
| Requirements | Test | 1.0 |
| Design | Design | 0.02 |
| Design | Code | 0.04 |
| Design | Test | 0.2 |
| Code | Code | 0.02 |
| Code | Test | 0.1 |
| Test | Test | 0.02 |

## BRAK Computation Without IV&V

- BRAK = FP * *k**w* * *k**L* * *k**sD*
*k**sD* : distribute *k**s* for remaining phases assuming developer discovered defect
- Replaces *k**s** *in BRAK formula
- Weighted average of *k**s* for remaining phases
- Weighting factors are developer relative issue discovery rates
- Project dependent

## Direct ROI Computation

- ROI = (Cost without IV&V – Cost with IV&V)         -------------------------------------------------
-                            Cost of IV&V
- Cost with IV&V – total development cost actually experienced
- Developer costs include system engineering, code production,and test
- Used to calibrate COCOMO-II
- Cost without IV&V
- Use w/o IV&V BRAK to get w/o IV&V equivalent SLOC
- Calibrated COCOMO-II to get effort estimate

## ROI Summary

- Project A	2.07
- Project B	1.24
- Project C	4.93
- Project D	3.14

## Conclusions

- Early lifecycle activities yield largest incremental ROI
- Artifact of direct ROI model
- Ignores indirect benefits
- Ignores criticality and other characteristics that make issues have different values
- IV&V cost-effective in terms of direct ROI for all projects studied
- Direct ROI model is a viable measurement tool

## Acknowledgements

- This research was supported by Titan Corp. and the NASA IV&V Center
- Valuable contributions were made by Don Ohi, Gary Lovstuen, and Sarma Susarla of Titan Corp.