# Presentation: 170029

## Nancy S. Eickelmann, PhDMotorola Labs1303 E. Algonquin Rd.Annex-2Schaumburg, IL 60196Phone: (847) 310-0785Fax: (847) 576-3280Nancy.Eickelmann@motorola.com

## Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making

**FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP)**
**for the**
**NASA Independent Verification and Validation Facility**** **
**COTR: Ken McGill**
**PI: Nancy Eickelmann**
**S-54493-G**
**September 5, 2001**

**Notes:**
Phd in Computer Science at University of California Irvine
MS CS
MBA Information and Decision Systems
Bachelor’s in Finance

I’ve been in WV for a year working on the IV&V ROI project, just prior to that I was with Advanced Research Program at MCC after spending 4 years at Hughes Research Labs in Malibu CA.

## PROBLEM STATEMENT

** **
- This research addresses NASA’s need to evaluate the ROI and cost/benefit of applying IV&V technologies.
- A *prototype* is to be developed that will provide financial valuation of IV&V for a given program.
- The prototype will be developed using an iterative process that will incrementally implement the models and methodology researched and developed during *prior years of this effort*.
- The tool will be evaluated for usability, accuracy, and consistency through *limited use scenarios* with NASA program managers.

## Return on Investment - Status

**This project was funded July 20, 2001**
**Evaluation of data sets is in progress**
**Benchmarking for key factor target value ranges in progress**
**Model integration and interface to existing programs in progress, Ask Pete, ARRT**

## RESEARCH APPROACH

- Phase 1:
- Reduce the models we developed earlier to actionable guidelines for practice
- Phase 2:
- Introduce these models, processes and support tools to a small group of carefully selected pilot projects
- Evaluate the results of applying the tools and methods
- Phase 3:
- Use the feedback from step 3 to adapt the tools and methods for widespread dissemination, if warranted within the software project decision-making community at NASA.

## HYPOTHESES/OBJECTIVE

** **
- The IV&V valuation methodology will be iteratively refined based on feedback from NASA program managers and statistical evaluation of  the methodology and results.
- Specific factors to be evaluated:
- 	Hypothesis 1: The cost relative to the potential benefits of IV&V is inversely proportional to key organizational factors, such as the capability maturity of the development organization.
-  	Hypothesis 2: The realization of potential IV&V benefits is directly related to the development organizations’ acceptance of IV&V.
-  	Hypothesis 3: The cost/benefit ratio for IV&V is directly related to the criticality of the application (and its individual subsystems).

## IV&V YIELD

- Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V.
- If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit.
- Should the resources saved be less than the resources consumed, we have a net cost.

## IV&V Yield

- Cost of Quality
- Key components...
- Cost of Poor Quality
- Key components...

## What we already know...3 issues of empirical studies...

**	**
**June 5-6, 1986 the **_1st Workshop on Empirical Studies_** of Programmers, Washington, D.C.**
*Need scientific rigor...***“A Plan for Empirical Studies” Victor Basili**
*Need to look at real world variable values...***“By the Way, Did Anyone Study Real Programmers” Bill Curtis**
*Need to study PITL...***“Meeting the Challenge of Programming in the Large (PITL)” Elliot Soloway**** **
**	**
**	****	**
**	**

## Why is it Difficult to Apply Quantitative Management Principles for Software Engineering?

**SE domain has a large number of key variables that have different degrees of significance depending on the environment**
**SE domain has key variables that have extreme variance within the same environment (i.e., programmer productivity 10:1) **
**SE domain variables in combination may create a “critical**** ****mass” not present when variables are studied in isolation****  ****	****	**
**	**1986 IEEE TSE, Basili, Selby and Hutchins, Surveyed software engineering empirical studies published to date. Cited 116 published studies.

**Notes:**
The technical approach used to accomplish these goals required 
applying a three-tier product line engineering framework that leverages the methodology developed by the Software Productivity Consortium (SPC). 
	The three-tier conceptual framework was developed under the Department of Defense Comprehensive Approach to Reusable Defense Software (CARDS) program. 

## Iterative IV&V Methodology 

**Software Requirements  Analysis**

**Software Interface Analysis**

**Software Code Analysis**

**Developer Test Analysis**

**Software Design Analysis**

**Iterative IV&V Methodology**** **

**IV&V Planning**
**- Activities****	****	****- Organization****	****	****- CARA**
**- Schedules****	****	****- Tools****	****	****- WBS**

**Inputs**

**Activities**

**Outputs**

**Software**
**IV&V Plan**

**Critical/High Risk**
**Functions List**

**IV&V **
**Technical Reports**

**Software Problem**
**Reports**

**IV&V **
**Traceability**
**Matrix**

**Findings and **
**Recommendations**

**IV&V**
** ****Metrics**

**Monthly**
**Progress/Status**
**Reports**

**Iterative Per Software Release**

- Phase Dependent IV&V Tasks

- Phase Independent IV&V Tasks

**TRACEABILITY** **ANALYSIS**

**CHANGE IMPACT ANALYSIS**

**DELIVERABLES VALIDATION**

**TECHNICAL REVIEWS AND AUDITS**

**SPECIAL STUDIES**

## IV&V Technologies - COQ

## Empirical Research Summary

- Experimental Simulation
- 	Qualitative and quantitative results based on non-deterministic or hybrid simulation model
- Math Modeling quantitative results based on a deterministic model

**Mirrors a segment of the real world, control of variables is high, supports testing of causal hypothesis, results can be replicated,  high internal validity and generalizability**
**Captures real world context in which to isolate and control variables**

**Researcher bias can be introduced through selection of variables, parameters and assumptions concerning the model. Modeling requires high degree of analytical skill, and interdisciplinary knowledge**
**Results are not typically generalizable to other populations or environmental contexts, researcher bias is common, **

## Process Modeling and Simulation

**Managed, measured, productivity **
**gains through: **
** ****process improvement**
** ****data driven decision-making**
** ****technological innovation**
** ****Quantitative valuation of **
**COQ vs COPQ **

## COQ versus COPQ

## Process Simulation Models

- Experimental Simulation
- 	Qualitative and quantitative results based on _*non-deterministic*_ or hybrid simulation model
**mirrors a segment of the real world **
**control of variables is high **
**supports testing of causal hypothesis**
**results can be replicated**
**high internal validity **
**high external validity, generalizability**

## IV&V Yield

- Organizational context factors for cost
- Key components

## Independent Verification and Validation

- An organization *independent* from the developers study the artifacts of software production.
- This requires:
***Technical independence****. M*embers of the IV&V team may not be personnel involved in the development of the software.
- .***Managerial independence****. *The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software.*** ***
***Financial independence****.* Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software.
- IV&V is often perceived as testing the code after the development is completed NASA IV&V is full life cycle activities

## State of the Practice: Process Maturity

**State of the Practice: Process Maturity**

- Source: SEI Web Site SEMA Report for March 2000

## Measuring IV&V Effectiveness

**Notes:**
COTS Analytical Tool Availability:  RTM, DOORS, McCabe Toolset, MATRIXx, BONeS, MATLAB, Code Wizard, Refine/C, Refine/Ada, ClearCase, Rational Rose, TeamWork, .....etc.

## IndustryBenchmarking

|  | US Data |  | ? |  |
| --- | --- | --- | --- | --- |
|  | Average | Best inClass | Current Level | % Improve-ment |
| Productivity
(KAELOC per staff month) | 3.23 | 7.14 | 4.03 |  |
| Cost
(dev. cost per KAELOC) | $4,334 | $1,962 | $1,008 |  |
| Defect Content
(defects per KAELOC) | 15.6 | 8.1 | 18.8 |  |
| Defect Removal Efficiency (by Technology) | 95% | 99.5% | 92.7% |  |
| Cost of Delivered Defects
(post-release defects per KAELOC, Domain) | $1.5M |  |  |  |

**Industry****Benchmarking**

- Source for US Data: 	Capers Jones (2000) Software Assessments, Benchmarks, and Best Practice,	Addison-Wesley, p 339, System Software Baseline.

## IV&V Yield

- System factors for cost and gain

## Prior Empirical ROI Studies 

- ROI: Independent V&V Benefits

- IV&V applied early in the lifecycle has the greatest ROI.
- Source Jet Propulsion Laboratory TR.

## IMPACT of Major Air & Space Software Problems

- Aggregate Cost:

- Loss of Life:

- ‘93

- ‘96

- ‘97

- ‘98

- ‘99

- 3

- 160

- $640 million

- Loss of Data:

**[Poseidon]**

**[Galileo]**

**[Pathfinder]**

**[Lewis]**

- $116.8 million

**[NEAR]**

- $255 million

**[DS-1]**

**[Galileo]**

- $1.6 billion

- [‘99] – NASA IV&V presentation

**Notes:**
1993: Airbus A320 Faulty braking logic
1996: Ariane 5 Register over-flow error
          Flight 965 American Airlines had an incomplete vector (way-point)                      command
         Galileo processing exceeded allocation time
         Poseidon data processing error
1997: Lewis spacecraft design flaws and COTS incompatibility
         USAF STEP unanticipated state of execution
         Pathfinder scheduling problem caused the system to freeze up
1998: Delta 3 Improperly programmed flight computer, and design issues
          Zenit 2 premature executed guidance command
          NEAR (Near Earth Asteroid Rendezvous) improperly programmed safety limits
1999: Titan 4BInaccurate software load
          Orion 3 satellites failed to reach orbit (still under investigation)
          DS-1 software logic and scheduling problem
          Galileo timing issue in the fault protection software

##      Tracing Impacts to Causes... Cause-Effect Graphing

**Mission Success at Reduced Cost**

- Reliability Objective

- Safety Objective

- Cost Objective

- Process Improvement
- Avoid Rework
- Eliminate Redundancy
- Efficient Resource Allocation

- Skilled Workforce
- Domain Experts
- Engineers
- V&V Experts

- Skills training program

- IT Infrastructure, Web-based reporting, DSS, ARM, PITS, RMS, Ask Pete, ARRT
-  Communication Channels & Reporting

- PL Reuse Technologies
- Domain Engineering
- Knowledge Maintenance
- V&V Models and Methods

- Quality Objective

- Defect Prevention

- Defect Detection

- Identify and Eliminate Hazards

- Identify and Manage Risks

- Information Analysis
- & Information Management,

- Product Certification

## BSC Cause and Effect Graphing

**Strategic and Financial Goals**

- Reliability Objective

- Competitive Objective

- Cost Objective

- Process Improvement
- Avoid Rework
- Eliminate Redundancy
- Efficient Resource Allocation

- Skilled Workforce
-        Black Belts
- Engineers
- Telecom Experts

- Skills training program - Motorola University

- IT Infrastructure, Web-based reporting, COMPASS, TIGERS, TeamPlay, Communication Channels & Reporting

**SIX SIGMA**
- Performance Excellence
- Knowledge Maintenance
- Communications
-  Models and Methods

- Quality Objective

- Defect Prevention

- Defect Detection

- Optimize resource allocation & utilization

- Identify and Manage Risks

- Information Analysis
- & Information Management

- Product Certification

**BSC Cause and Effect Graphing**

## Filter Attributes

## DTE – Rule Based

## NEURAL NETWORK

## Intelligent update of rule structure

## STATISTICAL ANALYSIS

## BENEFITS

- The benefits of this proposed Center Initiative would be applicable to all NASA software development organizations for whom IV&V is an option. The formalization of an objective decision-making process, along with enabling support tools would _***provide key capabilities***_ to make rational budgetary decisions that impact safety and mission critical aspects of all NASA software systems. This is significant in enabling NASA to engage in _***effective administrative and managerial control***_ based on objective, quantified information.
- The techniques proposed under this initiative will also provide NASA participants _***increased visibility into their process improvement***_ efforts. The _***ISO-9001 certification requires***_ that managers be able to document the benefits contributed to the organization by specific processes and process improvement effort [8]. A formalized, well-defined decision-making process would therefore make a significant contribution to NASA’s overall quality strategy.

## MILESTONES

- Start=July 20, 2001 + 3 mo
*IV&V Process Description – Product Characterization*
- Based on prior CSIP results
- Start=July 20, 2001 + 6 mo
*Information Analysis*
- Data gathering for methodology
- Start=July 20, 2001 + 6 mo
*Initial Prototype Demonstration(s) & Iteration(s) Delivered*
- GSFC IV&V interface required

## Nancy S. Eickelmann, PhDMotorola Labs1303 E. Algonquin Rd.Annex-2Schaumburg, IL 60196Phone: (847) 310-0785Fax: (847) 576-3280Nancy.Eickelmann@motorola.com