# Presentation: 849255

## Preventing and Eliminating Software Faults through the Life Cycle

- PI: Katerina Goseva-PopstojanovaStudent: Margaret HamillLane Dept.  Computer Science and Electrical EngineeringWest Virginia University, Morgantown, WV
- E-mail: Katerina.Goseva@mail.wvu.edu

## Problem

- NASA spends time and effort to track problem reports/change data for every project. These repositories are rich, underused sources about the way software systems fail and the software faults that cause these failures.
- Our goal: Based on systematic and thorough analysis of the available empirical data, build quantitative and qualitative knowledge that contributes towards improvement of software quality by
- preventing introduction of faults into the system
- more efficiently eliminating them through the life cycle
- compiling lessons learned & recommendations to be used by the pilot project and throughout the Agency

## Approach

- Propose appropriate classification scheme

- Refine

- Identify the most frequent fault types & most frequent failure types

- Compile Lessons Learned & Recommendations document

- Compile a check list to support avoidance & elimination
- of different faults types

## Pilot study: Basic facts

- A large NASA mission consisting of millions of lines of code
- 21 CSCIs, over 8,000 files
- Developed iteratively at two different locations
- Each CSCI has multiple releases with added functionality
- We analyzed over 2,800 Software Change Requests (SCRs) collected in a period of almost 10 years

| CSCI id | # of 
releases | # of
files | #non conf
SCRs | non conf SCRs
per file |
| --- | --- | --- | --- | --- |
| 1 | 1 | 207 | 392 | 1.89 |
| 2 | 2 | 200 | 24 | 0.12 |
| 3 | 2 | 287 | 9 | 0.03 |
| 4 | 2 | 228 | 16 | 0.07 |
| 5 | 2 | 269 | 13 | 0.05 |
| 6 | 2 | 321 | 22 | 0.07 |
| 7 | 2 | 289 | 77 | 0.27 |
| 8 | 2 | 270 | 107 | 0.40 |
| 9 | 2 | 125 | 21 | 0.17 |
| 10 | 3 | 356 | 27 | 0.08 |
| 11 | 3 | 444 | 43 | 0.10 |
| 12 | 3 | 277 | 12 | 0.04 |
| 13 | 3 | 599 | 121 | 0.20 |
| 14 | 3 | 280 | 83 | 0.30 |
| 15 | 3 | 81 | 108 | 1.33 |
| 16 | 3 | 169 | 135 | 0.80 |
| 17 | 4 | 747 | 201 | 0.27 |
| 18 | 4 | 587 | 211 | 0.36 |
| 19 | 5 | 552 | 263 | 0.48 |
| 20 | 7 | 1368 | 861 | 0.63 |
| 21 | 7 | 415 | 112 | 0.27 |
| Total | -- | 8071 | 2858 | 0.35 |

## Technical challenges

- Assuring data quality is an important step of any empirical research effort
- Inaccurate data may lead to misleading observations
- Both the IV&V team and the project team have been extremely valuable in helping us to understand the change tracking system, determine the meaning of different attributes, and verify the quality of the data
- The research approach and analysis techniques can be used by any project that tracks problem reports/change data
- However, due to the lack of a unified change tracking system, some amount of unique work on exploration of the data format and automation of data extraction may be needed

## Current capability

- Fault localization [SAS 07]
- 58-62% of failures map to faults in more than one file
- Sources of failures (i.e., type of faults)
- Dominating fault types: requirements faults (33%), coding faults (33%), data problems (14%)
- Activities when the problem was discovered (e.g., inspection, testing, analysis, on-orbit)
- Almost 50% discovered by analysis, only 3% on-orbit
- Severity
- Around 8% of SCRs are safety critical, less than 1% on-orbit
- Lessons learned & recommendations for product and process improvements
- Compiled a document for internal use based on our results and the feedback from the IV&V team and project team

## Current capability: Identifying the most common fault types

- Current capability: Identifying the most common fault types

- Most common fault types
- Requirements faults: 33%
- Coding faults: 33%
- Data problems: 14%

- Early life cycle activities (requirements & design): 38.25%
- Late life cycle activities (coding, interface & integration): 48.57%
- This finding contradicts the common belief that majority of faults are introduced
- during early life cycle activities [Boehm et al.1975, Endres 1975, Basili et al.1984]

## Current capability: Distribution of fault types Development & testing vs. On-orbit 

- Current capability: Distribution of fault types Development & testing vs. On-orbit

- Only 3% of SCRs are on-orbit.
- Note the logarithmic scale of Y axes.

## Current capability: Safety critical SCRs across fault types

- Current capability: Safety critical SCRs across fault types

- Only around 8% of all SCRs are safety critical.
- The highest percentage of safety critical SCRs comes from coding faults (3.60%), followed by requirement faults, design faults & data problems (3.35% total).

## Based on our results and the feedback from the IV&V team and project team we compiled a document for internal use which summarizes the Lessons learned & Recommendations for product and process improvement

Prevent the introduction of faults and improve the effectiveness of detection    
Example: Increase effort spent on design and implementation of data repository used to share data between CSCIs

Improve the quality of the data & change tracking process
Example: Ensure the changes to the software artifacts (e.g., requirements, code, etc) made to fix the problem are recorded and can be easily associated with a specific SCR

- Based on our results and the feedback from the IV&V team and project team we compiled a document for internal use which summarizes the Lessons learned & Recommendations for product and process improvement
- Prevent the introduction of faults and improve the effectiveness of detection
- Example: Increase effort spent on design and implementation of data repository used to share data between CSCIs
- Improve the quality of the data & change tracking process
- Example: Ensure the changes to the software artifacts (e.g., requirements, code, etc) made to fix the problem are recorded and can be easily associated with a specific SCR

- Current capability: Lessons Learned / Recommendations

## Planned capability [FY08-10]

- Classify faults and failures using several additional  attributes
- Conduct more complex, multivariable analysis
- Continually update Lessons Learned & Recommendations for Improvement document
- Explore the best ways to prevent and eliminate most common faults and failures throughout the lifecycle; compile the results in a check list
- Increase awareness of our work so other projects within NASA can benefit from it

## Relevance to NASA

- Our research is based on a large NASA mission as a pilot study
- The research work is done in active collaboration with the IV&V team and the project team
- To the best of our knowledge, this is the largest dataset considered so far in the published literature
- Quantitative and qualitative results are regularly delivered and discussed with the project
- Based on the internal & external validity of our results, we believe that observed trends are intrinsic  characteristics of large, complex software systems
- Other similar projects across the Agency may benefit
- Lessons Learned & Recommendations can be used proactively by newer projects

## Acknowledgements

- We thank the following NASA civil servants and contractors for their valuable support!

- Jill Broadwater
- Pete Cerna
- Susan Creasy
- Randolph Copeland
- James Dalton
- Bryan Fritch
- Nick Guerra
- John Hinkle

- Lynda Kelsoe
- Debbie Miele
- Lisa Montgomery
- Don Ohi
- Chad Pokryzwa
- David Pruett
- Timothy Plew
- Scott Radabaugh