# Presentation: 248501

## Making Use of a Decade of Widely Varying Historical Data

- Ms. Sally Godfrey (NASA POC)
- Mr. Andre Bechtel
- Mr. Raimund L. Feldmann
- Ms. Myrna Regardie
- Dr. Carolyn Seaman

**Making Use of a Decade of Widely Varying Historical Data**

**SARP project**
**“****Full Life-cycle Defect Management”**

## We are in the second year of our initiative and studying
Parameters that affect the results of inspection
The relation between V&V effectiveness in early lifecycle (e.g., inspection) and late (testing)

We are using this information to provide feedback and decision support to NASA projects, on questions such as:

Can I get guidance on how to plan my inspections based on resultsfrom projects like my own?

Based on my inspection results,what are the implications for the effortrequired to be spent on other non-optionalactivities, like system testing?

- Parameters that affect the results of inspection
- The relation between V&V effectiveness in early lifecycle (e.g., inspection) and late (testing)
- We are using this information to provide feedback and decision support to NASA projects, on questions such as:
- Can I get guidance on how to plan my inspections based on resultsfrom projects like my own?
- Based on my inspection results,what are the implications for the effortrequired to be spent on other non-optionalactivities, like system testing?

**Problem we are addressing**

## Our approach

- Literature Recommendations

- Historical Baseline Models

- Users:
- Projects
- SEPG
- Inspection Planners
- Researchers

- Current Model Formulation

- Outputs:
- Automated feedback
- What if Analysis
- Experience Bases
- Trends
- ...

- Inputs from:
- GSFC
- GRC
- JPL
- JSC
- MSFC
*More to come...*

**Notes:**
We are working with teams to contribute data that summarizes inspections
Consistent with NPR 7150.2 and usual practice
Historic data – to better calibrate the models
Current data – to understand current practice and provide feedback that can help ongoing projects
We can provide automatic wrappers and import functionalities

Data protection
We aren’t using Project names or defect descriptions
We are anonymizing and aggregating data
Only a project team will be able to see results for their project

## First year results

- Collected more than 2,529 inspection records in our database
- Evaluated old classification schema
- Developed new classification based on existing standards and the collected data
- Mapped data into new classification schema
- Developed prototype tool to support planning and reporting
- Incorporated latest analyses and models based on the data
- Designed capabilities for accepting data from various forms (e.g., JPL forms) as well as various databases
- Gained feedback on usability and possible enhancements
- Created central inspection experience base
- Provides materials necessary for applying inspections in various contexts: e.g., defect type definitions, mapping to various taxonomies, checklists, forms, ...

## Unifying different defect classifications

***Motivation*****:** Valuable defect data has been collected over the years across many Centers and projects
***Issue***: Different defect classifications used in historic and contemporary data sets, as well as across and within Centers
***Action***: Define a unified defect classification schema along with a mapping to existing data sets
***Benefits*****:**
- Leverages data required by NPR 7150.2 for analysis and feedback to teams
- Enables monitoring and validation of existing guidelines
- Unified classification schema is applicable to inspections and testing

## Mapping the different data sets

**historic data sets**

**contemporary data**

**actions**

- A7

**A**

**B**

**Y**

**Z**

**A**

**B’**

**W**

**Y’**

**Z**

**A**

**C**

**W’**

**Y**

**Z**

- Select candidate defect categorization scheme(e.g., ODC)

- ODC

## Mapping algorithm

- Any historical
- datasets left?

- end

- begin

- no

- Any defect
- categories
- left in **X**?

- Choose historical dataset **X**

- no

- Choose defect category **C****X**

- Remove category **C****X**

- Assign all historical defects in **C****X** into this category

- Use the assignment algorithm  to distribute defects in **C****X**

- Add **C****X** to **S**, along with all its defects.

- Is **C****X**
- already in **S**? (under another name)?

- yes

- yes

- yes

- Can all defects in **C****X** be included under any single existing category in **S**?

- Assign all historical defects in **C****X** into this category

- yes

- Can all defects in **C****X**** **be included under multiple (existing) categories in **S**?

- yes

- no

- no

- Remove **X** from historical datasets

- Select new categorization schema **S**(e.g., ODC)

- no

## Updating existing inspection guidelines

***Motivation*****:** NASA guidelines for effective inspections (e.g., 3 points of control) were formulated in early 1990’s
***Issue***: Development procedures (e.g., standards, languages, etc.) have changed over time;  New factors must be considered
***Action***:
- Validate guidelines based on a wider set of recent data;
- Refine the guidelines if needed (e.g., by adding more variables, tailoring to different domains, etc.)
- Integrate them into an inspection support tool and training courses
***Benefits*****:** Refined guidelines will increase **effectiveness** of inspections and provide better user guidance

## User guidance based on heuristics

**User guidance based on heuristics**

- optimum

- acceptable

- border
- line

- problematic

- border
- line

- problematic

**defines**

****

****

****

****

****

****

****

## Example: Comparison of team size

- Target team size: optimal is 4 to 6; borderline is 3 or 7

**Example: Comparison of team size**

## Comparing test and inspection data

***Motivation*****:** Better knowledge of inspection’s strengths & weaknesses could be used to better allocate resources among V&V activities.
***Issue***: Defects that slip through inspections aren’t found until much later; different defect type descriptors mean they often are hard to compare.
***Action***: Compare test and inspection defect profiles (on the same projects or within the same domain)
***Benefits*****: **Past knowledge about recurring defect types can be used to select the right overall strategy for optimal V&V planning
***Research Questions*****:**
- What defects types are typically removed by inspections vs. testing?
- What project characteristics (size, language, software domain, new development/enhancements) influence the types of defects found?
- What percent of logic errors can be expected to be removed by inspections?
- Can test results be used for post-mortem analysis ofinspection performance?

## Overview: Inspections vs. testing

**Overview: Inspections vs. testing**

## Initial results: Across projects

***Research Question*****: **What defect types are typically removed by inspections vs. testing? In this domain:
-  Overall the defect removal profile seems similar, but
-  Inspections found on average 64% of the total system defects

## Initial results: Within a project

***Research Question*****:** What defect types are typically removed by inspections vs. testing? Specifically, for a maintenance project:
-  Many more internal interface defects were found by inspections

## Improving tool support

***Motivation*****:** Data and resources from across NASA, that use different taxonomies, cannot easily be leveraged without centralized tool support.
***Issue***: Need to do mappings and analysis without requiring extra steps from the user, and to seamlessly integrate the results.
***Action***:
- Centralize existing materials and resources  Experience Base;
- Integrate Experience Base and results data into a combined dashboard
***Benefits*****: **Integrating real-time feedback into normal engineering activities, for:
- The planning of inspections,
- Collection of data,
- Analysis and building of up-to-date baselines,
- Feedback and improvement.

## Providing an inspection experience base

**http://fc-md.umd.edu/EB/**

## Improving tool support for inspections

- built-in experience base access

- reporting and evaluation capabilities

- file import capability

**7**

**Notes:**
Mention that as soon as a value is entered the traffic light reports (since there are now effort hours entered yet the traffic light is not reporting)
Note: We could do a face demo (animated ppt sequence)  – e.g., have a popup come up for entering the effort or better a single value such as the number of participants or LoC – fill in a number and have the traffic light change – need to know what you want here!

## Future work

**Refine the test and inspection data comparison**
- Obtain additional data sets for testing and refining our preliminary conclusions
- Integrate test results into inspection tool
**Initial deployment of tool **
- Obtain additional feedback on usability and future deployment
- Pursue expansion of the Experience Base with testing-related materials a centralized site for V&V resources
**Integrating with other existing inspection data forms and tool support**
- Especially eRoom-based tool available through Kevin Carmichael / GRC

## Contact information

**Forrest Shull (PI)**
- fshull@fc-md.umd.edu
- 301-403-8970

**Contact information**

**Sally Godfrey (NASA POC)**
- sara.godfrey@nasa.gov
- 301-286-5706

**Myrna Regardie**
- mregardie@fc-md.umd.edu
- 301-403-2050