# Presentation: 659465

## Quantitative Evaluation of Domain Testing 

**      ****Joel Henry, Ph.D****.**
**                          ****University of Montana**

**1st Annual NASA Office of Safety and Mission Assurance Software Assurance Symposium**
**                                                                                                   ****September 6, 2001**

## Overview 

- System development
- Testing problems
- Solution approach
- Results and conclusions
- Practical application

## Controls devices that control:
Wind generator
Model support
Tunnel atmosphere
Emphasizes reliability and safety
Utilizes multiple development strategies
 Based on simple structure

- Wind generator
- Model support
- Tunnel atmosphere
- Emphasizes reliability and safety
- Utilizes multiple development strategies
-  Based on simple structure

**System Development***Wind Tunnel Software*

## System Development Simple Structure

- Controlling
- Computers

- External
- Devices

_**ALGORITHM**_
**Sample Inputs**
**Run Software**
**Update Outputs**

## Engineer builds graphical model in MatrixX
Model enters simulate-debug-simulate-debug phase
Engineer autogenerates source code
Source code is compiled, linked, and then deployed
Hardware/software integration begins

- Model enters simulate-debug-simulate-debug phase
- Engineer autogenerates source code
- Source code is compiled, linked, and then deployed
- Hardware/software integration begins

**System Development**

## Recall simple model
Input variables – sampled every time step
Outputs variables – produced every time step
Sample time – system specified frequency
Consider test requirements
Input file/matrix
Output file/matrix
Analysis tools

- Input variables – sampled every time step
- Outputs variables – produced every time step
- Sample time – system specified frequency
- Consider test requirements
- Input file/matrix
- Output file/matrix
- Analysis tools

**Testing Problems*****Test Size***

## Consider an example
100 input variables
50 output variables
100 millisecond sample time
Assume you want to test a one hour operation period
100 inputs*10 per second*3600seconds = 3,600,000 values
50 outputs*10 per second*3600seconds = 1,800,000 values
Ignore issues of useful inputs and defect detection

- 100 input variables
- 50 output variables
- 100 millisecond sample time
- Assume you want to test a one hour operation period
- 100 inputs*10 per second*3600seconds = 3,600,000 values
- 50 outputs*10 per second*3600seconds = 1,800,000 values
- Ignore issues of useful inputs and defect detection

**Testing Problems*****Test Size***

## Domain determinants
Input variable – minimum, maximum, and accuracy
Output variable – minimum, maximum, and accuracy
Consider test requirements
Input file/matrix with all possible values for input
Output file/matrix much more complex problem

- Input variable – minimum, maximum, and accuracy
- Output variable – minimum, maximum, and accuracy
- Consider test requirements
- Input file/matrix with all possible values for input
- Output file/matrix much more complex problem

**Testing Problems*****Domain Coverage***

## Consider an example for input variable
Input variable for pressure in a tank
Min – 0
Max – 999.999
Accuracy – 3 (decimal places)
1000*1000 = 1,000,000 possible values
Ignore issues of legal sequencing and combinations

- Input variable for pressure in a tank
- Min – 0
- Max – 999.999
- Accuracy – 3 (decimal places)
- 1000*1000 = 1,000,000 possible values
- Ignore issues of legal sequencing and combinations

**Testing Problems*****Domain Coverage***

## Consider an example for output variable
Output variable for electric current
Min – 6400
Max – 31999.99
Accuracy – 2 (decimal places)
24800*100 = 2,480,000 possible values
Ignore issue of HOW to generate these values

- Output variable for electric current
- Min – 6400
- Max – 31999.99
- Accuracy – 2 (decimal places)
- 24800*100 = 2,480,000 possible values
- Ignore issue of HOW to generate these values

**Testing Problems*****Domain Coverage***

## Automation to:
Generate large input matrices\files
Perform simulation
Analyze output matrices\files
Methods to:
Evaluate domain coverage
Aid debugging
Suggest additional tests

- Generate large input matrices\files
- Perform simulation
- Analyze output matrices\files
- Methods to:
- Evaluate domain coverage
- Aid debugging
- Suggest additional tests

**Solution Approach***Overview*

## Control system for an air deflector sensor
Eight input variables
Six output variables
One hundred time steps
Test air deflector through the entire range of positions

- Eight input variables
- Six output variables
- One hundred time steps
- Test air deflector through the entire range of positions

**Solution Approach***Example – Air Deflector*

## Configure test for each input variable
Set test min, max, and accuracy
Select test type
Set number of test steps
Generate input matrix for simulation

- Set test min, max, and accuracy
- Select test type
- Set number of test steps
- Generate input matrix for simulation

**Solution Approach***Input Matrix*

## Consider an input variable for air deflector
Set test min to 0, test max to 10, and accuracy to 2
Select test type – Stabilizing
Set number of tests – 100 (ten seconds)

- Set test min to 0, test max to 10, and accuracy to 2
- Select test type – Stabilizing
- Set number of tests – 100 (ten seconds)

**Solution Approach***Input Matrix Example*

## Launch simulation automatically
Capture output matrix
Specify exception criteria for each output
Set min, max, accuracy
Specify defect criteria
Inspect input, output, or both

- Capture output matrix
- Specify exception criteria for each output
- Set min, max, accuracy
- Specify defect criteria
- Inspect input, output, or both

**Solution Approach***Simulation*

## Partition input variable domain into percentiles
Assume defects not uniformly distributed across domain
Assume defects are uniformly distributed within percentiles
Evaluate percentile domain coverage for input and output variables
Report testing completeness, MTTF, confidence

- Assume defects not uniformly distributed across domain
- Assume defects are uniformly distributed within percentiles
- Evaluate percentile domain coverage for input and output variables
- Report testing completeness, MTTF, confidence

**Solution Approach***Evaluate Domain Coverage*

## After a test simulation is run
Recall input test type – Stabilizing

For input air deflector input percentiles 1-7, 9-15, 16-26 untested

- Recall input test type – Stabilizing
- For input air deflector input percentiles 1-7, 9-15, 16-26 untested

**Solution Approach***Evaluate Domain Coverage - Example*

## Evaluate test for this input variable
75 of 100 test percentiles tested – 75%
100 of 1000 possible test values – 10%
Percentile completeness – 0% - 90%
Need to improve
Increase percentiles tested
Increase percentile completeness

- 75 of 100 test percentiles tested – 75%
- 100 of 1000 possible test values – 10%
- Percentile completeness – 0% - 90%
- Need to improve
- Increase percentiles tested
- Increase percentile completeness

**Solution Approach***Evaluate Domain Coverage - Example*

## Evaluate test results for an output variable:

Percentiles 73-100 uncovered
Four defects detected – MTTF 10/4 => 2.5 seconds

- Percentiles 73-100 uncovered
- Four defects detected – MTTF 10/4 => 2.5 seconds

**Solution Approach***Evaluate Domain Coverage - Example*

**Current to Motor**

## Capture percentile combinations:
Capture input variable percentiles for each defect
Find repeated combinations - pattern matching 
Suggest additional tests
Focus test to generate defects based on combinations
Investigate error leading to defect

- Capture input variable percentiles for each defect
- Find repeated combinations - pattern matching
- Suggest additional tests
- Focus test to generate defects based on combinations
- Investigate error leading to defect

**Solution Approach***Aid in Debugging*

## Solution ApproachAid in Debugging - Example

## Percentile combinations
Input variables 1 and 2 – percentiles 1, 2, and 3
Input variables 6, 7, and 8 – percentiles 99 and 100
Retest to debug
Replicate the defect
Focus debugging
Leverage application area knowledge

- Input variables 1 and 2 – percentiles 1, 2, and 3
- Input variables 6, 7, and 8 – percentiles 99 and 100
- Retest to debug
- Replicate the defect
- Focus debugging
- Leverage application area knowledge

**Solution Approach***Aid in Debugging - Example*

## Execute multiple tests
Evaluate testing effectiveness
Track trends in software reliability
Automate and evaluate
Specify effectiveness and reliability goals
Evaluate on a per test case basis
Track through testing phase over all tests

- Evaluate testing effectiveness
- Track trends in software reliability
- Automate and evaluate
- Specify effectiveness and reliability goals
- Evaluate on a per test case basis
- Track through testing phase over all tests

**Results and Conclusions**

## Results and ConclusionsExample – Multiple Tests

- MTTF (Seconds)

## Results and ConclusionsExample – Multiple Tests

**Bucket Coverage (%)**

## Results and ConclusionsExample – Multiple Tests

## Results and ConclusionsExample – Multiple Tests

## Can do domain testing supported with automation
Can set quantitative goals
Can evaluate progress toward goals
Can measure MTTF, domain coverage, confidence percentages, and reliability
Can create an organizational history

- Can set quantitative goals
- Can evaluate progress toward goals
- Can measure MTTF, domain coverage, confidence percentages, and reliability
- Can create an organizational history

**Practical Application**** ***What?*

## Invest in automation
Integrate domain coverage with application specific testing
Establish goals and collect data
Calculate MTTF, domain coverage, confidence percentages, and reliability
Use common sense with quantitative data

- Integrate domain coverage with application specific testing
- Establish goals and collect data
- Calculate MTTF, domain coverage, confidence percentages, and reliability
- Use common sense with quantitative data

**Practical Application**** ***How?*

## Questions and Contact Info

- Joel Henry
- _[henryj@cs.umt.edu](mailto:henryj@cs.umt.edu)_
- MATT and RATT
**http://www.cs.umt.edu/RTSL/design992/links/index.htm**
- Matlab and MatrixX users -