# Presentation: 518539

## Software Reliability Assurance for Real-time Systems 

**      ****Joel Henry, Ph.D****.**
**                          ****University of Montana**

**NASA Software Assurance Symposium**
**                                                                                                   ****September 4, 2002**

## Overview 

- System development
- Testing problems
- Solution approach
- Results and conclusions
- Practical application

## Controls devices that control:
Wind generator
Model support
Tunnel atmosphere
Emphasizes reliability and safety
Utilizes multiple development strategies
 Based on simple structure

- Wind generator
- Model support
- Tunnel atmosphere
- Emphasizes reliability and safety
- Utilizes multiple development strategies
-  Based on simple structure

**System Development***Example: Wind Tunnel Software*

## System Development Simple Structure

- Controlling
- Computers

- External
- Devices

_**ALGORITHM**_
**Sample Inputs**
**Run Software**
**Update Outputs**

## Engineer builds graphical model in MATLAB
Models enter simulate-debug-simulate-debug phase
Engineer auto-generates source code
Source code is compiled, linked, and then deployed
Hardware/software integration begins

- Models enter simulate-debug-simulate-debug phase
- Engineer auto-generates source code
- Source code is compiled, linked, and then deployed
- Hardware/software integration begins

**System Development**

## Recall simple model
Input variables – sampled over time
Outputs variables – produced over time
Sample time – variable or set frequency
Consider test requirements
Input file/matrix
Output file/matrix
Analysis tools

- Input variables – sampled over time
- Outputs variables – produced over time
- Sample time – variable or set frequency
- Consider test requirements
- Input file/matrix
- Output file/matrix
- Analysis tools

**Testing Problems*****Test Size***

## Consider an example
100 input variables
50 output variables
100 millisecond sample time
Assume you want to test a one hour operation period
100 inputs*10 per second*3600seconds = 3,600,000 values
50 outputs*10 per second*3600seconds = 1,800,000 values
Ignore issues of useful inputs and defect detection

- 100 input variables
- 50 output variables
- 100 millisecond sample time
- Assume you want to test a one hour operation period
- 100 inputs*10 per second*3600seconds = 3,600,000 values
- 50 outputs*10 per second*3600seconds = 1,800,000 values
- Ignore issues of useful inputs and defect detection

**Testing Problems*****Test Size***

## Domain determinants
Input variable – minimum, maximum, and accuracy
Output variable – minimum, maximum, and accuracy
Consider test requirements
Input file/matrix with all possible values for input
Output file/matrix much more complex problem

- Input variable – minimum, maximum, and accuracy
- Output variable – minimum, maximum, and accuracy
- Consider test requirements
- Input file/matrix with all possible values for input
- Output file/matrix much more complex problem

**Testing Problems*****Domain Coverage***

## Consider an example for input variable
Input variable for pressure in a tank
Min – 0
Max – 999.999
Accuracy – 3 (decimal places)
1000*1000 = 1,000,000 possible values
Ignore issues of legal sequencing and combinations

- Input variable for pressure in a tank
- Min – 0
- Max – 999.999
- Accuracy – 3 (decimal places)
- 1000*1000 = 1,000,000 possible values
- Ignore issues of legal sequencing and combinations

**Testing Problems*****Domain Coverage***

## Automation to:
Generate large input matrices/files
Perform simulation and/or test autogenerated code
Analyze output matrices/files
Methods to:
Evaluate domain coverage
Aid debugging
Evaluate results

- Generate large input matrices/files
- Perform simulation and/or test autogenerated code
- Analyze output matrices/files
- Methods to:
- Evaluate domain coverage
- Aid debugging
- Evaluate results

**Solution Approach***Overview*

## Solution Approach

**Command and Control Algorithm**

** **

** **

- Generate Tests

- Simulate Model

- Test Auto-code

- Detect Faults

- Evaluate Results

- MATLAB/Simulink Environment

**Source**

** **

**Code**

**Executable **

** **

**Code**

- Verification and Validation
- Methodology

**Solution Approach**

**Suite of testing tools**

## Solution Approach

** **

**Generate Tests**

** **

**Simulate Model**

** **

**Test Auto-code**

** **

**Detect Faults**

** **

**Evaluate Results**

** **

**Suite of testing tools**

**Verification and Validation **
**Methodology**

**Command and Control Algorithm**

** **

- MATLAB/Simulink Environment

**Model Information**

**Test Data**

**Test Results**

**Executable**
**Code**
** **

**Test Data**

**Test Results**

**Solution Approach**

## Solution Approach

**Generate Tests**

** **

**Simulate Model**

** **

**Test Auto-code**

** **

**Detect Faults**

** **

**Evaluate Results**

** **

**Suite of testing tools**

**Verification and Validation **
**Methodology**

**Data Graphs**

**Raw Value Files**

**Completeness, MTTF,**
**Reliability File**

## Execute multiple tests
Evaluate testing effectiveness
Track trends in model reliability
Automate and evaluate
Specify effectiveness and reliability goals
Evaluate on a per test case basis
Track through testing phase over all tests

- Evaluate testing effectiveness
- Track trends in model reliability
- Automate and evaluate
- Specify effectiveness and reliability goals
- Evaluate on a per test case basis
- Track through testing phase over all tests

**Results and Conclusions**

## Results and ConclusionsExample – Multiple Tests

## Results and ConclusionsExample – Multiple Tests

**Bucket Coverage (%)**

## Results and ConclusionsExample – Multiple Tests

## Results and ConclusionsExample – Multiple Tests

## Can do domain testing supported with automation
Can set quantitative goals
Can evaluate progress toward goals
Can measure MTTF, domain coverage, confidence percentages, and reliability
Can create an organizational history

- Can set quantitative goals
- Can evaluate progress toward goals
- Can measure MTTF, domain coverage, confidence percentages, and reliability
- Can create an organizational history

**Practical Application**** ***What?*

## Invest in automation
Integrate domain coverage with application specific testing
Establish goals and collect data
Calculate MTTF, domain coverage, and reliability
Use common sense with quantitative data

- Integrate domain coverage with application specific testing
- Establish goals and collect data
- Calculate MTTF, domain coverage, and reliability
- Use common sense with quantitative data

**Practical Application**** ***How?*

## Questions and Contact Info

- Joel Henry
**henryj@cs.umt.edu**
- MATT and RATT
**http://www.cs.umt.edu/RTSL/design992/links/index.htm**
- MATLAB users -