# Presentation: 544188

## A Framework for Early Reliability Assessment

- Bojan Cukic, Erdogan Gunel, Harshinder Singh,
- Lan Guo, Dejan Desovski
- West Virginia University
- Carol Smidts, Ming Li
- University of Maryland

- (WVU UI: *Integrating Formal Methods and Testing in a Quantitative *
*Software Reliability Assessment Framework 2003) *

## Overview

**Introduction and Motivation.**
**Software Reliability Corroboration Approach. **
**Case Studies.**
**Applying Dempster Shafer Inference to NASA datasets. **
**Summary and Further Work.**

## Introduction

**Quantification of the effects of V&V activities is always desirable.**
**Is software reliability quantification practical for safety/mission critical systems?**
- Time and cost considerations may limit the appeal.
- Reliability growth applicable only to integration testing, the tail end of V&V.
- Estimation of operational usage profiles is rare.

## Is SRE Impractical for NASA IV&V?

**Most IV&V techniques are qualitative in nature.**
*Mature* software reliability estimation methods based exclusively on testing.
**Can IV&V techniques be utilized for reliability?**
- Requirements readings, inspections, problem reports and tracking, unit level tests...

- Req        Design	 Code   	       Test (Verification & Validation)
- 			Unit	 **Integration****	****Acceptance**

**Life cycle long IV&V Implementation**

*Traditional*
*Software *
*Reliability*
*Assessment*
*Techniques*

## Contribution

**Develop software reliability assessment methods that build on:**
**Stable and mature** development environments.
**Lifecycle long** IV&V activities.
- Utilize all relevant available information
- Static (SIAT), dynamic, requirements problems, severities.
- Qualitative (*formal and informal*) IV&V methods.
**Strengthening the case for IV&V across NASA enterprise.**
- Accurate, stable reliability measurement and tracking.
- Available throughout the development lifecycle.

## Assessment vs. Corroboration

**Current thinking**
- Software reliability “tested into” the product through the integration and acceptance testing.
**Our thinking**
- Why “waste” the results of all the qualitative IV&V activities.
- Testing should *corroborate* that the life-cycle long IV&V techniques are giving the “usual” results, that the project follows usual quality patterns.

## Approach

- Software quality
- Measures (SQM)

- Reliability Prediction
- Systems (RPS)

- RPS Combination
- Techniques

- SW Reliability
- Corroboration
- Testing

- SQM1

- SQM3

- SQM2

- SQM4

- SQM6

- SQM5

- SQMi

- SQMj

- RPS1

- RPS2

- RPSk

- RPSm

- .  .  .

- RPS Combination (Experience,
- Learning, Dempster-Schafer...)

- BHT software reliability corroboration

- Null Hypothesis, H0
- Alternative Hypothesis, Ha

- Software Development Lifecycle

***Trustworthy Software ***
***Reliability Measure***

## Software Quality Measures (roots)

**The following ones used in experiments.**
- Lines of code
- Defect density
- No defect that remain unresolved after testing, divided by the LOC.
- Test coverage
- LOCtested / LOCtotal.
- Requirements traceability
- RT= #_requirements_implemented/#_original_requirements.
- Function points
- .  .  .
**In principle, any measures available could/should be taken into account.  **
- Defining appropriate Reliability Prediction Systems (RPS).

## Reliability Prediction Systems

**An RPS is a complete set of measures from which software reliability can be predicted.**
**The bridge between an RPS and software reliability is a MODEL.**
**Therefore, select (and collect) those measures that have the highest relevance to reliability.  **
- Relevance to reliability ranked from expert opinions [Smidts 2002].

## RPS for Test Coverage

** **

## Approach

- Software quality
- Measures (SQM)

- Reliability Prediction
- Systems (RPS)

- RPS Combination
- Techniques

- SW Reliability
- Corroboration
- Testing

- SQM1

- SQM3

- SQM2

- SQM4

- SQM6

- SQM5

- SQMi

- SQMj

- RPS1

- RPS2

- RPSk

- RPSm

- .  .  .

- RPS Combination (Experience,
- Learning, Dempster-Schafer...)

- BHT software reliability corroboration

- Null Hypothesis, H0
- Alternative Hypothesis, Ha

- Software Development Lifecycle

- Software Reliability
- Measure

## Reliability “worthiness” of different RPS

| Measure/RPS | Relevance to Reliability |
| --- | --- |
| Failure rate | 0.98 |
| Test coverage | 0.83 |
| Mutation score | 0.75 |
| Fault density | 0.73 |
| Requirement specification change request | 0.64 |
| Class coupling | 0.45 |
| No. class methods | 0.45 |
| Man hours/defect detected | 0.45 |
| Function point analysis | 0.00 |

**32 measures ****ranked by ****five experts**

## Combining RPS

**Weighted sums used in initial experiments.**
- RPS results weighted by the expert opinion index.
- Removing inherent dependencies/correlations.
**Dempster-Shafer (D-S) belief networks approach developed. **
- Network automatically built from datasets by the Induction Algorithm.
**Existence of suitable NASA datasets? **
- Pursuing leads with several CMM level 5 companies.

## Approach

- Software quality
- Measures (SQM)

- Reliability Prediction
- Systems (RPS)

- RPS Combination
- Techniques

- SW Reliability
- Corroboration
- Testing

- SQM1

- SQM3

- SQM2

- SQM4

- SQM6

- SQM5

- SQMi

- SQMj

- RPS1

- RPS2

- RPSk

- RPSm

- .  .  .

- RPS Combination (Experience,
- Learning, Dempster-Schafer...)

- BHT software reliability corroboration

- Null Hypothesis, H0
- Alternative Hypothesis, Ha

- Software Development Lifecycle

- Software Reliability
- Prediction

## Bayesian Inference

**Allows for the inclusion of imprecise (subjective) probability of failure.**
**Subjective estimate reflects **_*beliefs.*_
**Hypothesis on the event occurrence probability is combined with new evidence, which may change the degree of belief.**

## Bayesian Hypothesis Testing (BHT)

**Hypothesized reliability H****0**** comes as a result of RPS combination. **
**Based on the level of (in)experience, the degree of belief assigned: P(H****0****).**
**Corroboration testing now looks for the evidence in favor of the hypothesized reliability.**
*    **H**o** :**	**  ****<= ****o  ** **	**null hypothesis**    H**1** : **	**  ****> ****o**	**alternative hypothesis.*

## The number of corroboration tests 

| o | P(H0) | n0 | n1 | n2 |
| --- | --- | --- | --- | --- |
| 0.01 | 0.01 | 457 | 476 | 497 |
| 0.001 | 0.01  | 2378 | 2671 | 2975 |
| 0.0001 | 0.01 | 6831 | 10648 | 14501 |
| 0.01 | 0.1 | 228 | 258 | 289 |
| 0.001 | 0.1 | 636 | 1017 | 1402 |
| 0.0001 | 0.1 | 853 | 3157 | 6150 |
| 0.01 | 0.4 | 90 | 128 | 167 |
| 0.001 | 0.4 | 138 | 411 | 739 |
| 0.0001 | 0.4 | 146 | 1251 | 3260 |
| 0.01 | 0.6 | 50 | 87 | 126 |
| 0.001 | 0.6 | 63 | 269 | 552 |
| 0.0001 | 0.6 | 65 | 827 | 2458 |

**The number of corroboration tests **
**according to BHT theory**

## Controlled Experiments

**Two independently developed versions of PACS (smart card based access control).**
- Controlled requirements document (NSA specs).

## RPS Experimentation

| Measure | Relevance to reliability | Predicted failure rate |
| --- | --- | --- |
| Code defect density | 0.85 | 0.078 |
| Test coverage | 0.83 | 0.092 |
| Requirements traceability | 0.45 | 0.078 |
| Function point analysis | 0.00 | 0.0020 |
| Bugs per line of code (Gaffney) | 0.00 | 0.000028 |

- RPS predictions of system failure rates:

***Predicted Failure Rate: 0.084***
***Actual Failure Rate: 0.09***

## Reliability Corroboration 

**Accurate predictors appear adequate **
- Low levels of trust in the prediction accuracy.
- No experience in repeatability at this point in time.

| o | P(Ho) | # corroboration  tests |
| --- | --- | --- |
| 0.09 | 0.01 | 72 |
| 0.09 | 0.1 | 47 |
| 0.09 | 0.2 | 39 |
| 0.09 | 0.5 | 25 |
| 0.09 | 0.6 | 20 |

## “Research Side Products”

**Significant amount of time spent studying and developing Dempster-Shafer inference networks. **
**“****No hope” of demonstrating this work within the scope of integrating RPS results. **
- Availability of suitable datasets.
**But, some datasets are available. **_***So, use them for D-S demo!***_
- Predicting fault-prone modules in two NASA projects (KC2, JM1)
**KC2 contains over 3,000 modules, 520 modules of research interest**
**106 modules have errors, ranging from 1 to 13**
**414 modules are error free**
**JM1 contains 10,883 modules**
**2,105 modules have errors, ranging**** ****from 1 to 26**
**8,778 modules are error free  **
**Each dataset contains 21 software metrics, mainly McCabe and Halstead**

## How D-S Networks Work

**Combining distinct sources of evidence by the D-S scheme. **
**Building D-S networks by *****prediction logic.***
- Nodes connected by implication rules.
- Each implication rule assigned a specific weight.
**Updating belief for the corresponding nodes **
- Propagating the updated belief to the neighboring nodes, and throughout the entire network.
**D-S network can be tuned for a various range of verification requirements.  **

## D-S Networks vs. Logistic Regression

**KC2**

**JM1**

## D-S Networks vs. ROCKY

**KC2**

**JM1**

## D-S Networks vs. See5

**KC2**

**JM1**

## D-S Networks vs. WEKA

**KC2 dataset**

## D-S Networks vs. WEKA

**JM1**

## Status and Perspectives

**Software reliability corroboration allows:**
- Inclusion of IV&V quality measures and activities into the reliability assessment.
- A significant reduction in the number of (corroboration) tests.
- Software reliability of safety/mission critical systems can be assessed with a reasonable effort.
**Research directions.**
- Further experimentation (***data sets***, measures, repeatability).
- Defining RPS based on the “formality” of the IV&V methods.