# Presentation: 849127

## PI: Katerina Goseva – Popstojanova

Students: Sunil Kamavaram & Olaolu Adekunle

		Lane Department of  Computer Science and Electrical Engineering
   			 West Virginia University, Morgantown, WV

 				   katerina@csee.wvu.edu

- Students: Sunil Kamavaram & Olaolu Adekunle
- 		Lane Department of  Computer Science and Electrical Engineering
-    			 West Virginia University, Morgantown, WV
-  				   _[katerina@csee.wvu.edu](mailto:katerina@csee.wvu.edu)_

**Real-World Software ****Reliability Assessment****	**(WVU UI#7: Sensitivity of Software Reliability to Operational Profile Errors: Architecture-Based Approach)

## What we are doing?

- 		Anyone can see a fire
-    What we need are smoke
- 				 detectors

- But what about the sensitivity and
- accuracy of the alarms ?

## Problem statement & Our goal

- Traditional view: Point estimate of software reliability computed from the model using point estimates of input parameters
- Problem: Estimation of a trustworthy operational profile is difficult
- IV&V information on operational profiles  - limited, may be inaccurate
- Single operational profile could not be sufficient to describe the use by different users
- Software systems evolve - operational profile may change
- Our goal: Reliability “sensitometer” that enables us to answer the question “How parameters uncertainty propagates into overall application reliability?”
- Develop an architecture-based methodology for uncertainty analysis of software reliability & apply it on case studies

## What we can do?

- Benefits to IV&V
- Software reliability assessment throughout the life cycle (keeping track of the software evolution)
- Allocation of testing efforts
- Software certification

## Architecture - based methodology for uncertainty analysis

## Methods for uncertainty analysis

- Sensitivity
- studies

- Entropy

- Confidence
- intervals

- Analytical

- Monte Carlo
- simulation

- Method of moments

- Perturbation
- analysis

## Choice of the method

- Choose the method using the following criteria
- Data requirements & ability to collect data
- Reliability measures
- Accuracy of the solution
- Scalability with respect to the number of components
- Our goal: fill the table

| Method  | Data requirements | Reliability
measures | Accuracy of the solution | Scalability |
| --- | --- | --- | --- | --- |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |

## Construction of the software architecture model

- Structural phase – establishment
-   of static software architecture
-  Software specifications
-  Architectural design
-  Parser-based or lexically based tools
-    (SIAT tool - Titan Systems Corporation)

- Statistical phase – estimation of
-   the relative frequencies of component
-   interactions, that is, transition probabilities
-  Uniform distribution – maximum entropy approach
-  Historical data
-  Software specification (e.g. UML use case & sequence diagrams)
-  Component traces from profiles or test coverage tools
-         (Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU)

## European Space Agency case study

- Two faulty versions were obtained
- reinserting the real faults discovered
- during the integration testing and
- operational usage

- Component traces obtained during testing were
- used for constructing software architecture &
- estimating transition probabilities

- Almost 10.000 lines of C code
- The program has been extensively used after the last fault removal without failures; this gold version is used as an oracle

## Parameter estimation

- Two versions
- Version A: faulty components 1&2, fault-free component 3
- Version B: faulty components 2, fault-free components 1&3
- Transition probabilities
-    where        is the number of times
-     control was transferred from
-     component *i** to component  **j*, and
- Component reliability
-     where      is the number
-     of failures and       is the
-     number of executions of
-     component *i** *in N randomly
-     generated test cases

| Version |  R1 |  R2 |  R3 |
| --- | --- | --- | --- |
| A | 0.8428 | 0.8346 | 1 |
| B | 1 | 0.8346 | 1 |

## Construction of the architecture – based software reliability model

**E**

## Traditional View: Point estimates of software reliability  

- Actual reliability of the software
- 	where *F** *is the number of system failures in N randomly generated test cases
- Estimated reliability from the  model
- Results

| Version | Actual reliability | Estimated
reliability | Error |
| --- | --- | --- | --- |
| A | 0.7393 | 0.7601 | 2.81% |
| B | 0.8782 | 0.8782 | 0% |

## Methods for uncertainty analysis

- Sensitivity
- studies

- Entropy

- Confidence
- intervals

- Analytical

- Monte Carlo
- simulation

- Method of moments

- Perturbation
- analysis

## Sensitivity of software reliability to variations in operational profile

- Version A reliability

- Version B reliability

- Rmax = 0.8414
- Rmin = 0.7048

- Rmax = 0.9983
- Rmin = 0.8363

## Methods for uncertainty analysis

- Sensitivity
- studies

- Entropy

- Confidence
- intervals

- Analytical

- Monte Carlo
- simulation

- Method of moments

- Perturbation
- analysis

## Uncertainty study based on entropy

- Entropy quantifies the uncertainty present in a stochastic source
- 	where      represents the usage distribution and     the transition probabilities
- Higher entropy implies an exponentially greater number of statistically typical paths
- Maximum entropy – all transitions that are exit arcs from each state are equiprobable

## Uncertainty of the operational profile

- Hmax = 0.5514
- Hmin  = 0.0404

-  Operational profile A (H=0.4707) is more uncertain than operational profile B (H=0.4604)
-  Software systems that have uniform operational profile are more uncertain and thus would require more testing

- Hmax = 0.5514
- Hmin  = 0.0404

## Uncertainty of software reliability

- Operational profile

- Considering software failure behavior increases the uncertainty for both versions compared to the uncertainty due to operational profile
-  Version B, which is more reliable, is less uncertain  than version A

- Version A uncertainty

- Version B uncertainty

- Version A reliability

- Version B reliability

## Uncertainty of components for the operational profile

- Uncertainty of component *i* is estimated using the conditional entropy
- Uncertainty of component *i *will be higher if it transfers the control to more components and the transition probabilities are equiprobable

- Version A

- Version B

## Uncertainty of components for the software reliability model

- Uncertainty of component 1 version B remains the same because
- For all other components uncertainty increases due to
- Components that have higher expected execution rate, higher component uncertainty, and moderate reliability should be allocated more testing effort

- Version B

- Version A

## Methods for uncertainty analysis

- Sensitivity
- studies

- Entropy

- Confidence
- intervals

- Analytical

- Monte Carlo
- simulation

- Method of moments

- Perturbation
- analysis

## Uncertainty study based on the method of moments

- Method of moments involves the following steps
- Obtain the expression for the system reliability using the architecture-based software reliability model
- Expand the expression for system reliability using Taylor series
- Determine the moments of the components reliabilities
- Estimate the mean and the variance of the system reliability using the parameter moments and Taylor series coefficients

## First order Taylor series

- First order Taylor series expansion
- 	where                    is the mean component reliability, and
- Mean reliability is
- Variance of the reliability is
- 	where                      is the variance of the component reliability

## Second order Taylor series

-  Second order Taylor series expansion

-  Mean reliability is

-  Variance of the reliability is

## Method of moments for the case study

- Second order approximation does not improve accuracy significantly

|  |  | First order Taylor series | Second order
Taylor series |
| --- | --- | --- | --- |
|  | Mean reliability | 0.7601 | 0.7601 |
| Version A | Standard deviation | 0.0825 | 0.0825 |
|  | Variance | 0.0068 | 0.0068 |
|  | Mean reliability | 0.8782 | 0.8782 |
| Version B  | Standard deviation | 0.0589 | 0.0589 |
|  | Variance | 0.0035 | 0.0035 |

- Version B is more reliable with less variance of the reliability

## Methods for uncertainty analysis

- Sensitivity
- studies

- Entropy

- Confidence
- intervals

- Analytical

- Monte Carlo
- simulation

- Method of moments

- Perturbation
- analysis

## Uncertainty study based on Monte Carlo simulation

- Monte Carlo simulation involves the following steps
- Obtain the expression for the system reliability using the architecture-based software reliability model
- Assign probability distributions to the transition probabilities and components reliabilities
- Sample the distributions
- Compute the reliability of the system using the sampled values
- Repeat steps 3&4 until the desired number of values of system reliability has been generated
- Calculate the moments, frequency chart and percentiles  for the system reliability, do the distribution fitting

## Variation of the operational profile:Frequency chart and distribution fitting

| Mean | 0.7600 |
| --- | --- |
| Standard deviation (Spread of the distribution) | 0.0210 |
| Variance (Spread of the distribution) | 0.0004 |
| Skewness (Lean of the distribution) | 0.2072 |
| Kurtosis (Peakedness of the distribution) | 2.6047 |

## Variation of the operational profile:

- Percentiles

## Convergence of the mean

- The estimation of the mean reliability converges after around 3000 iterations

- Mean reliability =0.7600

## Variation of the operational profile:

-  Reliability is also sensitive to p12;  the variance is negative

- Sensitivity measured by contribution to variance

## Variation of the operational profile and component reliabilities: Frequency charts

- Version B

|  | Version A | Version B |
| --- | --- | --- |
| Mean | 0.7589 | 0.8780 |
| Standard deviation (Spread of the distribution) | 0.0860 | 0.0660 |
| Variance (Spread of the distribution) | 0.0074 | 0.0044 |
| Coefficient of variation  (Relative measure of spread) | 0.1493 | 0.0752 |
| Skewness (Lean of the distribution) | -0.5190 | -0.9646 |
| Kurtosis (Peakedness of the distribution) | 3.1367 | 4.2254 |

## Variation of the operational profile and component reliabilities: Distribution fitting & percentiles

- Version B

## Making a choice

| Method  | Data requirements | Reliability measures  | Accuracy of the solution | Scalability |
| --- | --- | --- | --- | --- |
| Sensitivity | Point estimates | Sensitivity of the point estimate | Exact analytical solution | Large systems |
| Entropy | Point estimates | NA | Exact analytical solution | Large systems |
| Method of 
moments | Moments of the parameters |  Moments | Approximate solution: accuracy may be increased by higher order Taylor series | Small to medium systems |
| Monte Carlo simulation | Distribution functions of the parameters
Generation of random numbers
 | Distribution
Moments | Approximate solution: accuracy may be increased by increasing the sample size
Sampling errors may be involved in case of long tail distributions | Large systems |

## Accomplishments

- Architecture-based methodology for uncertainty analysis of software reliability was developed
- Four different methods already developed
- These methods were illustrated on the European Space Agency software

## Future work

- Develop other methods for uncertainty analysis
- Complete “Make a choice” table
- Apply & validate all methods using NASA case studies
- SIAT tool - Titan Systems Corporation
- Testing tool for JSC AERCam project - Dr.Yann-Hang Lee, ASU