# Presentation: 037058

## Achieving High Software Reliability Using a Faster, Easier and Cheaper Method

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

- Taghi M. Khoshgoftaar

- The Software Measurement Analysis and Reliability Toolkit

## Outline

- Introduction
- Overview of SMART
- Data Analysis and Modeling Features
- Current Utilization of SMART
- Case-Based Reasoning
- Empirical Study
- Conclusions

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Introduction

- Necessity of an integrated tool for efficient empirical software quality research
- Commercial packages are available but expensive and don’t always match our exact needs
- In house development gives us availability, flexibility and possibility to evolve

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Overview of SMART

- First version back in 1998
- Current version 2.0
- Written in Microsoft Visual C++
- Runs on Microsoft Windows based platforms
- User friendly GUI

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## SMART GUI

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## SMART GUI

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Features included in SMART

- Data management
- Multiple Linear Regression (MLR)
- Case-based reasoning (CBR),
- Case-based reasoning with two group clustering
- Case-based reasoning with three group clustering
- Module order modeling (MOM)

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Organizational Flowchart

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Current Utilization of SMARTat the ESEL Laboratory

- Empirical research:
- Comparative studies of software quality models
- Case studies based on real world systems

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Based Reasoning

- Based on automated reasoning processes
- Easy to use
- Results are easy to understand and to interpret
- Looks at past cases that are similar to the present case in an attempt to predict or classify an instance

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Based Reasoning: Additional Advantages

- The ability to alert users when a new case is outside the bounds of current experience
- The ability to interpret the automated classification through the detailed description of the most similar case
- The ability to take advantage of new or revised information as it becomes available
- The ability for fast retrieval as the size of the library scales up

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Based Reasoning

**Working hypothesis for software quality modeling:**
- Current cases that are in development will more than likely be fault-prone if past cases having similar attributes were fault-prone

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Based Reasoning: Comparing the Cases

- Similar cases to a new module or nearest neighbors are determined by similarity functions:
- Absolute Distance
- Euclidean Distance
- Mahalonobis Distance

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Based Reasoning: Prediction Methods

- The value of the dependent variable is estimated using the values of the dependent variables of the nearest neighbors and a solution algorithm:
- Unweighted Average
- Inverse-Distance Weighted Average

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Based Reasoning: Classification Methods

- Used to classify a software module into a particular class (fault-prone, not fault-prone).
- The types of classification methods include:
- Majority Voting
- Data Clustering

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study: System Description

- Two data sets were obtained from two large Windows-based applications used primarily for customizing the configuration of wireless products. The data sets were obtained from the initial release of these applications. The applications are written in C++, and they provide similar functionality.

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study: System Description

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study:Data Collection Effort

- Data collected by engineers over several months using the available information in:
- Configuration Management Systems
- Problem Reporting Systems

## Case Study:Independent Variables

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study:Accuracy Evaluation

- Average Absolute Error:
- Average Relative Error:

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study:Prediction Results

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study:Classification Evaluation

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study:Classification Results

- Entire data set:
- Fit and Test data set:

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Case Study:Return On Investment

- Classification using CBR

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Conclusion

- A tool that matches our needs
- Used for our extensive empirical work
- Proved useful on large scale case study
- Faster
- Easier
- Cheaper
- Ready for future enhancement

***NASA OSMA SAS '01***

**September 5-7, 2001**** **

## Reminder

- We will be presenting the tool on Friday, Please feel free to visit us!

***NASA OSMA SAS '01***

**September 5-7, 2001**** **