# Presentation: 224102

## FY 2004 Initiative: Risk Assessment of Software Architectures

- Hany Ammar, Katerina Goseva-Popstojanova,Ajith Guedem, Kalaivani Appukkutty,
- Walid AbdelMoez, and Ahmad Hassan
***LANE Department of Computer Science and Electrical Engineering******West Virginia University***

**Less risk, sooner**
**WVU UI: **
**Risk Assessment of Software Architectures**

## Outline

**Introduction**
**Severity Analysis Methodology **
**Requirement Risk Analysis Methodology**
**Relevance to NASA Case studies**
**Conclusion**

## Introduction

- Risk is a combination of: The probability of failure and the consequence of the failure (Severity)
- Severity analysis is conducted on Unified Modeling Language (UML) models to quantify the severity of failure
*Severity is estimated based on cost of failure and Hazard analysis techniques:  *
*Functional Failure Analysis (**FFA**)*
*Failure Mode and Effects Analysis (**FMEA**)*
*Fault Tree Analysis (**FTA**)** *
- Requirement Risk is assessed based on: Probability of failure (Normalized Dynamic Complexity) and Severity

## Outline

**Introduction**
**Severity Analysis Methodology **
**Requirement Risk Analysis Methodology**
**Relevance to NASA Case studies**
**Conclusion**

## Proposed Severity Analysis Methodology

***INPUT:****** DYNAMIC UML SPECIFICATIONS***
*System level scenarios related to use cases, annotated with the **	**following:*
*** ******Guide words ****(Omission, Commission, Late, etc)*
***Cost of hazard***
*Sequence diagram that show the component/connector interactions  annotated with the following:*
***Component states, probability that a component fails in a given failure mode, and the cost of  failure in each state***
***Messages transmitted between components, probability that connectors fail to transmit a message, and cost of failure of each message***
***OUTPUT:   ******COMPONENTS/CONNECTORS and SCENARIO SEVERITY***

## Proposed severity analysis methodology

***Proposed severity analysis methodology***

**Scenario Severity**

**Component/Connector Severity**

## Proposed Severity Analysis Methodology

- Identify system hazards:  states of the system that can contribute to accidents and mishaps
***	******Perform FFA using system scenario diagrams as an input***
- Identify component/connector failure modes
***	******Perform FMEA using UML Sequence Diagrams as an input ***
- Construct a detailed cause-and-effect model, to record how failures propagate from component/connector level to the system level
***	******FTA is used to combine the outputs from FFA and FMEA ***
- Develop the *Cost of Failure Graph** *to estimate severity of each component/connector in a given scenario, or severity level of the scenario
- Estimate severity of component/connector and system scenario using cost of failure graph and cost-severity graph** **
*	*The final result is a table of component/connector severity and system scenario severity.

## Step 1: FFA Table 

| Event Name | Class of failure | Failure | Effects on System | Cost of Failure ($) | Comments |
| --- | --- | --- | --- | --- | --- |
| Command | Value | A Fault in processing command routine | Heart is continuously triggered but device is still monitored by physician, need immediate fix or disable. | 1000 | The  component received the command misinterpret it |
| VSense | Omission | Timer not set correctly | No pace is generated for the heart, patient could die | 100000 | Some component’s timers does not work well |
| Pace | Commission | Pacing hardware device malfunctioning | Heart is always paced while patient condition requires only pacing the heart when no pulse is detected. Heart operation is irregular | 100000 | Some component’s sensor failed to sense the heart. |

## Step 2: Example (Pacemaker AVI Scenario)

***Sequence diagram of the AVI scenario***

## Step 2: FMEA Table

| Component | Failure Mode | Effect on the system | Cause of failure | Cost of failure mode ($) |
| --- | --- | --- | --- | --- |
| AR  | ToOn Value Error
(AR stuck in Idle state) | The component will not work and there is no pace of the heart | The component miss interpret signal received from CG
 | 1000 |
| - | AR stuck in Refractory State | The component will stay in Refractory state and there is no pace | Connector VT-AR sends a wrong message, or component AR failes to understand the message | 1000 |
| - | The component receive GotVSense but there is no pace
(AR stuck in Waiting state) | The component will stay in waiting state and there is no pace | The component sensor does not work | 100000 |
| ... |  |  |  |  |

## Step 3: Fault Tree Analysis (FTA)

**Step 3: Fault Tree Analysis (FTA)**

- Step 1:Top event hazard identified by applying FFA

- Step 2: Component/Connector failure modes identified through FMEA

- Step3: Combining the results of steps 1 and 2 to build a cause-effect model by applying FTA

- Commission “*Pace”* Fault Tree

## Step 4: Cost of Failure Graph forAR Component

**System Level hazards (Fault Trees Top Event**)

- Commission “*Pace*”

-  “*Command*”  Value Error

*P(*“Command”
- Value Error*)*=0.02

*P(Commission “Pace”)* = 0.65

**Failure Modes probabilities**
**of *****AR***** Component**

**Consequence (Cost) of**
*** ******AR***** Failure Modes**

*AR *“stuck in Refractory”  State  *P(“*stuck in Refractory”*)* = 0.05

***AR*** Cost
- Of Failure

- Sense TimeOut Error
*P(“*Sense TimeOut Error”*)* = 0.1

- PaceTimeOut Error
*P(“*PaceTimeOut Error”*)* = 0.03

- AR “stuck in Pace” State
*P(*“stuck in Pace”*)* = 0.03

*AR *failed to handle ToOn
*P(“*failed to handle ToOn”*)*
- =.09

- AR “ stuck in Waiting” State
*P(“*stuck in Waiting”*)* = 0.05

- $ 1000
- (Regular Care)

- $ 1000
- (Regular Care)

- $ 100000
- (Intensive Care)

- $ 100000
- (Intensive Care)

- $ 100000
- (Intensive Care)

- $ 100000
- (Intensive Care)

- Sense TimeOut Error
*P(“*Sense TimeOut” Error*)* = 0.1

*P(Omission *
*“**VSense")* = 0.5

*Omission “VSense”*

## Step 5: Severity of components/connectors

**Cost-Severity Graph**

## Step 5: Component/connector severity and scenario severity

| Component/connector name | Severity |
| --- | --- |
| Connector CG-AR | 0.50 |
| Connector CG-VT | 0.50 |
| Connector AR-VT | 0.94 |
| Connector VT-AR | 0.95 |
| Component CG | 0.05 |
| Component AR | 0.96 |
| Component VT | 0.95 |

***Severity of components/connectors ***
***in the AVI scenario ***

| Scenario | Severity |
| --- | --- |
| AVI Scenario | 0.94 |

***AVI scenario severity***

## Outline

**Introduction**
**Severity Analysis Methodology **
**Requirement Risk Analysis Methodology**
**Relevance to NASA Case studies**
**Conclusion**

## Proposed Requirement Risk Matrix

| Requirements | Scenarios | Failure Modes |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | FM1 | FM2 | ... | FMn |
| R1 | S1 |  | Rf |  |  |
|  | S2 |  |  |  |  |
| R2 | S3 |  |  |  |  |
| ... |  |  |  |  |  |
| Rm |  |  |  |  |  |

***Risk factor of scenario S1 in Failure mode FM2***

**Requirements are mapped to UML use case and scenarios**
**A failure mode refers to the way in which a scenario fails to achieve its requirement**
**Failure mode of a scenario is determined based on the severity methodology**

**According to Dr. Martin Feather’s DDP Process, “The Requirements matrix maps the impacts of each failure mode (should it occur) on  each requirement.”**

**Impact Matrix (DDP Process)**

## Calculating Risk Factor

**The dynamic complexity is calculated at the scenario level**
**The risk factor of a particular scenario (a), in a particular failure mode (b)  is:**
**Rf****ab**** = Normalized Dynamic Complexity****ab****  X  Severity****ab**
**Severity of a scenario in a specific failure mode is estimated using the previously discussed severity analysis methodology ***[Slide 6]*

## Calculating Complexity

**We estimate complexity as:**
**Complexity = CC * Msg**
**	****	****	****	****	****	****	****	****	**
**CC** : McCabe’s cyclomatic complexity of a scenario for each failure mode
**Msg**: The number of messages exchanged between the components in the sequence diagram to the point where the failure mode occurs in the scenario
- 	According to Fenton et al, “The product of cyclomatic complexity (CC) and sigFF was shown to be a good predictor of fault proneness. ..The combined metrics appear to be better than both sigFF and Cyclomatic complexity on their own and also better than the size metric”

## Proposed Requirement Risk Analysis Methodology

**STEP 1:** Map the requirements to UML sequence diagrams
***For each Sequence diagram***
**STEP 2:** Construct the control flow graph of the scenario from the sequence diagram
**STEP 3:** Identify the different failure modes on the sequence diagram and the control flow graph
***For each failure mode***
**STEP 4:** Assess the severity of the failure mode using Function Failure Analysis (FFA)
**STEP 5:** Determine the cyclomatic complexity of the sub-control flow graph
**STEP 6:** For the failure mode, measure the number of messages exchanged between the components in the sequence diagram.
**STEP 7:** Calculate the complexity of the scenario for the failure mode as:
-           **Cyclomatic complexity * Number of messages**
**STEP 8:** Calculate the Risk factor of the scenario for the failure mode as:  **Complexity * Severity**
***End For each failure mode***
***End for each sequence diagram***

## Atrial-Ventricular Inhibit Scenario: Control Flow Graph (Obtained from Sequence Diagram)

- Requirement R2 of Pacemaker : Mapped to Atrial-Ventricular Inhibit Scenario

- Failure Modes and Msg Values marked in the sequence diagram

## Calculating Risk

**Calculating Complexity**
**Severity index of failure modes FM1, FM2 and FM3 of AVI scenario is 0.95**
**Risk is calculated as the product of Complexity and Severity index**

| Failure Modes | Cyclomatic Complexity
(CC) | Number of Messages (Msg) | Complexity 
(CC * Msg) | Normalized Complexity |
| --- | --- | --- | --- | --- |
| FM1 | 3 | 4 | 12 | 0.111 |
| FM2 | 6 | 7 | 42 | 0.389 |
| FM3 | 6 | 9 | 54 | 0.5 |

| 
Requirements | 
Scenarios | Failure Modes |  |  |
| --- | --- | --- | --- | --- |
|  |  | FM1 | FM2 | FM3 |
| R2 | AVI | 0.11 | 0.37 | 0.475 |

## Requirement Risk Matrix for Pace Maker 

| 
Requirements | 
Scenarios | Failure Modes |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | FM1 | FM2 | FM3 | FM4 | FM5 |
| R1 | Programming |  |  |  | 0.017 | 0.233 |
| R2 | Atrial-Ventricular
Inhibit | 0.11 | 0.37 | 0.475 |  |  |
| R3 | Atrial-Atrial
Inhibit | 0.151 | 0.283 | 0.566 |  |  |

## Outline

**Introduction**
**Severity Analysis Methodology **
**Requirement Risk Analysis Methodology**
**Relevance to NASA Case studies**
**Conclusion**

## Severity Analysis of EOS

**Severity Analysis of EOS**

**Severity Analysis Methodology is applied on NASA’s Earth Observing System (EOS)**

| Event Name | Actor sends or receives this event |
| --- | --- |
| Status | From IDB (external database) to system |
| Uplink Command | From system to Space craft |

**Preplanned emergency scenario**

## FFA Table for EOS 

- FFA Table for EOS

| SEQUENCE DIAGRAM | EVENT | FAILURE | EFFECTS | COST | SEVERITY |
| --- | --- | --- | --- | --- | --- |
| Preplanned Emergency | System retrieves the Instrument status from IDB database | IDB takes longer time to return the status | Emergency cannot be handled at proper time | Mission Loss | Catastrophic |
|  | System uplinks commands to the space craft | Commands don’t reach the space craft at proper time | Command groups cannot be up linked in time | Mission Loss | Catastrophic |

## Requirement Risk Matrix for  HCS case study

| 
Requirements | 
Scenarios | Failure Modes |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | FM1 | FM2 | FM3 | FM4 | FM5 |
| R1 | Retry both Pumps | 0.029 | 0.366 | 0.289 |  |  |
| R2 | Single LT Mode |  |  |  | 0.059 | 0.662 |

## Outline

**Introduction**
**Severity Analysis Methodology **
**Requirement Risk Analysis Methodology**
**Relevance to NASA Case studies**
**Conclusion**

## Conclusion

**We have ****developed a methodology to assess severity**** of failures of components, connectors, and scenarios based on UML models**
**We have ****developed a methodology for assessing requirements based risk**** using normalized dynamic complexity and severity of failures**
**These methodologies were applied on ****NASA case studies**
**Our future work will address the validation of risk methodology using software artifacts and data from the ****Metrics Data Program**