# Presentation: 468835

**Introduction**

**Putting Safety**
**In the Software**

## Presenter

- Agency Software Safety & Assurance Manager
- Formerly: Software Product Assurance Lead Senior Software Systems Engineer at NASA Glenn Research Center in Cleveland, OH

**Martha Wetherholt**

- _[martha.wetherholt@hq.nasa.gov](mailto:martha.wetherholt@hq.nasa.gov)_
**(202) 358 - 0470**

- Over 20 Years in Software
- 11 years in software assurance/software safety
- Experience as technical supervisor, software project lead, software developer

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Who is this class aimed at?

- Managers who need an ideal of what software safety is all about and what might be required to analyze, develop, and manage it.
- Developers who need an understanding of software safety.
- Safety Personnel not familiar with software processes.
- Will NOT go into detail on specific regulations (FAA, DoD, NASA, etc.).

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## What we will cover

- What is Safety Critical Software?
- System Safety and Hazard Determination
- Class exercise
- Development and Analysis techniques by lifecycle phase
- Tailoring the Safety Effort
- Good Programming Practices for Safety
- Resources for further information

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**If Software can’t hurt you, .....?**

**Software Safety**
**A Practical Approach**

## Software Safety, a Practical Approach

- Getting Started
- Setting the Scene
- Why care
- Definitions
- Software’s contribution to hazards
- Software’s contribution to safety controls

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
I’m, sure some of this will be singing to the choir but if so, just hang in there while we address those not so sure of Software’s role in safety and as we lay out the terms we will be using as we go through this class.
We want you to think of software as part of the safety critical system, a major part.  And we want folks to realize software’s contributions, both the good and the potentially bad.
How does software contribute to the hazardous world we know and are trying to build
We want you to see software in a whole new light, see it as a contributor to system hazards and also as a possible fix or mitigation to some of those Hazards.

## Why you really care about SW Safety

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Many people still claim that software can’t hurt you, unless the paper work falls on you.  When we even take a cursory look to see what software controls and monitors in our life, I’m not sure how those folks can be so blind.  Software has become and integral part of all our lives, from transportation to medical devices to cash flow.

## Why you really care about SW Safety

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Even when we are not using software to control either a few thousand tons of metal flying through the air and gliding along on some rails, or a pocket-watch sized heart pacer; we use it to provide and display critical information such as the various status information for controlling a nuclear power plant  or the heart beat of an in vitro infant. 

## What is meant by “safety”

- A system/product is _***Safe***_** **when:
- There is little to no chance (low probability) for it to blow up, break, malfunction, or otherwise fail in such a way as to _potentially_ injure someone (low impact)
- Something is _***Critical ***_when there is a potential  for:
**Serious injury or death**
**Serious impact to the bottom line, or **
**Bad publicity, public reputation**
**Vital information is accessible to the wrong folks**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
First let’s talk about just what is “safety”, what does it mean if something is safe or unsafe.  Each organization needs to define their definition and even what levels of safety to consider.  We will talk more on this later but as a quick point of reference, NASA usually defines 4 categories:     Catastrophic, Critical, Marginal, Negligible.  Each has a lesser degree of injury to personnel and equipment. But the basics are, will it or can it hurt some one? 
While, there is no defense against stupidity and ignorance but you try to be as safe as possible.  We all know of the cases where some genius decides his lawnmower is a good hedge clipper as well.
Some failures can impact your business (incorrect ATM functions, international bank transfers, etc.), your reputation, and can lead to liability. These may also be considered under your definition of Critical software, but they are not necessarily a “safety” concern.  Of course if your ATM doesn’t deliver the money to pay-off Guido, your loan shark, that might be classified as “safety critical”.
And, no matter what, you feed the lawyers – failure, of any type, often leads to liability and lawsuits! 
There are a lot of terms we use, and we need to be careful with them, notice the “could”, “potentially”, 
, “when used...”   We need to talk a common language and that language is ... next slide

## Definitions

- Hazard
- Hazard cause
- Hazard control
- Error
- Fault
- Failure

- Mishap/Accident
- Risk
- Criticality
- Severity
- Reliability

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
So, you are thoroughly convinced that Software has a role to play in safer systems. Before we go too far, we need to make sure that we understand if software has a role in contributing to a possible ‘failure”? Accident? Error? Hazard? Lets define some terms so we are talking the same language
Most of these terms we will cover over the next hour but the main ones to concentrate on are the following...

## Definitions - Risk

- Since we want to make software safer, or at least the systems they control, monitor and verify, we need to look at software and the system before it is designed, _when danger is only a risk_
- But how risky?
**What is the potential or likelihood for an accident to occur?**
**How bad of an accident could it cause?**
- The combination determines the ‘risk’, and from that we try to speculate on the proper amount of effort necessary to assure the software operates safely.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Again, we struggle with not only defining safety and criticality, we must categorize it. Not all situations have the same potential for destruction either the extent of damage or the chance of the damage even occurring.

## Definitions - Hazards

- A _**hazard**_ is the presence of a potential risk situation caused by an unsafe act or condition.  Every hazard has at least one cause, which in turn can lead to a number of ***effects*** (i.e., damage, illness, etc.).
- A _**hazard cause**_ is typically a fault, error, or defect in either the hardware or software or it can be a human operator or programmer error.
- A _**hazard control**_ is a method for preventing the hazard from occurring or lessening it’s effect if it does occur.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
For NASA, first and fore most we try to eliminate Hazards by design choices where possible, 
those we can’t eliminate, and there are quite a few in our business, we need to Mitigate and Control.  
When Mitigation can’t lessen either the likelihood of occurrence or reduce the potential impact of the hazard, then we must provide safe guards to prevent inadvertent use or exposure, - physical barriers in the hardware world.
That failing to be practical, we resort to warnings.
This will be discussed further in a software light later in this presentation/paper.

Of course, the control part of the software is then “safety critical” as well as any software routines and algorithms used to detect critical hardware and environment situations.

hardware (e.g. pressure relief valve)
software (e.g. detection of stuck valve and automatic response to open secondary valve)
operator procedures
combination  of methods

## Definitions: Faults, Failures & Errors

***Definitions: Faults, Failures & Errors***

_**Fault**_ - Any change in state of an item which is considered to be anomalous and may warrant some type of corrective action.
_**Failure**_ - The inability of a system or component to perform its required functions within specified performance requirements
- A fault may or may not lead to a failure.
- One or more faults can become a failure.
- All failures are the result of one or more faults.
_**Error**_ – Human mistake which could cause a failure
- Mistake in engineering/development: requirement specification, design, implementation
-  Mistake during operation.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Fault - Examples of faults included device errors reported by BIT/BITE, out of limits conditions on sensor values, loss of communication with devices, loss of power to a device, communication error on bus transaction, software exceptions (e.g., divide by zero, file not found), rejected commands, measured  performance values outside of commanded or expected values, an incorrect step, process, or data definition in a computer program etc. Faults are preliminary indications that a failure may have occurred or will occur. (NSS 1740.13,.SSP 50038), (MSFC) + (IEEE Std 610.12)
Failure - Example- Consider the position of the robot arm that is transmitted to a processor controlling the movement of the arm.  If that transmission is  garbled the controlling processor may not have the correct position of the arm.  This would be a fault.  If the controlling processor repositions the arm based on faulty (not the latest) data the arm may be positioned next, into the side of the shuttle.  This would be a failure.   
A fault may or may not lead to a failure.
Instructor’s note:  It may be beneficial for the participants to discuss how these terms relate to each other, so that there is a clear understanding of the terms. 

## Definitions: Safety vs Reliability

- Now the reliability folks are able to add the caveat that the product/system be used within specified limits and over a specified time period, and often  consider only one fault’s impact at a time.
***Usually not so with safety.***
- Turned off, turned on, improperly operated, secondary or simultaneous errors/faults
- - products are expected to be relatively safe.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Two points to make right up front, first, safety and reliability are different but can work to help each other.

## Definitions: Safe vs. Reliable

***Definitions: Safe vs. Reliable***

- A system is:
**safe** if it doesn’t kill anyone, or the system itself, while either performing its normal operations or, when unable to perform correctly, “fails-safe” .
**reliable **if it performs the required functions within specified parameters/environment and within predicted working timeframe consistently
***Some consider Software to be ***_***very***_*** reliable, it does just what its programmed to do, over and over and over again.  It doesn’t wear out or ‘break’.  ***
***However, anyone with Windows ******tm****** may have a different perspective on Software reliability!***

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Now is probably a good time to straighten out this confusion.
While a reliable system is often safer, it is not necessarily so.  If the system can be counted on to always perform a certain way - that way can be made safe.
Example: A relay contact is usually used to signal elevators to return to the bottom floor of a building and open the doors in the event of a fire alarm.  This is reliable, but is it safe?  Yes, unless the fire is on the ground level - then it is lethal!
A safe system is not always reliable.  But there is a better chance of it being safe if there does not have to be safeguards for numerous unpredictable possibilities.
Example:  A pipe works within specified parameters (i.e.pressures) if regulator allows pressure to go outside pipes range, then the pipe becomes a hazard which may rip open, explode, whatever.  The regulator may, or may not, be reliable.  If regulator doesn’t properly regulate pressure according to specs, it is unreliable.  Pipe is only unreliable if it breaks even when proper pressure is within.
For software, while anyone working with Microsoft Windows will deny software’s reliability, is it really the software or something else?

Software is very reliable. It just does what its told to do!  It won’t break!
Note: - There are, of course, arguments that it is not possible to fail safe and that software, though reliable, does not always perform its function.  These are topics that can be debated (even in class!).

## What Makes Software Hazardous?

**Software is hazardous if it:**
**Controls** hazardous or safety critical hardware
**Monitors** safety critical hardware as part of a hazard control
- Provides **critical data** upon which a safety-related decision is made
- Performs analysis that impacts automatic or manual hazardous operations
**Verifies** hardware hazard controls
- Is used to verify safety critical hardware and/or software
- Is used to model or simulate safety critical applications

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
How does software contribute to the hazardous world we know and are trying to build

## Generic Software Hazard Cause Examples

- Failure to detect a hardware problem - **MONITORING**
- Failure to perform a function - **CONTROL**
- Performing a function at the wrong time - **CONTROL**
- Executing a function out of sequence
- Performing a function when the program is in the wrong state
- Executing the wrong function - **CONTROL**
- Performing a function incompletely - **CONTROL**
- Failure to “pass along” information or messages – **CONTROL/ MONITORING**
- Passes misleading or old information – **MONITORING**
- Provide information too soon or too late
- Simulation incorrectly models hardware reactions - **VERIFY**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Software can be the cause of a hazard, if it controls hazardous hardware and the software fails, or fails to perform as needed or expected.  

## Hazard Examples

- Hazard:	Unexpected movement of a robotic arm 		during manual checkout.
- Cause:	Operator error.
- Control 1:	Software disables Robotic Arm 				movement during 	maintenance periods 		or when detect ‘foreign” object in path.
- Control 2:	Warning screen and light when in 			Maintenance mode
- Control 3:	Put up physical barrier to prevent entry

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
(Commanded wrong mode and state).  A third control for the second hazard might be that the software can detect when certain conditions leading to gas exposure are present are detected, it will over ride the command to open the valve and provide a warning of a requested critical state.

## Hazard Examples

- Hazard:	Toxic gas released inadvertently from 			sealed container.
- Cause:	Software commanded the valve open 			during experiment set up
- Control 1:	Independent software process monitors 		modes and commands to valve.  Rejects 		valve-open command in specific modes.
- 			Independent commanding.
- Control 2:	Hardware Valve enable/disable

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Software Controls

| Hazard Cause | Software Control Location |
| --- | --- |
| Physical/hardware problem, outside of computer processing arena (e.g. stuck valve or actuator) | With “regular” in-line software can monitor, alert, and possibly provide work around or safing. Partitioning from other non-critical code is recommended. |
| Software Error | Separate monitoring task.  Partitioning required or second processor. |
| CPU failure | On separate, independent processor. |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Generic Hazards

| Contamination / Corrosion  | Electrical Discharge 
/Shock  | Environmental / Weather
 | Fire / Explosion  | Impact / Collision  | Loss of Habitable Environment | Pathological / Physiological/ Psychological  | Radiation
 | Temperature Extremes  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Chemical Disassociation

Chemical Replacement / Combination

Moisture

Oxidation

Organic (Fungus, Bacterial, Etc.)

Particulate

Inorganic (Includes Asbestos)  | External Shock

Internal Shock

Static Discharge

Corona

Short  | Fog

Lightning

Precipitation (Fog, Rain, Snow, Sleet, Hail)

Sand / Dust

Vacuum

Wind

Temperature Extremes  | Chemical Change (Exothermic, Endothermic)

Fuel & Oxidizer in Presence of Pressure and Ignition Source

Pressure Release / Implosion

High Heat Source  | Acceleration (Including Gravity)

Detached Equipment

Mechanical Shock / Vibration / Acoustical

Meteoroids / Meteorites

Moving / Rotating Equipment  | Contamination

High Pressure

Low Oxygen Content

Low Pressure

Toxicity

Low Temperature

High Temperature  | Acceleration / Shock / Impact / Vibration
Atmospheric Pressure (High, Low, Rapid Change)
Humidity
Illness 
Noise
Sharp Edges
Sleep, Lack of
Visibility (Glare, Surface Fogging)
Temperature
Workload, Excessive  | EMI

Ionizing Radiation (Includes Radon)

Non-ionizing Radiation (Lasers, Etc.)
 | High

Low

Variations  |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Possible Software Generic Hazard Controls

| Contamination / Corrosion  | Electrical Discharge / Shock  | Environmental / Weather
 | Fire / Explosion  | Impact / Collision  | Loss of Habitable Environment | Pathological / Physiological/ Psychological  | Radiation
 | Temperature Extremes  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Receive data input from hardware sensors (gas chromatograph, particle detector, etc.).  Activate caution and warning indicators if levels surpass programmed limits, and/or automatically shutdown sources or activate fans.  | Shut down power prior to access of electrical components.  | Receive data input from sensor readings of hardware devices (particle detector, wind velocity probe, etc.).  Send commands to shut down hardware if programmed limits are surpassed.
 | Monitor temperature, activate fire suppression system if temperature goes over set threshold.  | Monitor position of rotating equipment. Keep position within defined limits, or shutdown motion if exceeding limits.  | Receive data input from sensor readings of hardware devices.  Send commands to operate proper sequencing of valve operation.
 | Monitor pressure and rate of change.  Control pressure system to keep rate of change under set limit.  | Receive data input from sensor readings of hardware devices.  Shut down high gain antenna when operational time limit is reached.
 | Monitor temperature.  Sound warning if temperature outside of limits  |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Software Safety - Why care?

- One or more people are injured or worse
- Regulatory requirements
- NASA requirements
- Liability if software fails
- Reputation (business or personal)
- “Good practice” for mission critical or business critical software

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Now days we need to worry about more than the obvious “safety” issue of injury and potentially death.  In this make it  or break it business world we must be aware of any regulatory requirements, perhaps your company’s good reputation can make your system “critical”.  There is always the chance of getting sued these days even if the product doesn’t injure some one – failure to perform as expected is enough cause in some arenas.  So what we will be talking about today, though we stress safety critical, can be used as good practice and good approaches for mission/business critical systems as well. 

- A second look

**Safety Critical Software**

## Overview

- A Little Review
- Just what is software really?
- About Safety Critical Software
- Other ways to look at/ define
- Categorizing  or quantifying Criticality

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
In our first section we talked about what makes software safety critical, this short segment is to expand on that theme

## What Makes Software, Safety Critical?

***What Makes Software, ******Safety****** Critical****?*

- software that directly or indirectly contributes to the occurrence of a hazardous system state

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
So how does one contribute to the occurrence of a hazardous state anyway?
First the software must be connected somehow to a critical function.  That is, the software must first be capable of affecting a safety critical function, a function that “must work” or  “must not work”.  To prove this is or is not the case may take some analysis (we’ll get to that later).
Only the ability to affect a function in a manner that causes a hazardous system state is necessary to classify the software as safety critical.

## Safety critical software is:

_***Safety critical software is:***_

- Software which controls or monitors safety critical functions including mitigation of hazards
- Software which runs on the same system as safety critical software or impacts systems which run safety critical software
- Software which handles safety critical data
- Software used to verify and validate safety critical hardware and software

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Therefore, determining if the software is safety critical won’t take a rocket scientist.
These four qualifiers and definition mentioned earlier are usually enough to make a determination of criticality.  But the second qualifier can be confusing.  
What it means is if there is software resident on the system that also contains previously defined “safety critical software”, it can be thought of as safety critical as well.  This is because, being on the same system, it may be possible that a fault of non-critical software is passed on to a portion of safety critical software.  One approach to prevent the propagation of software faults is to establish Fault Containment Regions (FCRs).  Techniques known as “firewalling” or partitioning should be used to provide sufficient isolation of FCRs.   There still will be debate, of course. 
There are many who will say that “there is no safety critical software in this vehicle”, “payload”, “experiment”, or “product”.  Some will go to great lengths to remove the title of “safety critical software” from their systems.  For some systems it may be necessary to remove the software from the system, or have a minimum of three mechanical inhibits to make the software superfluous.  So then, what is the purpose of the software?
In the past, hardware was always the first and last line of defense.  Software just hasn’t been trustworthy.  But if properly developed and analyzed, software can perform up to its new obligations in the world of faster, better, cheaper and still be SAFE.

## What Software is included

- Software
- Firmware
- Programmable Logic Devices
- ASICs - Application Specific Integrated Circuits
- FPGAs  - Field Programmable Gate Arrays

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Software is all these things though some try to define some of them as hardware.  In deed at times the lines do seem to blend in the case of ASICs and FPGAs
Still when you look at  what goes into “programming” these devices, it can be seen that while they end up as basic hardware devices, their configuration is determined by the same means as if creating a program to run on a CPU.
Programmable Logic has come a long way to and while in some ways still resembles the hardware switches and relays it was created to replace, it is still programmable and thus must be analyzed, developed and tested as software. And when operating large equipment or safety critical ones, they need to be assessed for safety criticality.

## Categories of S/W Safety Functions

**Must work/must not work** functions
- Mode and State Dependent
- Must never work
- Must never fail to work
- Fault tolerance
- Redundancy
- How many levels and where are they best put
- Multiple Commanding (Ready, Arm, Fire)
- Independent Commanding

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Thinking about software in these categories often helps define the problem so as to better work it.  While these may not be the only categories of safety critical software, they do cover a wide range of the operational software we need to consider.  When we think of software as performing these functions or when they are thus described within their specification or requirements document, it is much easier to address their consequent analyses and operations.

## Categories of S/W Safety Functions

- Caution And  Warning (C&W) Functions
- Failure Detection, Isolation, and R---------
- Recovery
- Restart
- Reduced operation
- Automatic safing software
- Hot, Warm, Cold Backup
- Autonomous Decision Making & Operation

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
More will be added to these categories and they can be expanded upon.  In the next few slides we will give some examples to further your thinking along these lines.  But due to time, much will need to be left to the paper or better yet, to your own discovery and decisions.

## Software in Hazard Mitigation

- Software that is used to **mitigate (reduce)** a potential hazard is safety critical.  For example:
- Logic designed in for redundancy of Hardware or Software performing a safety critical function
- Software that detects fault trends and responds
- Automatic response to failure/automatic safing
- Software used for Caution and Warning

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## FDIR

- Fault or Failure
- Detection/ Determination
- Isolation or Identification
- Recovery
- or Response
- or Report
- or Reduction of functionality - safing
- or Restart

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
What are we looking for?A fault, a combination of faults, a certain number of the same fault? or do we wait for a failure and respond to that.
Can or should this be over ridden?  If so by what, a human, another software program, hardware?/  The system design needs to lay this out.
what do we really need to find in order to react appropriately?  These are questions that need to be asked during requirements and preliminary design.
Once It is decided what must be detected, then how becomes the issue. Are there the proper number and or type of sensors, what algorithms are needed what needs to be monitored, how invasive is that monitoring, will it reduce the needed functionality,...
These issues are worked out at both the system and Software design levels
The “I” usually stands for Isolation of the fault or failure to a specific location, device, subroutine, etc. Some say this is a function of Identification of the fault or failure. Either way, where is the best place to locate the problem and what can you use to do so.  Has your safety critical software been shielded within a kernal, isolated so to speak?  What of software specifically set to watch hardware, those are easy, but it can get tangled.
Once the problem is found, and you know what it is, then what? Depending on the situation, the software and the system it operates must do something else why bother building in the D and I parts!?!

## Must Work/Not Work Functions

**Must Work Functions**
- Open valve upon command
- Monitor position and keep within specified limits
- Read temperature sensor attached to furnace
- Switch operational modes at specific times
- Move filter wheel prior to new image exposure

**Must Not Work Functions**
- Scan Platform movement during maintenance
- Gas vent valve open during experiment
- Power to connector when panel open
- High voltage energized when not in vacuum
- Maintenance light on during operations

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Multiple Commanding

- Ready, Arm, Fire
- With Checking and Acknowledgements
- Independent commanding
- Different sources must give the command to proceed within a specified time
- Operator Verification Dialog
- Are you sure you want to delete/command/... that

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
This is important when it is necessary to command a hazardous operation  but you want to assure in some way that all precautions have been taken.
Ready, Arm, Fire  provides two preparatory steps prior to actually commanding the hazardous operation
Independent means either from 2 different CPUs’ software, an Operator and a software command - via two routes
And then we are all familiar with the “do you really want to delete ...”
There are other variations on this.  You think of a few of your own.  Where do you need this kind of hazard mitigation?

## Hazard Severity Definitions

| Catastrophic | Critical |
| --- | --- |
| Loss of entire system; loss of ground facility; loss of human life or permanent disability | Major system damage; severe injury or temporary disability |
| Moderate | Negligible |
| Minor system damage; minor injury | System stress but no damage; no, or very minor injury |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Now that we have given you some further ideas of safety critical software, now we come to trying to define just how critical are they.
This shows you how NASA defines the level or severity of a hazard.  You need to define this for your own company or even from product to product. 

## Likelihood of OccurrenceEXAMPLE!!!

| Likely | Greater than a 1 in 10 chance of occurring. |
| --- | --- |
| Probable | Between a 1 in 10 and a 1 in a 100 chance of occurring. |
| Possible | 1/100 to 1/1000 chance. |
| Unlikely | Less than a 1/1000 chance. |
| Improbable | Buy a lottery ticket if it occurs! |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Then we look at the likelihood of it’s occurrence. And Again, this is something that your company determines - this is just an example even NASA has a hard time with this.  At best it is “Best Engineering Judgment” that is used.  And that is where we currently stand.  Of course, this can lead to lots of arguments and it general does in some cases.  But we need to look at worst case possibilities and design for that - but with common sense.

Right, we’re very helpful here aren’t we?

## Hazard Risk Index

| Severity Levels | Likelihood of Occurrence |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Likely | Probable | Possible | Unlikely | Improbable |
| Catastrophic | 1 | 1 | 2 | 3 | 4 |
| Critical | 1 | 2 | 3 | 4 | 5 |
| Moderate | 2 | 3 | 4 | 5 | 6 |
| Negligible | 3 | 4 | 5 | 6 | 7 |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Putting the two together  we try to map out a system hazard risk index. We try to do this very early on because we eliminate anything that ends up with a #1 rating.  WE must find a work around or a way to reduce either the likelihood or severity before proceeding.  

## Level of Safety Effort

| System Risk Level | Safety Effort |
| --- | --- |
| 1 | Prohibited |
| 2 | Full |
| 3 | Moderate |
| 4,5 | Minimal |
| 6,7 | None (Optional) |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Then we map the required development and analyses processes required for the effort based on the admittedly imperfect “best guess”
Some say they can get it down to a finer number and have all sorts of reliability 
predictions, but when all is said and done, probability is still a “best guess”.

## Creating Safer Software & Safer Systems

***Creating Safer Software & Safer Systems***

- Good SW Development Process
- Development Tools
- Appropriate Reviews
- Diverse Review Teams
- Formal Inspections
- Communication
- Appropriate Analysis, both Safety & Development
- Caveat Emptor

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Once the level of effort is defined via a method like in the last few slides, for software these items are where we determine our level of effort.
Thus far the discussion has been centered on what can be done to create safe systems from the standpoint of identifying, then eliminating or mitigating hazards.  Methods of developing safer software, and not introducing hazards, includes establishing a good software development process.  There are many procedures and methodologies for developing software.
Good SW Development Process - Help Prevent Introduction of New Hazards
Development Tools - To Uncover Errors.
Reviews - To Uncover Errors.
Software Inspections - To Uncover Errors.
Communication - To better understand and create the right, safe system.
Appropriate Analysis - To discover potential hazards and design approaches.
Buyer beware - for COTS and also contracted software.

## Don’t Panic

- In the Tailoring section, we will look at how to break down your approach
- Not all your software need be created at the same level of process and analyses - if you are careful how you design it.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

- Getting them right.
- Getting them safe.

**Requirements**

## Establishing Software Requirements

- System requirements are complete and correct.
- System requirements allocated to software.
- Identified safety standards.
- Known environment.
- System constraints well understood.
- Interfaces with hardware and users defined.
- Critical performance requirements identified.

- No or minimal system requirements.
- Verbal requirements and changes.
- Unsure what, if any, safety issues.
- Hardware in flux, so constraints not understood, interface not defined.
- Only performance requirement is “it should do this...”.
- User interface left to the programmer.

_**Ideal**_

_**Worst Case**_

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The ideal world rarely exists, but the “worst case” happens far too often!  At least, having documented requirements and defined/understood hardware is unfortunately not too common.  The desire to “jump right in” means that many requirements are discovered later in the development cycle.  They are not “designed in”, but “backfilled”, once the requirements are understood.  This adds cost and increases the chance that safety issues will not be handled well.  Safety is often not thought of when frantically backfilling.

Rapid prototyping is one lifecycle that helps get the requirements “more right” early on.  Especially user interface and performance issues.  However, when safety is a concern, the rapid prototype code should NOT be used.  Recreate the functionality in code developed with a more well defined process.  

## Dealing with the “Worst Case”

***Write down what you know*****!**** **
-  Everything your boss, the systems team, or the engineers told you is a good place to start.
***Document your assumptions*****.**
- What the software will **not** be able to do, what engineering units will be used...**Write them all down!**
***Research the hardware*****.**
- Understand the limitations of the hardware.
***Develop a “user’s perspective”.**** **(Better yet, get their help.)*
***Take the next step...think about safety.***
***Get “sign off” or approval on your requirements from all parties.***

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Write what you know:  Also keep appropriate emails.  Take notes during meetings.  Put your understanding of what they want down on paper.

Assumptions: Many things are “obvious” to the software developer.  They may not be obvious to the user or the hardware developer.  Put everything you can think of on paper! Think about the order functions can be called in (and sequences that they should not be called in). Include assumptions about what the hardware will do in response to software commands or conditions.  For example, will a software reset help when there is an error, or could it leave the hardware in an unsafe state?

Hardware:  This is a good time to talk with the hardware folks, if you’re not the hardware side as well.  They made assumptions, too, and not all of them may be obvious.

Users: The customer/user is making assumptions too.  If you can, talk with them. If not, try to understand who the users are and what may be important to them.

Safety: How can your device/system mess up.  How can the software help it mess up, and how can it prevent the system from screwing up. When is it NOT okay to reboot, what state information must be propogated across a reboot, etc.

Sign off: Don’t just show the requirements to your manager.  Make sure hardware, operations, the customer, anyone else involved, agrees.  Or at least doesn’t have too much heartburn with them.

YOU DON’T HAVE TO CREATE A FORMAL, VERBOSE, STUFFY DOCUMENT! It’s far more important for it to be readable by all parties.  Keep it simple but complete.  It is a good idea to include the project name, who created the document, date and revision.  Putting it under configuration management is a good idea.  

DON’T PUT THE REQUIREMENTS ON THE SHELF AND NEVER LOOK AT THEM AGAIN.  Revisit them on occasion to see if any requirements have been added, removed, or modified.  Look at them when the design is done and while you are coding.  Does the software you are creating match the requirements?

## Changing Requirements

- Document the changes
- Keep the requirements document up to date
- Fight to keep interfaces unchanged (w/in reason)
- Estimate the “***full ***cost” of changes
- Keep management informed of the “risks”
- Re-evaluate the software safety!

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Even if requirements are documented early, there is no guarantee that they will not change.  Even projects with well thought out,  inspected, and dissected requirements will sometimes find “missing” requirements later in the development.  Those projects usually have a documented change process as well. All disciplines, including safety, usually agree to the changes.

More common is random or constant change. This can be due to a fluctuating market, the desire to use new or “cutting edge” technology, customers who cannot make up their minds, among other reasons.  The requirements are a moving target.  What can you do in this situation?

Document:  Don’t just act on a verbal from your boss.  At least write it down first!  Run it by the person requesting the change, to make sure you are on the same wavelength.  This helps focus you on what really needs to be modified, and makes sure your boss hasn’t changed his mind.  
[Kal story - The scientist I worked for would often change his experiment scenarios (how the system worked to take the required data).  I would report on my progress in modifying the software at the next staff meeting, only to find that he had rethought the idea and decided it wasn’t what he wanted. My system was to write down what he wanted and show it to him.  I had to get agreement at least three times before I would start to alter the code.]

Update the requirements:  If you got hit by a car tomorrow, could someone find all the little scraps of notes you have?  Keeping the requirements document up-to-date lets anyone involved in the project understand what is being developed, at any given time.  And since the document is under configuration management :-) you have a record of what was changed, and when it was changed.  This can be helpful when someone in management asks why the project isn’t completed yet!

Interfaces: Interfaces are the parts that connect your software to the hardware, the users, and other software elements.  They are an agreement.  When one side changes them, it affects the other aspect of the system.  These kinds of changes tend to propagate through the system, leading to unanticipated modifications of other components. It is best to leave these alone!  If you suspect the interfaces may change, encapsulate them and make sure that no other aspect of your software depends on the format of those interfaces.  

Cost: It’s very easy to say “software can do that”, “sure, we can add that function”, or “no problem, that’s an easy change”.  But what is the cost of that change, really.  Do you have to update or redo any analyses?  What about the time it takes to update the requirements and design documentation?  Does the change invalidate any testing already completed?  And just how long will it really take?

[Kal story: When I was first starting out, with no real embedded experience, it was easy to agree to implement a function in software rather than hardware. Many of those functions took more work (and cost more) to design, implement, and test than if a custom hardware board had been designed.  Experience taught me to calculate the effort involved before agreeing.  My answers became “Yes, we can do that, but...”  “But it will take 3 weeks.  But it will mean we cannot do function X, or complete function Y.  But we may need a more powerful processor to meet the timing demands on the system.”  This information allowed the project to distribute the functions to the most appropriate and cost-effective domain.]

Management:  Let’s be honest - management rarely understands what we really do!  Software is magic.  It’s very easy for management to ask for changes in something they don’t understand fully.  It’s our job to make sure they understand the cost and the risk of making the changes! 

Safety: It’s always important to evaluate the safety aspects of any change,  even if the change affects only non-safety-critical modules.  You have to make sure that the modification cannot affect the safety critical portions of the code.  You also have to make sure that the modification doesn’t add a new hazard (if it works, or if it fails).  And if the change is in a safety critical section, even more care needs to be taken.

## Requirements “Analyses”

- Preliminary Hazard Analysis (PHA)
- Software Requirements Flow-down Analysis
- Requirements Criticality Analysis
- Checklists and Cross References
- Timing, Throughput, Sizing Analysis
- Formal Inspections

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Don’t Panic when you see “Analysis”.  Many analyses are just applying common sense in an organized manner. Many of them are things you already do, though perhaps not in a formal way. You don’t have to be a mathematical genius to perform these analyses. For example, “flow down analysis” is simply a structured method to make sure requirements from a variety of sources are included in the software requirements.  All these analyses are discussed in the following slides.

Preliminary Hazard Analysis (discussed previously) is a way of determining what aspects of the system could cause hazards.  Software Hazard Analysis looks at how software relates to the system hazards, as a cause, control, mitigation, warning, etc.
Flow-Down verifies that all appropriate requirements that software must meet are actually in the software requirements document!  
Requirements Criticality analysis looks at each requirement and determines which are safety or mission critical and require extra attention.
Checklists and Cross References are simple ways to make sure you haven’t forgotten anything.
Timing, throughput and sizing looks at what resources the software will use vs. the hardware capabilities.  
Formal Inspections are a structured review of any software product.  They are especially effective in finding errors when applied to the requirements.

## Where do Software Requirements Come From?

- Software Requirements

- System Requirements

- ISS
- Requirements

- “Company” Requirements

- Customer
- Input

- MIL-
- STD-
- 882

- Safety Standards

- Hardware and
- Environmental
- Constraints

- Hazard
- Analyses

- Generic
- Software
- Safety

- Risk
- Analyses

- Your Experts

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Not from the stork.  Software requirements can come from a variety of sources.

## Software Requirements Flow-down Analysis

**Flow-down Of**
- System Requirements
- Software Specific Requirements
- Safety Specific Requirements
- S/W Safety Specific Requirements
- Regulatory requirements
**Generic Software Safety Requirements**
**Hazard Analyses (PHA, etc.)**
**System & Software Risk Analysis**
**Software Standards & Guidelines**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The purpose of a software requirements flow-down analysis is to make sure that all software requirements are captured, and that they are correct, consistent, unambiguous and verifiable.  It doesn’t do any good to have a lot of vague software requirements that no one really understands.  If the requirement isn’t clear, various people will interpret it differently.  If it is not verifiable, how will you know if it has been met?
Sources of software requirements include:  
	1) System requirements; 2) Safety standards or regulatory requirements; 3) Requirements imposed by the organization – for example, ISS imposes requirements for instruments that will operate in it.; 4) Customer input; 5) Hardware and environmental constraints; 6) Lists of generic software safety requirements; 7) Hazard Analyses; 8) Any risk analyses performed at the system or software level.; 9) Standards and guidelines, e.g. IEEE 12207
Its important to know how to get hold of these requirements and analyses.  This means knowing where to go, who to contact.
Not all “requirements” may be imposed from on high (either the project, organization, or regulatory body).  Consider including additional requirements if they do not pose an undo burden on the project.  You may not be told that you have to follow a particular standard, such as IEEE 12207, but choosing to do so may make your software safer and more reliable.  Not all of the generic software safety requirements may be obligatory, but using them adds to the safety of your system.
 Finally, you need to think about the characteristics of the system, and make sure they are included in the software requirements.  Characteristics to be considered include at a minimum:  1) specific limit ranges; 2) out of sequence event protection requirements (e.g., if-then statements); 3) timing; 4) relationship logic for limits.  Allowable limits for parameters might vary depending on operational mode or mission phase.  (Expected pressure in a tank varies with temperature, for example.); 5) voting logic; 6) hazardous command processing requirements; 7) fault response; 8) fault detection, isolation, and recovery; 9) redundancy management/ switchover logic; what to switch and under what circumstances, should be defined as methods to control hazard causes identified in the hazards analyses.  For example, equipment which has lost control of a safety critical function should be switched to a good spare before the time to criticality has expired.  Hot standby units (as opposed to cold standby) should be provided where a cold start time would exceed time to criticality.

## Generic Software Safety Requirements

***Generic Software Safety Requirements***

- Common Software Safety Problems
- Lessons Learned
- Problems/issues from similar systems
- Generic Checklists
- Other sources
- Tailor them to the specific application

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Software safety requirements can be derived from sets of requirements and best practices used in different programs and environments to solve common software safety problems.  Similar processors/ platforms and/or software can suffer from similar or identical design problems.  Generic software safety requirements capture these lessons learned and provide a valuable resource for developers.

Generic requirements prevent costly duplication of effort by taking advantage of existing proven techniques and lessons learned rather than reinventing techniques or repeating mistakes.   Most development programs should be able to make use of some generic requirement; however, they should be used with care.  There are many sources of generic software safety requirements: UL, NSTS 19943 Command Requirements And Guidelines for NSTS Customers, MSFC, the Food & Drug Administration, etc..
Even if you have a generic list, you should use that list to create your own for your specific domain.  Think of them as generic shopping lists where you choose what is needed for a particular project.

## Requirements Criticality Analysis

***Requirements Criticality Analysis***

- Identifies Requirements with safety implications
- Identifies missing requirements
- Identifies areas of S/W hazards/faults
- Updated throughout development
- Must Work/Must Not Work Functions

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Within a collection of requirements, there will be varying levels of criticality.  Most requirements will be non-critical.  This analysis determines which requirements you really need to pay attention to.
The Requirements Criticality Analysis procedure is given in the paper.  The basic idea is to review all relevant requirements (software and system) and look for those that can cause, mitigate, or contribute to hazards. Also look for requirements not flowed from the system requirements or other sources.  

## Checklists and Cross References

- Checklist - make sure you don’t forget anything important
- Required hazard controls and their corresponding safety requirements
- Software safety requirements (output)
- Generic safety requirements
- Cross Reference - matrices that list related items
- Hazard requirements flowdown matrix
- Requirements/verification matrix
- Aids in Flow-down and Criticality Analyses

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Checklists are a way to put the collective experience of those who created and reviewed the checklist to work on your project.  They are a starting point, and should be reviewed for relevance for each project.

Cross references help track things so they are not lost.  Making sure that requirements, especially hazardous ones, flow down through all levels and design phases is important.  So is making sure that all requirements are verified. 

## Timing, Throughput, Sizing Analysis

*Timing* - how long does it take to accomplish a task
*Throughput* - how much data, etc. will the software “handle”. Includes I/O and telemetry.
*Sizing* - how big will the software be (memory, PROM, flash, etc.).

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
You want to know how big, how fast, and how much data your program can handle. This analysis focuses on program constraints.  Typical constraint requirements are maximum execution time, minimum data throughput, and maximum memory usage.  This analysis verifies if adequate resources (and margin) exist under worst-case scenarios. 
Assessing memory usage can be based on previous experience of software development, or you can attempt detailed estimates based on the size of the program, stack and heap sizes,  and other factors.  Remember to update these estimates when you have some hard facts.  Early on, the numbers will be unrealistic. 
Address I/O for data collection, housekeeping, and control.  Evaluate resource conflicts between primary data collection and safety critical data availability. 
Investigate time variations of CPU load, determine circumstances of peak load and whether it is acceptable.  Consider multi-tasking effects.   Note that excessive multi-tasking can result in system instability leading to “crashes”.

## Formal Inspections

- A very structured process designed to uncover and remove defects as early as possible in the development process.
- Well-defined inspection steps and rules
- Well-defined roles for participants, but working as a team.
- Checklists help identify defects or problem areas.
- Formal collection of process and product data (metrics)
- The product being inspected must meet certain input criteria and output criteria
- A supporting infrastructure is necessary

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Formal inspections were developed by Michael Fagan at IBM to improve software quality and increase programmer productivity. They are not formal milestone reviews, like PDR/CDR.  Formal inspections can be used on many software products - not just code!  In fact, they are very useful when performed on the requirements document!
One of the keys to success is to have people from various disciplines, including hardware, systems, safety, and users.  Different perspectives spot different defects.  And if a requirement, design, or piece of code is ambiguous, then it is probably a defect!
Keep the focus on the defects - not design decisions or style. 
Inspections are time and resource consuming.  Choose wisely what to inspect, and when.

## Software Safety Analyses

- A practical exercise in software safety

## Two Types of Analyses

- _FMEA (Failure Modes and Effects Analysis)_ analyzes each component for possible failures, then determines the effects that each kind of failure would  have if it were to occur during operational use.
- _FTA (Fault Tree Analysis)_ starts with an undesirable event, such as a failure or mishap, accident or malfunction and then determines all the ways it can happen.
***Bottom Up = FMEA***
***Top Down = FTA***

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Two Types of Analyses: FMEA

- FMEA
- Failure Modes and Effects Analysis

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Failure Modes & Effects Analysis Procedure

*Failure Modes & Effects Analysis Procedure*

_**Identify:**_
- 1. Project/system/software components
- 2. Set Ground Rules and Assumptions
- 3. Potential Functional & Interface Failure Modes
- 4. Potential consequences of each failure mode
- 5. Failure/fault detection methods and compensating provisions
- 6. Corrective design or actions to eliminate or mitigate failure/fault
- 7. Impacts of corrective changes

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Methods used to perform a FMEA (Failure Modes and Effects Analysis) can be used to perform a FMEA on S/W by substituting software components, or at least adding S/W functions to the system FMEA, but this usually confuses the reliability & systems folks for hardware components.  The steps on this and the next chart are based on the widely used FMEA procedure,  MIL-STD-1629.
Note to students:  Each step is shown in order and then sub-labeled and further explained later.
Instructor’s Note: The steps are listed here and then each step is explained and then used in an example FMEA after all process steps are defined.  For those that are not familiar with FMEA, seeing the information repetitively will reinforce the concepts of how to perform the analysis.

## FMEA: Possible Areas for Faults

*FMEA: Possible Areas for Faults*

- Data Sampling Rates
- Data Collisions
- Command Failure to Occur
- Command Out of Sequence
- Timing

- Illegitimate Command
- Safe Modes
- Multiple Events or Data
- Communications overloaded by task
- The Improbable

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
These are all sorts of things that software can do to cause system or subsystem failures.  Not every S/W fault will lead to a system failure or shutdown, and even those failures that occur may not be safety critical.  There are lots more types of faults than these, but these are good start when looking for things that can go wrong.  The guidebook lists these as well.  Some of these are obvious 
Data Collisions - there are different types of data collision
in communications for instance on a LAN, transmission by many processors at the same time
modification of a record when it shouldn't be because of similarities
modification of data in a table by multiple users in an unorganized manner
Command Failure to Occur - The command was not issued
Command out of sequence - there may be an order to the way equipment is commanded on.  For instance, it is wise to open dampers to the duct work going to the floors, as well as the dampers to bring in outside air before turning on the air handling units of  a high rise office building.
Timing - Dampers take a long time to open (especially the big ones) so,  timing is critical.  A time delay would be necessary keep from imploding (sucking in) the outside air dampers or possibly exploding the supply air dampers, by turning on the air hander prematurely.
Safe Modes -  Its sometimes necessary to put a system which may or may not have software in a mode in where everything is safe (ie. Nothing melts down or blows up) or in the case of software the software maintains itself and other systems in a hazard free mode.
Multiple Events or Data  - What happens when you get the data for the same element twice, within a short period of time?  Do you use the first or second value?

## FMEA: Possible Failure Modes

*FMEA: Possible Failure Modes*

- Processing
- Operations
- Communication
- Data
- Hardware
- Sensors
- Effectors

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Identification of the possible failure modes and effects in an Events Table and Data Table
Examples:
Hardware Failures/Design Flaws   - Broken sensors lead S/W down wrong path   - no sensors or not enough sensors - don’t know what H/W is doing
   - stuck valves or other actuators
 Software  - Memory over written (insufficient buffer or processing times).
  - Missing input parameters, incorrect command, incorrect outputs, out of range values, etc.
  - Unexpected path taken under previously unthought of conditions.
Operator
  - Accidental input of unknown command,  or proper command at wrong time.
Environment   - Gamma Radiation
   - EMI
   - Cat hair in hard drive
   - Power fluctuates

## FMEA: Start At The Bottom

*FMEA: Start At The Bottom*

- Identify a component (method, class, object, data)
- Identify its failure modes
- Determine the effect of each failure mode on other components.
- Determine the effect of each failure mode on the System/Subsystem/Computer System Configuration Item (CSCI)

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Go back to the block diagrams, starting at the lowest level, look at a component and determine the effect of that component failing in one of its failure modes on the components in the level above it.
There may a need to consider the effect of this component and all the effected components at the next level of components.  This must be worked all of the way up the chain.  This is a long process and the class will work through an example to bring points home.
Remember though, if the safety critical portions are fairly isolated in a system, then, when working at the detailed design and implementation phases/versions of this analysis, the analyst will be looking at only those parts of the system that can lead to a critical failure.

Instructor’s Note:  Show complex FMEA. Show ripple of component through system (or how it doesn’t effect overall system functions).

## FMEA: Consequences of each Failure

*FMEA: Consequences of each Failure*

- Severity Level
- Likelihood of Occurrence
- Risk Level

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Next:  What was the effect of that failure/fault.
What is the criticality of the failure/fault.
This is the “effects” part of FMEA. 
All of this has been presented  in the safety perspective before in the overviews but, a look at the reliability perspective may be in order. Like safety , reliability, looks at:
Severity may be catastrophic, critical, marginal, or negligible.
Likelihood of occurrence may be probable occasional, remote or improbable.
Risk Level is 1 through 5 with 1 being prohibitive(ie. not allowed- must make requirements or design change)
The critically categories as identified by reliability, include the added information of whether the component or function has redundancy or would be a single point of failure.
For each project and center there may be some variation in  the ranking of severity level and risk level.  This is, after all, not an exact science so much as a professional best guess.  (Best engineering judgment)

## Software Elements

- Data
- Events/Functions
- I/O
- Commands
- State
- Mode
- Others ?

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Before addressing the worksheet, the components of software necessary for analysis must be considered, since some of these show up as elements on the worksheet.
Data is “factual information (as measurements or statistics) used as a basis for reasoning, discussion, or calculation”  This is the major element of software which is  manipulated (processed) and passed around throughout the operation of the software. 
Events are “noteworthy happenings” (e.g. a command, change of state, interrupt, etc.) which cause the software to perform some action (e.g. read a memory location, move a robot arm, change a status bit, etc.).
Commands are instructions to do some thing, a directive from the user or from a software to perform an action.
I/O - Communication between a computer and the user or the outside world.. 
State - > How something is; its configuration, attributes, condition, or information content.
The state of a system is usually temporary (i.e. it changes with time) and volatile (i.e. it will be lost or reset to some initial state if the system is switched off).  A state may be considered to be a point in some space of all possible states. A simple example is a light, which is either on or off. A complex example is the electrical activation in a human brain while solving a problem.  In computing and related fields, states, as in the light example, are often modeled as being discrete (rather than continuous) and the transition from one state to another is considered to be instantaneous. Another (related) property of a system is the number of possible states it may exhibit. This may be finite or infinite. A common model for a system with a finite number of discrete states is a finite state machine. Finite Statae Machines may be talked about later.
Mode - A general state, usually used with an adjective describing the state. Use of the word "mode" rather than "state" implies that the state is extended over time, and probably also that some activity characteristic of that state is being carried out. More technically, a mode is a special state that certain user interfaces must pass into in order to perform certain functions. 
Finite State Machine - An abstract machine consisting of a set of states (including the initial state), a set of input events, a set of output events and a state transition function. The function takes the current state and an input event and returns the new set of output events and the next state. Some states may be designated as "terminal states". The state machine can also be viewed as a function which maps an ordered sequence of input events into a corresponding sequence of (sets of) output events. 
Others - Maybe the class has some ideas

## System Operating Modes

- Mode: A distinct operating state with its own unique functions and specifications.
- Functions Allowed and Disallowed
- Allowed - Some Must Work Functions & Others
- Disallowed - Must Not Work Functions

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Walk into the office in the morning and turn on the computer.  The computer is in the off state with no mode.  When the power is turned on, the computer goes through some initialization routines before the user is allowed access.  During this time the computer can be said to be in the initialization mode, and in the first of many on states called initialization (on initialization).  After the computer has gone through each of the initialization routines the computer is ready for the user to access it and begin work. The computer is in the ready state and the ready mode.  At this point the computer could accept  input from the computer keyboard or mouse.   If the user selects an application such as JINNI, using the keyboard or mouse, the computer will begin to load the application software, and disallow any further input from the computer keyboard or mouse until the application software has been loaded. During this Application Loading Mode the computer would pass from the accept keyboard entry state to the disallow keyboard entry state to the accept keyboard entry state.  There are other states which which components of the computer would  pass through during the Application Loading Mode.  
After the user has been using the JINNI application software for some time it often looses track of the data that was entered into it.  Although the JINNI software is in the user friendly Data Entry Mode it seems to be slipping into an undocumented and unforeseen state called the Drop Data State periodically.  This is just one of many different scenarios possible for operation of a simple PC.  
Whether the terms “state” or “mode” are used as we have in this example, or other definitions of state and mode are used, it is important to examine not only the states and modes in which the software operates correctly.  It also important postulate the existence of states which are not pleasant and determine if these states are possible.  
Think about Safe Modes and Safe States.  What exactly are there meaning in a system.

## Two Types of Analyses: FTA

- FTA
- (Fault Tree Analysis)

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## FTA Procedure

*FTA Procedure*

- Identification of the top event(s) to be analyzed.
- Identify the events or series of events that directly contribute to the top level event.
- Continue this process until the lowest level defined or basic level is reached.
- Go to next top event

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Identification of the top event(s) to be analyzed.
	This top event in a fault tree is a hazardous condition or state of the 	system where a hazard  is defined as a state or set of conditions of a	system (or an object) that , together with other conditions in the	environment of the system will lead to an accident.
	Identify the events or series of events that directly contribute to the	top level event.
	List all of the events that are required in combination or individually	which could result in the top-level failure.
Continue this process until the lowest level defined or basic level is reached.
	The level of fidelity to which the fault tree is taken is based on sound 	engineering judgement, the knowledge of the analyst, and the level of	design fidelity.
	Guideline may:
	1) Think of some lower level events 1st then group.
	2) Work so far on one then work another & come back. 

## FTA Requirements

*FTA** **Requirements*

- Thorough knowledge of how the system works.
- Knowledge of the logic relationships in the system (interlocks, control interfaces, power supply feeds).
- Thorough knowledge of how the software works (evaluated separately) for the SFTA.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Based on field experience with specific products, warranty information, expert knowledge, lessons learned, failures on predecessor products etc.., ---detailed checklists are developed to provide the designer with information on potential hazards, the injuries that can result and the specific types of design actions that can be taken to minimize the risk.  If a new product, or technology, expert “guessing” is also important.
The point of analysis is to see if, and how easily (or difficult) it will be for the top event to occur.
The cause and effect relationship leading to the failure of the system or part of the system under scrutiny are identified. Knowledge of the interfaces, including how software controls the system is necessary once that level of detail is reached.  Software roles must be represented.

## FTA Symbols

*FTA Symbols*

- Intermediate event
- Basic fault/failure
- Undeveloped event
- External/Normal event
- Condition

- And
- Or
- Inhibit

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Intermediate Event: A fault or event caused by a combination of other events via a logic gate.
Basic Failure: A basic fault or event caused by a component or sub-assembly for which a probability can be assigned.
Undeveloped Event: A fault that is not subdivided into basic events due to lack of information or importance. The event must be expanded or developed later.
Basic Event: A basic event is a normal occurrence while the system is operating.
Instructor’s Note: There are numerous variations on the meaning of these symbols dependent on the organization performing the analysis. The standard used here is consistent with the Guidebook (appendix B).

## SFTA and SFMEA Exercise

*SFTA and SFMEA Exercise*

- Perform a SFTA and a SFMEA on the “Unlock Furnace Door” example.
- Try to think of not only hardware failures, but also software failures (memory overwrites, out-of-sequence execution, illegal command, etc.)

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## UNLOCK CHAMBER DOOR BLOCK DIAGRAM

- Unlock
-  Chamber Door

- Shut
- Power Off

- Purify
- Chamber Air

- Pressure
- Sensor

- Temperature
- Sensor

- Exhaust
- Current Air

- Introduce
- New Air

- Ensure Safe
- Temperature

- Temperature
- Sensor

- Temperature
- Sensor

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
In this example the opening of a chamber door is restrained by the following:
Three (3) temperature sensor values must average to plus or minus 5 degrees of a pre-established limit of 25o C. 
The chamber air pressure must match the cabin air pressure
 The chamber air that existed during any experiments must be vented and replaced by cabin quality air
 The power to the internal heating elements must not be capable of being powered on .
 This high level design includes a software clock, software to monitor 3 temperature sensors, 1 pressure sensor, and software which determines that the door is, or is not, safe to open.  It includes quite a bit more but the class will have to discover this information for themselves.

## SFTA Exercise Software Preliminary Design

*SFTA Exercise** Software Preliminary Design*

- Exhaust

- Freshen
- Air

- Temperature
- within
- Limits

- Pressure
- within
- limits

- Shut off
- Power

- Unlock Door

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## SFTA Exercise

*SFTA Exercise*

- Exhaust
- (A)

- Freshen
- Air
- (B)

- Temperature
- within
- Limits

- Pressure
- within
- limits

- Shut off
- Power

- Unlock Door

**A Before B**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Data Table

**Mode 1: On - operating          Mode 2: Off          Mode 3: Transitioning to off**

**Performing a FMEA Relevant to Software**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Instructor’s Note:  Using the diagrams and description for the Furnace door exercise in the appendix, walk through the first couple of FMEA components with the class to get them started.

## Event Table

**Performing a FMEA Relevant to Software**

**Mode 1: On - operating          Mode 2: Off          Mode 3: Transitioning to off**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Instructor’s Note:  Walk through the first event with the class to get them started.

## Data Table

**Performing a FMEA Relevant to Software**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Instructor’s Note:  Have the class continue the exercise.

## Event Table

**Performing a FMEA Relevant to Software**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Instructor’s Note:  Have the class continue the exercise.

## FAULT TREE ANALYSIS  of Crew Exposed to toxic Fumes  _

- Purify Chamber
- Air Software
-  Did Not

- Introduce
- New Air

- Air Not
- Exhausted

**Furnace Door**
**Unlocked Before Toxic Air is**
**Exhausted **

- Pressure Sensor
- Passes Incorrect
- Value

**Crew exposed to Toxic Fumes**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Instructors Note:
Discuss having a top level FTA, then where other FTA investigate conditioning events & undeveloped events and even lowest level intermediate as information is available.
Instructors Note:  Have the class consider the following and develop a fault tree.  This chart will be in the appendix along with a solution.
Develop a FTA to show all possible events that could lead to this event.

## FTA  of Crew Exposed to Hot Gases/surfaces

- Purify Chamber
- Air Software
-  Did Not

- Introduce
- New Air

- Air Not
- Exhausted

**Furnace Door**
**Unlocked Before Safe Temperature reached**

- Pressure Sensor
- Passes Incorrect
- Value

**Crew exposed to hot gases & surfaces**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Instructors Note:
Discuss having a top level FTA, then where other FTA investigate conditioning events & undeveloped events and even lowest level intermediate as information is available.
Instructors Note:  Have the class consider the following and develop a fault tree.  This chart will be in the appendix along with a solution.
Develop a FTA to show all possible events that could lead to this event.

## Safety Is “Job 1”

*** ***

**Design**

## The Design Process

***The Design Process***

- Identification of Design Features and Methods
- Selection of tools
- Allocation of All Software Requirements to the Software Design
- Identification of Safety Critical Computer Software Components (sccscs)
- Design Analyses to assure feasibility and functionality
- Safety Analyses of the  Design
- Test Plans and Procedures

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The design process includes identification of design features and methods (object/class choice, data hiding, functional distribution, etc.)  including safety  features(e.g., inhibits, traps, interlocks and assertions) that will be used throughout the software to implement the software requirements.  
Tools (language, compilers, operating systems, CASE, etc.) need to be selected with safety in mind.
After allocation of the software requirements to the software design, Safety Critical Computer Software Components (SCCSCs) are identified.  Bottom-up safety analysis is performed on the architectural design to identify potential hazards, to define and analyze SCCSCs (Safety Critical Computer Software Components) and the early test plans are reviewed to verify incorporation of safety  related testing.
During the Detailed Design phase, more detailed software artifacts are available, permitting rigorous analyses to be performed.  
Test plans and procedures are created.

## Selecting Tools

- Language
- OS/RTOS
- Compiler/Editor/Other Tools
- Debuggers
- “CASE” tools (UML, etc.)
- Configuration Management
- Defect Tracking

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Languages get debated among the safety circles.  Whether one language should be used over another depends on many circumstances.  Whenever possible, select a language with “safety features” that make it harder for the programmer to make mistakes (or easier for the compiler to catch them). Some features to look for are: strong data typing, control of pointers (or no pointers), and support for modularity. Being comfortable with the idiosyncrasies of a language is important.  Find or create checklists of common errors for the language chosen, and review the list regularly.  Use the checklist during inspections or reviews.  Add your own “personal” bugs to the checklist.
OS issues include memory management, priority inversion, interrupt latency, and many other factors.  From a safety perspective, make sure you understand how the OS handles errors generated within system calls.  Some OS’s have some level of safety-related certification, often for use in medical or aviation applications.  
When possible, select tools that make it easier to catch errors.  Compilers should have known-defect information available.
With CASE tools,  review automatically generated code closely, both safety-critical sections and non-critical sections (for interactions with critical). 
Don’t forget CM!  Last week’s buggy code should not be included in this week’s build...and definitely not in the delivered system!  Use CM for more than just code, though.  Requirements, design, etc. should all be CM’d to some extent.   Defect tracking is a related endeavor.  You don’t want to forget any unresolved problems, only to have them show up in the delivered version.  

## Using COTS Software

- Consider the COTS within **your** system
- Stability of the software
- Documentation
- Source code
- Errors and failure modes
- Who created the software?
- Defects
- Missing or Extra Functionality

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
COTS software comes with a lot of baggage, most of it unknown.  When safety or reliability is an issue, tread lightly.  Consider carefully before committing to COTS software.  This includes your OS.

Systems vary.  The COTS software may work perfectly in one system and fail miserably in another.  (Ariane 5 with Ariane 4 software.)  
Software with bug-fixes every week is definitely not stable enough for safety-critical systems.  Also consider upgrades - how different are the versions?  Can you stick with the tried-and-true?
Do you get more than a users manual?  Can you get design info, testing reports, or source code?  Can you talk to the team that built it, if you get into deep trouble?
Source code may be needed for inspection or analyses.  You may also need to modify the software to work in the system, or for safety reasons.  Will you still be able to get support if you do so?
What error handling does the software have (codes, exceptions).  Is there information on how it can fail or what its limits are (stress, load, etc.)?
Do you know anything about the team that created the software?  Did they follow a standard process?  What is their track record for quality software?
Can you get a list of known defects?  Check user groups and web-sites, though always consider the source.
Do you need to add functionality (glueware)?  How will extra functionality affect the system resources?  What happens if any of those functions are accidentally executed?

##  Design Goals

*** ******Design Goals***

- To develop a complete and correct software design.
- To flow software general & specific requirements, and design  specifications, into the software design.
- To develop a design that meets safety requirements.
- Assure interfaces within, and external to, the software will be correct.
- Develop verification plans & procedures for testing.
- Assure no new hazards arise due to design choices or (mis)interpretation of requirements.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Design is an important period for determining the trade-offs of different approaches to building in safety features.  What is the role of software and what is the role of hardware? Usually, the major choices for this trade-off were made when the software requirements were created from the system specification; however,  any true safing needs to be done with the proper balance of h/w functionality and s/w functionality.  And there is still time to make some “minor” adjustments to this balance.
Communication is critical to assure software doesn’t commit its design to missing and/or misunderstood (or changing) hardware specifications.
Make sure than any critical requirements are kept in the forefront when designing.  Safety features work best when they are designed in, rather than added on.

## Good Design Process

***Good Design Process***

- Communication.
- Inspections and reviews.
- Risk management.
- SW simulations and/or prototypes.
- Problem Tracking and Reporting
- Standards.
- Software configuration management plans
- Verification/Validation
- Documentation.
- Metrics

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Communication tops the list because it is key to all aspects of a project.  You need to understand clearly what the desired system is (hopefully spelled out in the requirements document), what the hardware capabilities and limitations are, and communicate the resulting software design back to the system team.  Good communications helps catch the hidden assumptions that can bite you later.
Risk management helps you focus your design on the highest risks.  Events that happen once in a million, even if serious, are usually a lower risk than those that are less serious but occur more often.  Be careful of being too optimistic in your probability estimates, though.  Do you really know that the chance of the software failing in a certain way is one-in-a-million?
CM should be an on-going process.  Code CM may be setup now, if it isn’t already in place.  
Documentation should not be a dirty word.  It should also not force you to kill a forest of trees.  Document the why’s and assumptions as well as the design.  You’ll forget why you chose a certain structure or design element.  You’ll move on to another job, but someone else may have to modify you’re program.  
Metrics help most when you have a history of information.  Try to select a few that give an idea of the complexity and defect-rate of the software.  Balance the extra time it takes with what you get out.  Also think of how the information may help in future projects.

## Design of Safety Critical Systems

***Design of Safety Critical Systems***

- Fault/Failure Detection, Isolation and Recovery (FDIR)
- Fault Tolerance/Defensive Programming
- Must Not Work Functions
- Must Work Functions
- Critical Interfaces/Commands
- Critical Data
- Coding Standards

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Fault DIR - stop the problem at a low level before it cascades into a failure. 
Failure DIR - recovering to a safe state (if possible).  
Detection - system needs way to determine there is a problem! Consider whether the software is getting enough information (sensors, etc.) to detect faults or failures.  Make sure there is enough hardware to do the job.  
Isolation - don’t let fault/failure propagate.  During design, consider what level to contain the fault/failure at.  Some will be local, some higher-up, depending on the fault/failure.  
Recovery - system must be “smart” enough to know what to do.  Could be request retransmission, switch to alternate storage if first is full, or return to a known safe state if the failure is severe enough.  Could be restore to a properly operating state, or reduce it’s functionality in the problem area until human intervention.  *** Know what state the hardware could be in and make sure the recovery doesn’t turn a minor problem into a catastrophic one!
Define what functions will be Must Work and which will be Must Not Work and for what states.  A function could be MWF in one state (valve closed during experiment) and MNWF in another state (valve must not be closed when venting in post-experiment state), or vice versa. 
Critical interfaces/commands - what interfaces allow software to influence safety critical hardware or to pass safety critical information? Can the state of a critical interface be saved between reset/power cycles (preserve state)? Consider hardware interfaces - how to recognize and deal with failed sensors, malfunctioning actuators, communications problems. 
What commands can initiate/stop actions that involve hazard causes or controls.  For commands, consider ready/arm/fire sequences.  Make sure one keystroke cannot initiate the command. Consider protection from out-of-sequence critical command. “Command” also include function calls - look at what can call a safety critical function and when. 
Critical data - is it isolated from “regular” data?  Multiple copies.  Stored in memory that is not cleared by reset/powerup? Reasonableness check?
Coding standards (not style) can be used to limit certain constructs or functions that should not be used. Create from lessons learned, previous projects, and information on the chosen language/compiler. 

## Keeping Safety in Mind

***Keeping Safety in Mind***

- Flow down safety requirements.
- Build the safety approach into design
- Feedback of safety analyses into design
- Have good verification plans.
- Keep safety in mind during design analyses, and all the way through the process

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Safety requirements need to be considered at design time as well.  Too often, safety measures are held till after the rest of the design is in place  when it  becomes a matter of too  little, too late.  Or, to do it properly would require major redesign
so why is left till late?  Some excuses go....
We’ll know more what  we’re up against, later.
We just can’t think of all the ways it could go wrong right now. 
There’s no time and besides, what could go wrong?
You can’t do it all, so there  needs to be a way to focus the safety effort to the areas that need  the most attention and to build in, up front, the proper level of control and mitigating features.  What are some of these features?
What types of data collision must be guarded against, what happens when things get mixed up?
  input/output timing
  multiple event
  out‐of‐sequence event
  failure of event
  wrong event
  inappropriate magnitude
  incorrect polarity
  adverse environment
  deadlocking in a multitasking system
hardware failures 

## Software Design Analyses

***Software Design Analyses***

- Design Logic Analysis
- Design Data Analysis
- Design Interface Analysis
- Design Constraint Analysis
- Design Traceability Analysis
- Complexity Measurement
- Independence and Interdependence Analysis
- Fault Tree and Failure Modes/Effects Analyses

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The software designer needs to determine the flow of data and commands through the system. Design Logic analysis determines the best flow for the control logic, what equations and algorithms are needed to bring this about,  and what the control hierarchy must be. The Data Analysis aspect of design analysis considers data needed w/in and w/out the system, the flow of data through the system, and identifies data required by critical processes. Check for safety critical data sharing the same memory area as non-critical data, or being accessed by functions that are designated non-critical.
Interface Analysis: Interface characteristics - interprocess communication methods, data encoding, error checking and synchronization. The sophistication of error checking or correction that is implemented should be appropriate for the predicted bit error rate of the interface. 
Design Constraint Analysis: where the timing, sizing, throughput factors are determined for the system hardware and required performance levels. Also consider equations and algorithms limitations, input and output data limitations (e.g., range, resolution, accuracy), sensor/actuator accuracy and calibration, noise, EMI,  digital word length (quantization/roundoff noise/errors), human factors, human capabilities and limitations, physical time constraints and response times, off nominal environments (fail safe response).
Design Traceability Analysis: assure the design incorporates  requirements and map all safety critical requirements to a design element.
Complexity: understand the level of complexity of your design.  Less complexity is usually better (less errors), but not always.  Look at the complexity of individual components, and the complexity of the interactions.
Independence/Interdependence: Look at design to verify that safety critical functions are independent of non-critical functions.  Conversely, look at the non-critical functions to see how they may inadvertantly affect the safety critical functions, via shared data, potential function calling sequence, communication, or other mechanisms.
Fault Tree: Like the PHA, a top-down analysis commonly applied to the system, now being used with software.  Looks at potential failures, and traces back to find what could cause the problem. Can be started with the architectural design, updated as the design progresses.
FMEA: A bottom-up analysis that looks at each component and how it can fail, then follows the failure through the system to see if a hazard/failure can occur.  Needs at least the detailed design.  Time consuming, so don’t use for all components!  Can be combined with FTA.

## Software Safety Tasks

***Software Safety Tasks***

**Analyze S/W Design**
- Safety Requirements incorporation
- Identification of safety controls & features
- Identify safety critical Software Components
- Assure no new Hazards
**Update Analyses **(HA, Criticality, ...)
**Participate in Software and System Design Activities (especially Inspections & Analyses)**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The previous analyses were mostly focused on creating a good design, though safety should be a factor in all of them.  Safety tasks during design focus on updating previous analyses with new data, and verifying the safety of the proposed software design.
One of the goals of a good, safe design is to lower the risk assessments for the software and distill down to a minimum area that requires more stringent attention.
While analyzing any sub-system such as software, the safety analyst must always keep in mind any potential impacts to the entire system.  Remember that “everything is connected to everything else”.  
When updating the different analyses, that’s a good time to verify previous assumptions about the system requirements and potential hazards.

**Coding for Safety**

**Implementation**

## Implementation Goals

***Implementation Goals***

- To develop complete and correct code.
- To develop code that meets safety requirements
- To flow software detailed design into code.
- To flow software safety design features into code.
- Unit test all code
- Finalize s/w system  integration test procedures

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Implementation is when you get to take a good design and create an even better program. Some things to keep in your mind when coding:
safety requirements and features (including FDIR)
Hardware capabilities and constraints
Testability - ability to show errors
Ways a module/unit can interact with hardware, other software
Ways a module/unit could fail

## Coding Considerations

- Requirements Passed Down Through Designs to the Coding Level.
- Coding of Software Controls of Safety Hazards Are Implemented.
- Checklists and Standards
- Defensive Programming
- Refactoring
- Unit Level Testing
- Sub-Unit Integration (S/W to S/W)
- Simulations & Real H/W Interface Verifications

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Be cognizant of the types of errors which can be introduced into non-safety-critical code which can compromise safety controls.   Coding checklists should be provided to alert for these common errors. Safety checklists verify that safety requirements have, in fact, been flowed into the software code.
Defensive Programming incorporates a degree of fault/failure tolerance into the code, sometimes by using software redundancy or stringent checking of input and output data and commands. An example of defensive programming is sometimes called "come from" checks.   Critical routines have multiple checks in them to test whether they should be executing at some particular instant.  One method is for each preceding process to set a flag in a word.   If all the proper criteria are met then the routine in question is authorized to execute. However, software alone cannot achieve true system redundancy and failure tolerance. Appropriately configured hardware is necessary. 
Refactoring: if you apply refactoring to your code, be especially careful that the new code is truly identical in functionality to the original code.  For safety critical code, any tests will need to be re-run.  
Make sure your unit/class works first as designed, before you begin to integrate it.  
Integration tests is where you find the unexpected interactions between software elements, and with the hardware.  It’s also where you find out how good your ICD was.

## Defensive Programming

- Range and Limit Checking
- Input verification/validation (per mode/state)
- Communication checks (ack/nak, CRC, etc.)
- Redundancy w. functions, data, sensors, etc.
- Background memory checks
- Start-up self checks (is everything where it should be & working  prior to start of operations?)
- Inhibits – Design in, not an implementation decision
- Multiple commanding (e.g. Ready/Arm/Fire)

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
If fault/failure handling wasn’t designed it, you can at least add something now.  These are examples of things you can do to make your code safer.

## Coding Analyses

- Logic, Data, and Interface Analyses
- Unused Code Analysis
- Interrupt Analysis
- Measurement of Complexity
- Code Constraint Analysis
- Underflows/Overflows
- Update previous analyses

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Code logic analysis evaluates the sequence of operations represented by the coded program and detects logic errors. 
Data analysis focuses on how data items are defined and organized, and ensures that these data items are defined and used properly. Of particular concern to safety is ensuring the integrity of safety critical data against being inadvertently altered or overwritten. 
Code interface analysis verifies the compatibility of internal and external interfaces of a software component. 
Unused code is at best a waste of space.  At worst, it is a potential safety hazard.  Look for it and remove it.  
Interrupt Analysis looks at the effects of interrupts on program flow and data corruption.  Among other things, examine the critical areas to make sure they cannot be interrupted, and that non-critical areas can be interrupted if necessary.
Measurement of Complexity: As a goal, software complexity should be minimized to reduce likelihood of errors.   Complex software also is more likely to be unstable, or suffer from unpredictable behavior. Complexity can be measured via McCabe's metrics and similar techniques.
Code Constraint Analysis: The criteria for design constraint analysis applied to the detailed design in section 5.3.4 can be updated using the final code.  At the code phase, real testing can be performed to characterize the actual software behavior and performance in addition to analysis. The physical limitations of the processing hardware platform should be addressed. Timing, sizing and throughput analyses should also be repeated as part of this process to ensure that computing resources and memory available are adequate for safety critical functions and processes.
Underflows/overflows: Eliminate all potential underflows, overflows, divide-by-zero, and other errors.  To catch the ones you missed, make sure you have error handle all potential errors. 

**Finding the Problems**
**(Bugs, Errors, and Faults)**

**Testing**

## Unit Testing

- Usually performed by the developer, or another member of the development team.
- A “unit” is a flexible and variable term. Don’t make a “unit” the entire program!
- Units use stubs and drivers to simulate the rest of the software.
- White Box testing (path, statement, memory, safety)
- Black Box testing (typical, extreme, combinations of inputs)
- Error Handling - make sure this works!

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Units can be modules, classes, collection of modules, tasks, CSCI, etc. A unit should be a logical division that is “big enough” to be tested but “small enough” to allow access to the innards of the software
Stubs represent entities called by the unit.  Drivers represent entities that call the unit.
Error handling:  Simple error handling (bad data rejection) can be tested at this level.  More complicated error handling (FDIR) may not be testable until well into integration testing.
Safety tests - verify that safety critical units meet the safety requirements.  Perform any safety tests that require access to the code (instrumentation) at this time.  

## Unit Test “Best Practices”

- Design the test and write down the procedure, including the files used for stubs and drivers.
- Make sure the stubs and drivers are under some sort of configuration control.
- Review your software using a coding checklist prior to testing it.
- Keep a list of errors found in your code.  Feed this information back into your coding checklist.
- Share your bug list with your team, if you won’t be penalized for it.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Procedures don’t have to be complicated documents.  At least scribble in a log book what you did (and why). The more information you record, the easier it is to reproduce the test if necessary.  
For reproducibility, know the files you used, and be able to locate them again (in original form) if necessary.
The more you learn from your own errors or those of others, the less you will make those particular errors.  Hopefully, your management will be enlightened and not try to penalize you for admitting to errors!

## Testing COTS Software

- Previous testing may or may not be valid *in the new system.*
- Test for ways it can fail.  How does your software respond to those failures?
- Look for ways that the OTS software can influence the safety critical code.
- Stand-alone testing prior to integration.
- Glueware and wrappers.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Ariane 5 - reused Ariane 4 software but in new system/environment.  It worked as designed, but not as desired!
Software fault injection can be used to find out how the COTS can fail, so you can protect your system from those failures.  Examples of COTS failures: 
overwriting a memory location where a safety critical variable is stored
getting into a failure mode where the OTS software uses all the available resources and prevents the safety critical code from executing
clogging the message queue so that safety critical messages do not get through in a timely manner.
Be aware that glueware and wrappers (code to integrate the COTS software and to “buffer” the COTS) may be necessary.  The earlier you find out how the COTS software works, the earlier you can add these types of software to your design.  If you wait until the testing phase to really understand the COTS software, you may find you need to add wrappers to protect your code from malfunctioning COTS!

## Integration Testing

- Putting the “puzzle” together.
- Who performs the tests?
- Decide integration order during design phase. Still uses stubs and drivers, but “unit” under test is a larger entity.
- Mostly “black box” tests - functionality, error handling, performance.
- Safety tests that need access to software details should be done now (if not at unit testing).
**Test for interactions between modules, especially those designated safety critical.**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
In larger organizations, integration test is performed by a separate test team.  In smaller groups, the developers will perform these tests.
Keep hardware schedule in mind - don’t hold up integration because hardware isn’t done yet!
This is where problems with the Interface Control document (or interface definition) will surface.  This is also where problems relating to interaction among the pieces will start to be seen.  
Safety critical units should be thoroughly tested for undesired interactions with the rest of the system.  Watch for memory overwrites, system resource hogging, message queue problems, and other difficulties that can cripple the safety critical unit.

## Testing on the “real” hardware

- As soon as hardware is available, use it for testing (unit, integration, system)!
- Don’t do any tests that can “break” the hardware on the real system.  Use a simulator or engineering unit for those.
- Real hardware always acts differently from how you expected it to.
- Software developers and hardware designers need to communicate, early and often, to prevent misunderstandings and problems.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
No matter how well you believe you understand how the hardware operates, or how good your simulator is, real hardware has a mind of its own!  Don’t wait until you have a complete program to try it out on the hardware.  Do as much testing as possible, as early as possible, on the real hardware.
Communication, as always, is vital.  

## System Tests

- Functional/Acceptance Test
- Performance/Load Testing
- Stress Testing
- Stability/Reliability Testing
- Safety Verification Testing
- “Oops” Testing
- Other tests specific to environment (installation, compatibility, conformance to standard, etc.)

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
You want to test that the system does what it is supposed to do (functional against the requirements) and what the customer desires (hopefully the same – acceptance).  You also need to test that the system does not do what it is not supposed to do.  (must work/must not work)
Stress testing is a good way to find out how “reliable” your system is, and to find buried bugs.
Oops tests include Resistance to Failure (where are errors caught, and how gracefully does the system respond to them?), Disaster Testing ( What happens if you pull the plug?  Does the system “fail safe”?), and “Red Team” Testing ( unstructured test where users try to break the system.  No hardware breakage allowed! Use a simulator or modify the hardware to prevent the possibility of a hazard.)

## Stress Testing

- See how much the system can handle before it “breaks”.
- Aspects that can be stressed (examples):
- CPU usage
- sensor input or output rates
- telemetry rates
- memory utilization
- amount of available memory
- network utilization
- Adequate margin between the system’s peak usage and the breakdown point is desired.
**DON’T BREAK THE HARDWARE!**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Stress testing can teach you much about your system.  It’s not designed to find bugs, but to give confidence that the system can handle most unexpected situations. 

## Stability/Reliability Testing

- How well does the system operate for extended periods of time?
- System is run in normal mode of operation - occasional peak performance allowed, but not stress testing the system.
- Can the system handle intermittent bad data?
- Is there a sensitivity to event sequences?
- Does memory leakage cause problems after a period of time?

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Unlike stress testing, stability or reliability tests are run with the system operating normally.  Getting up to occasional peak loads is part of running “normally”. 
You do want to input “expected” errors and anomalies, such as a user mistyping a command, two sensors wanting servicing at the same time, or other situations that may occur.  Keep away from the really unlikely errors for this type of testing.  

## Safety Verification Testing

- Safety tests designated for each hazard control.
- Verify partitions, firewalls, or other software constructs that isolate safety critical code.
- “Fail” the hazard controls in a multi-tolerant system.  For example, in a two-fault-tolerant system (three controls), try all combinations of two failures.
- Verify hazardous commands.
- Verify software correctly handles out of sequence commands, hazardous commands issued in an incorrect state, and other possible errors.
- Software Safety (usually SQA) should witness all software safety testing.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
All the great safety features you designed and coded now need to be tested.  
Formal testing will verify each hazard control or mitigation that software is a part of, as well as hazardous commands. This testing should have written test plan that is reviewed/signed off on, be witnessed by Software Safety or SQA, and be carried out according to a written procedure.
Slightly less formal safety testing would check the software FDIR/error handling system, look for ways that the protection systems might be accidentally circumvented, and other types of problems.  

**Scoping and **
**Tailoring a Software**
**Safety Program**

## Overview

***Overview***

- Scoping:
- Categorization of Software
- Identify safety critical software
- Categorize safety critical software subsystems - How critical is it?
- Tailoring:
- Determine extent of development effort and oversight required
- Determine the Required Software Safety Effort
- Match Safety Activities to meet Development Effort

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
More important than any one factor related to scoping a software safety effort is the risk factor.
Starting with systems level functions which have been identified as safety critical,  identify the role of  software in safety mitigation.   We just looked at those in the Safety Critical module.
Within software, scoping consists of identifying areas where there is safety critical software, planning the software development effort and then tailoring the software safety effort to fit the development. 
First search the PHA, S/W HA, S/W Risk Assessments, etc.  From these analyses system/sub-systems can be categorized as safety critical or not.
If the safety engineers responsible for software are fortunate, they will be involved early in the planning for the project.  It is easier to plan for software safety when the plan for development is being put together, rather than as a reaction to plans already made.
 Both software development planning and scoping of the software safety effort are impacted by risk. 
A full scale software safety effort is typically performed on a safety critical flight system, e.g. a manned space vehicle, or high value, one-of-a-kind spacecraft, all aircraft, critical ground control systems, critical facilities, critical ground support equipment, unmanned payloads on expendable vehicles, etc.   Other types of  aerospace systems and equipment often utilize less rigorous development programs, such as non-critical ground control systems, non-critical facilities, non-critical ground support equipment, non-critical unmanned payloads on expendable vehicles, etc.
In those cases, subsets of the milestone reviews and software safety development and analysis tasks can be used. 

## Scoping, Software Categorization

- Real World Software Level Indicators
- Degree of Control
- Complexity
- Timing Criticality

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
So, where do we start looking at software categorization for the purpose of ultimately determining the safety criticality and level of effort needed  to produce safer software?

Degree of control:
Does the software control the hazardous operation?  
Does the software provide hazard prevention (monitoring, automatic safing, etc.)?
If the software malfunctions, can it cause a failure or fault?  
Does it provide the only source of information for an operator to make a safety-related decision?

Complexity:
Number of Interfaces between software elements, software/hardware, and software/user.
How many subsystems are controlled by software?
Does the software contain interacting, parallel executing processes

Timing Criticality:
How fast does the software have to respond to a hazardous condition?
How many processes have real-time requirements (safety related or not)?

## Software Control Categories(From NASA Software Safety Guidebook)

***Software Control Categories******(From NASA Software Safety Guidebook)***

- IA - complete control over potentially hazardous hardware
-           - No time for intervention
- IIA - control over potentially hazardous hardware
-             - Time for intervention
- IIB - displays information requiring immediate operator action to mitigate hazard
- IIIA - issues command over potentially hazardous hardware requiring human action -several redundant independent safety measures
- IIIB - generates information of safety critical nature used to make safety critical decisions -several redundant independent safety measures
- IV - does not control safety critical hardware - nor provide safety critical information

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
A different perspective that is more software specific will help scope the software effort.
Categorizing software which controls safety critical functions in the system  is based on the degree of control the software exercises over the functions.
The Military have nicely set up some categories for us.  MIL-STD-882C
Software which can cause a hazard if it malfunctions is included in the category of high risk software.  Software which is required to either recognize hazardous conditions and implement automatic safety control actions, or which is required to provide a safety critical service, or to inhibit a hazardous event, will require more software safety resources and detailed assessments . 
Software which which is required to recognize hazardous conditions and notify a human operator to take necessary safety actions will not require as rigorous a safety effort as the former examples.   This example assumes that the human operator has redundant sources of data,  independent of  software, and can detect and correctly react to misleading software data before a hazard can occur.
In cases where an operator relies only on software monitoring of critical functions, then a complete safety effort is required. 

## System Hazard Risk Index

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Remember this, I hope so, it was only a few minutes ago.  

## Software Sub-system Categories

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**

Complexity increases the possibilities of errors, an important point to consider.  Increasing errors lead to the possibility of fault, which leads to failures.  This chart takes into consideration the complexity of the software when categorizing the software.  The chart also relates back to the system risk index and has already eliminated level 1.  At this point the software category links the complexity of the software, control which the software exerts on a system and the system risk index.  The links will be important in creating a Software Hazard Criticality Matrix. 

## Software Hazard Criticality Matrix

_***Software***_*** Hazard Criticality Matrix***

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The Software Hazard Criticality Matrix is established using the hazard categories for the columns and the Software Control Categories for the rows.  The matrix is completed by assigning Software Hazard Risk Index numbers to each element just as Hazard Risk Index numbers are assigned in the Hazard Risk Assessment Matrix.
This table relates the software category to the software risk index NOT to the system risk index.  The two may appear similar.  The difference is mainly that the top row of the Software Hazard Criticality Matrix is related to System Risk Index level 2.  The 1’s that are shown in the top row of this matrix are not the same as the 1’s in the System Risk Matrix.  1’s are not allowed for the System Risk index.  Having eliminated them there, the software risk index now starts with 1.  Hope this is not too confusing!
Unlike the System Risk Index,  a low index number  from the Software Hazard Criticality Matrix does not mean that a design is unacceptable.  Rather, it indicates that greater resources need to be applied to the analysis and  testing of the software and its interaction with the system. 

Instructor’s Note: Show the next slide on the second projector along with this slide.

## Software Hazard Risk Index	Suggested Criteria

- 1	High Risk - significant analysis and testing  resources
- 2	Medium risk - requirements and design analysis and in-depth testing required
- 3-4	Moderate risk - high level analysis and testing acceptable with management approval
- 5	Low Risk - Acceptable

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The index numbers are from the Software Hazard Criticality Matrix.

## Recommended Safety Effort

| Software Category | Hazard Severity |  |  |  |
| --- | --- | --- | --- | --- |
|  | Catastrophic | Critical | Moderate | Negligible |
| IA
System Risk Index 2 | Full | Full | Moderate | Minimum |
| IIA & IIB
System Risk Index 3 | Moderate | Moderate | Moderate | Minimum
 |
| IIIA & IIIB
System Risk Index 4 | Minimum
 | Minimum
 | Minimum
 | Minimum
 |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Using the Military’s categorizations and the System Risk index (Remember Level 1 of the System risk Index is eliminated - we don’t do that sort of project - at all) we map to this matrix for a basic determination of the level of effort for software development and safety analyses .

So now, what does this mean in real terms.
At this point we know something about the system and some of its components.   We know the system risk index for some of the system components (this information again, was uncovered by FMEA’s , CILs, PHA’s etc.). 
The software complexity, critical timing requirements, and level of control of safety critical functions,  lead to categorizing the software.  This, along with the system risk index lead to a Software Risk Index from the Software Hazard Criticality Matrix.  Scoping consists of using this information and the Required Software Safety Effort  table (above) to determine the level of effort needed for the different subsystems/systems which make up  a project.

## TAILORING

***TAILORING***

- Determine the Required Software Safety Effort
- Match Safety Activities to meet Development Effort

**Tailoring**

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
The scope of the software development and software safety effort is dependent on risk.  Software safety tasks related to the life-cycle phases and milestone reviews are listed in Table 3-1 of the guidebook.  The tasks that will be performed are dependent on the risks associated with the software.  Here we determine whether the development will be minimum, moderate, or full.
We do this by first looking  at the software sub-system categories risk and then categorizing safety critical sub-systems. This will allow us, through the use of the Required Software Safety Effort Table, to determine the level of the  development effort and required oversight. 

## Full Software Safety Effort

- Example systems:
- life support
- fire detection and control
- propulsion/pressure systems
- Pyrotechnics and ordnance systems

- Systems and subsystems in this category have severe hazards which can escalate to major failures in a very short period of time.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Moderate Software Safety Effort

***Minimum Software Safety Effort***
- Systems in this category have either a very low hazard potential, or control of the hazard is accomplished by non-software means.

- Systems or subsystems in this category have either a limited hazard potential or, if they control serious hazards, the response time for initiating hazard controls to prevent failures is long enough to allow for notification of and response from human operators.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Full Software Safety Effort

- Independent Verification and Validation (IV&V) and in house Software QA.
- Formal, Rigorous Software Development for all code.
- Safety design features in the software.
- All or most of the Safety and Software Safety Analyses.
- May need to meet specific standards, such as FAA, military, or medical.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Moderate Software Safety Effort

- In house Software QA oversight.
- Rigorous development process only for safety critical code.
- Rigorous safety process, including analyses, only for safety critical code.
- Not all safety analyses are required.
- Monitor non-safety critical code for potential impacts on safety critical areas.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Minimum Safety Effort

- Normal Software QA effort.
- Normal software development process, with extra attention given to safety related code.
- Minimal Software Safety analyses.
- Software safety as part of the development process, not a separate effort.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## What can you Tailor?

***What can you Tailor?***

- The Process
- Method
- Life-Cycle Model
- Tools
- Type and level of reviews
- Verification & Validation
- Analyses
- Which will provide the most
- Extent of the Analyses
- Organization
- Communications

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
There are quite a few elements which can be tailored.
For Example,
The process, - What development process is best for a particular system, environment, schedule, cost?
What reviews
method-  make heavy utilization of reusable modules; OOA/OOD; embedded web technologies, structured design. - but becareful!!! Remember Arian 5’s Mistake
life-cycle- modify spiral model, making use of well documented reusable s/w
tools- make use of Formal Inspections during requirements phase and then only on safety related design, portions; create or use simulators, code generators
The organization, - use senior members of team to handle flow down of requirements and safety issues.  Make sure there is good communication between systems, hardware, safety , software engineering, software QA, and project management.  
Combinations:  Then there are the combinations of all of the above.  This isn’t like going through a menu, this process, that method, the other tool.  Each element must work well for the specific aspect for which it was chosen (client, development organization, schedule, technology, etc.,) and work well with each of the other elements of development (the tools, the organization, life-cycle, method, etc.)  

## What to tailor in your Software Safety Program

| Analysis | Select the analyses that are necessary to assure a safe system, based on the level of effort.  Eliminate analyses that are not appropriate, based on cost, benefit, or applicability. |
| --- | --- |
| Inspections | Determine how many inspections to have.  If choosing to have only a few inspections, select the most critical documents or code to be inspected. |
| Reviews | Select the number of formal reviews necessary to get adequate insight into the software development process. |
| Testing and Verification | Modify the level of testing based on the criticality of the software under test. |

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Software Safety Effort per Phase

***Software Safety Effort per Phase***

**NR** - Not Recommended   ** R** - Recommended    **HR **- Highly Recommended    **M** - Mandatory
- See NASA Software Safety Guidebook for other life-cycle phases

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
This chart as an example provides software safety guidance for planning both development and analysis tasks during the requirements phase.  In the guidebook the rest of the phases of the life-cycle are also listed.  these tables are modifications of tables that appear from an early International Electrotechnical Committee (IEC) draft standard IEC 1508.
Each tables lists techniques and recommendations for use based on safety effort level for a specific software development phase or phases.   The recommendations  are coded as:  Mandatory, Highly Recommended,  Recommended, and  Not Recommended.
These tables provide guidance on the types of assurance activities which may be performed during the life-cycle phases of safety-critical software development.  For this class, the Required Software Safety Efforts values displayed will determine which development activities are required for each level of effort.
Most of the "Not Recommended" entries result from consideration of time and cost in relation to the required level of effort.  A mixture of entries marked as "Recommended" may be performed if extra assurance of safety is desired. "Highly Recommended" entries should receive serious consideration for inclusion in system development. If not included, it should be shown that safety is not compromised.
These are guidelines.  Every project is different.
There is direct application of these phases to fast prototyping and the spiral model of life-cycles.  
 Instructor’s Note: A general discussion on this chart is in order at this point.

## Summary

- Determine software control and complexity
- Determine each software portions’ contribution to safety
- Establish categories with a cross index to hazard level
- Determine a level of effort needed to assure safer software
- Further tailor the effort to your particular needs and situation

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
There is just too little time to go into this and we must only give you a briefest look into what is possible. Of course, you should carefully consider what is needed for your organization and how best to categorize and tailor your efforts.  We are just giving you an example here.

**Or how to keep your software from really screwing up!**

**Good Programming Practices**
**for Safety**

## Vital Processes

**Get the requirements right!**
- Configuration Management!
- Document *what* you are doing and *why* you are doing it! You will not remember when asked 6 months from now!
- Document interfaces and keep them up-to-date!
- Understand your hardware and purchased software (OS).  Know how they can fail, throw exceptions, or impact your software.
- Design error detection and handling in from the start - not as an afterthought
- Track your bugs
- Plan and script all tests
- Analyze your test outputs.
- Perform load and stress tests.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Highlights from the list:
Requirements need to be understood and documented.  Use prototyping, sessions with customer, whatever means necessary to get these right.  When they change, make sure it propagates back up to the requirements document!
“Why” is important, especially on a longer-term project, or one where parts will be reused down the road.  People move on, and memories fade.  
Design fault/error handling in from the start, don’t tack it on at the end.
Examine the outputs from your test closely.  [One flight, we had a communications problem that would have been obvious to someone familiar to the system, but due to scheduling problems someone else ran the test.  With a verbal “everything was fine”, the test logs were not examined.  If they had been, the problem would have been discovered.]

## Things to Do

- Initialize all variables, especially pointers
- Enforce data typing!
- Use () to force precedence order
- Don’t trust your input values!
- Be nice to downstream functions - check your output!
- Isolate hardware-dependent code
- Isolate code that depends on anything not nailed down (hardware, OS, etc.)
- Use a watchdog timer.  Don’t strobe it in an interrupt routine, though!
- Store multiple copies of critical variables, preferably on various media or in different segments.
- Check the stack - at least during development and test!
- Put safety critical code in ROM when possible

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Highlights:
Input/output should be checked.  If not both, select one and be consistent
Make sure that your critical data is safe.  If putting on various media, consider how you will tell which one is correct (two out of three?).
Know how close you get to filling up the stack.  Check this as early as possible, but at least during load or reliability testing.  
When safety critical code or data doesn’t need to be changed, use ROM.  Don’t let your OS put it in RAM!

## Things to Do

- Check your resource usage - memory, disk, ROM, etc. Don’t just calculate!
- Use come-from checks for safety critical modules.
- Use two-stage commanding (arm/fire) and functions for critical operations.
- Define all interrupts, traps, etc. - even ones you don’t think could happen!  Have something innocuous happen if they do fire.
- Perform an orderly shutdown if commanded or unsafe.  May need battery or capacitor power backup to give enough time.
- Define safety critical sequences.  Check if commands or messages are received in proper sequence. Updated status flag between steps of a hazardous operation is one method of checking.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Don’t set off safety-critical operations without safeguards, such as come-from checks, proper sequence of messages or steps,  or two-stage commanding. Make sure it’s what you really wanted to do! 
Think about what you need to shutdown safely.  If it needs some hardware, bring it up early in the design.  

## Things to Do

- Perform reasonableness checks on all safety-critical inputs.
- After warm boot, make sure software in known safe state.  Initialize all variables, static and automatic.
- When you have to safe, send the software/hardware configuration information to the operator/ground computer.
- Check your complexity.
- Read back what you wrote (hardware, memory)
- Know what modules/variables depend on others.
- CPU Self test and ROM test to verify your processor is working correctly and your stored program has not been corrupted.
- Fill ROM/RAM/Flash with a known pattern.  Avoid “halt” if stopping the CPU leaves the system in an unsafe state!

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
An input might be “in range”, but unreasonable based on the values of other sensors.  Check for the situation, and determine what is the best thing to do.
Know what state or configuration you were in before safing.  If a sensor was turned off, you don’t want to rely on it when you come back up. 
Program slicing can help you find out what code can affect a variable.
“Halt” may not be the best CPU instruction.  Stopping the CPU when a problem develops may turn an error into a disaster.

## Things Not to Do

- Gotos
- Scratch files for safety critical information
- Halt or stop CPU if it can leave system in an unsafe state
- Program patches.  One may not kill you, but two or more is a nightmare.
- Same I/O port for both critical and non-critical operations.
- I/O addresses for critical/non-critical operations where an address bit failure could get them mixed up!
- Initiate hazardous sequence from one keystroke.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Keep your bits apart (safety and non-critical).  It’s too easy to get the ands/ors wrong when writing out a port.  You can put extra checking in for the safety critical port, and not have to worry so much about the non-critical port.  This will affect the hardware design!  
Patching may be necessary, if your system develops a problem in flight.  It’s not recommended when other options exist.
Avoid “macros” for hazardous operations. 

## Things Not to Do

- Make “proceed” question so automatic no one pays attention. If command innocuous even if incorrect, don’t require agreement.
- Turn off safety interlocks for a test and then not turn them back on when testing is completed.
- Don’t use all 1s or all 0s to compare against hardware (sensor) values.
- Call non-reentrant modules from interrupt routines.
- Optimize too early.

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Do you even look at “OK to delete” or similar questions under MS Windows any more?  Operators are human, and they get tired of mindless questions.  Save the confirmations for the really important commands.
Sensors often fail to all 0’s or all 1’s.  
Optimizing is good, but don’t start until your code works.  Then profile – know where the bottlenecks really are, which is often not where you think it is.

## Things to Think About

- Dynamic memory
- Memory paging or swapping
- Number of entry/exit points in function
- Functions with side effects
- Variable reuse
- Timing dependent code (such as timing loops, etc.) that will change with processor
- What safety critical aspects (timing, sequence, etc.) will users be allowed to change.
- Multithreaded programs
- Stability of requirements
- Compiler optimization

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Resources**

## Standards

- NASA Standards (http://standards.nasa.gov/sitemap.htm)
- NPG 8715.3	NASA Safety Manual
- NSTS-1700-7B Safety Policy and Requirements for Payloads (Shuttle and ISS)
- NASA-STD-8719.13A	NASA Software Safety Standard
- NASA-GB-A302 Software Formal Inspections Guidebook
- NSTS-22254	 Methodology for Conduct of Space Shuttle Program Hazard Analyses
- SSP-50038 Computer-Based Control System Safety Requirements**, **ISS Program
- NPD/NPG 8730 “NASA IV&V Processes “
- IEEE Standards
- IEEE 12207  Information Technology - Software Life Cycle Processes
- IEEE 830-1998 Recommended Practice for Software Requirements Specifications
- IEEE 1016-1998 Recommended Practice for Software Design Descriptions
- IEEE 1228-1994 Standard for Software Safety Plans
- Other Standards
- MIL-STD-882D System Safety Program Requirements (C version January 19, 1993)
- DO-178B Software Considerations in Airborne Systems and Equipment Certification (Federal Aviation Administration).
- ISO 9000-3	Guidelines For The Application Of  ISO 9001 To The Development, Supply, Installation And Maintenance Of Computer Software

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Status of the new NASA Software Safety Guidebook:
	number assigned?
	published? If not, when?
	where available?

## Books

**Software Safety and Reliability : Techniques, Approaches, and Standards of Key Industrial Sectors**, Debra S. Herrmann, et al., March 2000
**Safeware : System Safety and Computers**, Nancy Leveson, April 1995
**Safety-Critical Computer Systems**, Neil Storey, August 1996
**Software Assessment: Reliability, Safety, Testability**, Michael A. Friedman and Jeffrey M. Voas (Contributor), August 16, 1995
**Semsplc Guidelines : Safety-Related Application Software for Programmable Logic Controllers**, February 1999

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

**Notes:**
Many good software engineering books exist, and are not listed here.  These books focus on software and system safety.  

## Websites

***NASA Websites***
- NASA Lessons Learned		http://llis.nasa.gov/llis/llis/main.html
- NASA Technical Standards	http://standards.nasa.gov/sitemap.htm
- NASA Online Directives Information System (NODIS)	http://nodis.hq.nasa.gov/
- ISS Documentation (PALS) http://iss-www.jsc.nasa.gov:1532/palsagnt/plsql/palshome
- NASA Langley Formal Methods group http://atb-www.larc.nasa.gov/fm/index.html
- GSFC Software Engineering Laboratory	http://sel.gsfc.nasa.gov/
- NASA Software Assurance Tech. Center	http://satc.gsfc.nasa.gov/homepage.html
- NASA IV&V Center		http://www.ivv.nasa.gov/
- NASA Software Working Group	http://swg.jpl.nasa.gov/index.shtml
***Software Safety***
- Software System Safety Working Group 	http://sunnyday.mit.edu/safety-club/
- Safety critical systems links:	http://archive.comlab.ox.ac.uk/safety.html
- A Framework for the Development and Assurance of High Integrity Software 	 			http://hissa.ncsl.nist.gov/publications/sp223/
- Safety Critical Resources http://www.cera2.com/WebID/realtime/safety/blank/org/a-z.htm

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Websites

***Reference Websites***
- Software  Engineering Body of Knowledge (SWEBOK)	http://www.swebok.org/
- Software metrics links:  		http://www.totalmetrics.com/resource/links.htm
- Software Methods and Tools:	http://www.methods-tools.com/html/tools.html
- Standards (and 37 Cross References) http://www.cmpcmm.com/cc/standards.html
-  ***Software QA and Testing***
- Society for Software Quality:		http://www.ssq.org/welcome_main.html
- Software Testing hotlist:	  	http://www.io.com/~wazmo/qa/
- Guidance for Industry, General Principles of Software Validation, Draft Guidance Version 1.1  (FDA) 		http://www.fda.gov/cdrh/comp/swareval.html
- Software Testing Stuff:			http://www.testingstuff.com/testing2.html
- Software QA/Test Resource Center		http://www.softwareqatest.com_[/](http://www.softwareqatest.com/)_
- SR/Institute's Software Quality HotList http://www.testworks.com/Institute/HotList/index.9.html
- TestingCraft – tester knowledge exchange	http://www.testingcraft.com/index.html

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001

## Websites

-  ***Miscellaneous***
- Software Project Survival Guide http://www.construx.com/survivalguide/chapter.htm
- Sample Software Documents	http://www.construx.com/doc.htm
- Software Documents, military,	http://www.pogner.demon.co.uk/mil_498/6.htm
- Annals of Software Engineering	http://manta.cs.vt.edu/ase/
- Software Engineering Readings
-              http://www.qucis.queensu.ca/Software-Engineering/reading.html
- Introduction to Software Engineering http://www.caip.rutgers.edu/~marsic/Teaching/ISE-online.html
- Software Engineering hotlist http://www.cc.gatech.edu/computing/SW_Eng/hotlist.html
- Brad Appleton's Software Engineering Link http://www.enteract.com/~bradapp/links/swe-links.html
- Best Manufacturing Practices guidelines		http://www.bmpcoe.org/guideline/books/index.html
- Embedded systems programming	http://www.embedded.com/
- Embedded systems articles	http://www.ganssle.com/

- NASA Headquarters Office Of Safety & Mission Assurance

- November 27, 2001