STAMP/STPA
Beginner Introduction
Dr. John Thomas
System Engineering Research Laboratory
Massachusetts Institute of Technology

Agenda• Beginner Introduction– What problems are we solving?
– How does STAMP/STPA solve those problems?
– Simple STAMP/STPA examples• Intermediate tutorial– Guided exercise: Apply STPA to a real system• Research presentation– Recent STPA research results
2©

Mars Polar Lander• During the descent to Mars, thelegs were deployed at analtitude of 40 meters.
• Touchdown sensors (on thelegs) sent a momentary signal• The software responded as it
was required to: by shutting
down the descent engines.
• The vehicle free-fell and wasdestroyed upon hitting thesurface at 50 mph.
No component failed!
All components performed exactly as designed!

3

Boeing 787 Lithium Battery Fires• 2013 – 2014
• Reliability analysis
predicted 10 million
flight hours between
battery failures• Two fires caused by battery
failures in 52,000 flight hours• Does not include 3 other lessreported incidents of smoke
in battery compartment

Just a simple component
failure?

4©

Boeing 787 Lithium Battery Fires• A module monitors for
smoke in the battery bay,controls fans and ducts toexhaust smoke overboard.
• Power unit experienced
low battery voltage, shut
down various electronics
including ventilation.
• Smoke could not be
redirected outside cabin
All software requirements were satisfied!
The requirements were inadequate8©

Schiaparelli Lander (2016)• 11km: Parachute deployed• 3.7km: IMU saturated• Negative altitude calculated• Parachute jettisoned• Thrusters off• Impact at 300 km/h (186mph]• Designed to withstand 10km/h

All components operated as designed!
No component failure!
http://spacenews.com/esa-mars-lander-crash-caused-by-1-second-inertial-measurement-error/

HITOMI Satellite (2016)• Unable to detect brightstars for reference• Parameters for Safe Hold
Mode were incorrect• Investigative subcommittee• Need for “approach to
examine the overall designof the spacecraft”• JAXA• “We were unable to let goof our usual methods”

All components operated as designed!
Not a simple component failure!
http://www.asahi.com/ajw/articles/AJ201606200006.html

Basic Control TheoryController
ProcessModel (beliefs)
Control
Actions

FeedbackControlled Process• Provides another way to think about accidents• Forms foundation for STAMP/STPAProblems with Software

Quote• “The hardest single part of building a softwaresystem is deciding precisely what to build.”-- Fred Brooks, The Mythical Man-MonthAutomotive recalls rapidly increasing!

Image: https://hbr.org/2010/06/why-dinosaurs-will-keep-ruling-the-auto-industry/ar/1©

2013 Ford Fusion / Escape• Engine fires– 13 reports ofengine fire– Short time frame• (~Sept - Dec)

• Owners asked to “park theirvehicles until further notice”• 99,153 brand new vehicles
affected*Images from:https://autos.yahoo.com/blogs/motoramic/ford-tells-89-000-escape-fusion-owners-park-230316605.html
http://gearheads.org/stop-driving-your-ford-escape/16The Problem• Ford press release:– “The original cooling system design was not able to address a lossof coolant system pressure under certain operating conditions,which could lead to a vehicle fire while the engine was running.”• Ford VP:– "We had a sequence of events that caused the cooling system
software to restrict coolant flow," he says. Most of the time, thatwould not be a problem and is the intended behavior. But in rarecases the coolant pressure coupled with other conditions may
cause the coolant to boil. When the coolant boils, the engine may
go into extreme overheating causing more boiling and rapid
pressure increase. This caused coolant leaks near the hot exhaustthat led to an engine fire.
– Ford has seen 12 fires in Escapes and one in a Fusion.

Quotes from:http://corporate.ford.com/news-center/press-releases-detail/pr-ford-produces-fix-in-voluntary-37491
http://www.usatoday.com/story/money/cars/2012/12/10/ford-recall-escape-fusion-ecoboost/1759063/

17

Quote• “Almost all software-related accidents can betraced back to flaws in the requirementsspecification”-- Prof. Nancy Leveson, MIT

These problems can pass every component andsubsystem test, every simulation, and every
verification effort!

Toyota Unintended Acceleration• 2004-2009: 102 incidents

22

Toyota Unintended Acceleration• 2004: Push-button ignition• 2004-2009– 102 incidents of uncontrolled acceleration– Speeds exceed 100 mph despite stomping on
the brake– 30 crashes– 20 injuries• Today– Software fixes for pushbutton ignition, pedals

Pushbutton was reliable, Software was reliable.
All component requirements were met.
Overall system unsafe, unexpected!
http://www.reuters.com/article/2010/07/14/us-toyota-idUSTRE66D0FR20100714
http://www.statesman.com/business/u-s-toyota-cite-driver-error-in-many-803504.html

23©

Toyota Unintended Acceleration• 2004: Push-button ignition• 2004-2009– 102 incidents of uncontrolled acceleration– Speeds exceed 100 mph despite stomping on
the brake– 30 crashes– 20 injuries• Today– Software fixes for pushbutton ignition, pedals

How can we be sure the requirements are right?
How can we integrate human and technicalconsiderations?
http://www.reuters.com/article/2010/07/14/us-toyota-idUSTRE66D0FR20100714
http://www.statesman.com/business/u-s-toyota-cite-driver-error-in-many-803504.html

24©

Honda Odyssey• 344,000 minivans recalled• Stability control software problem• In certain circumstances, an error in
the software can prevent the systemfrom calibrating correctly, leading topressure building up in the braking
system, the National Highway TrafficSafety Administration said.
• If pressure builds to a certain point,"the vehicle may suddenly andunexpectedly brake hard, andwithout illuminating the brake lights,
increasing the risk of a crash from
These problems made it throughbehind," the NHTSA said.
all existing processes: design
reviews, testing, etc.
• 2007-2008 models affected– Problem discovered in 2013

25

Addressing potential issues
ReactionHigh
Cost of Fix“Bolt-on”

Systems
Thinking

SystemRequirements

SystemsEngineeringLowConcept

Requirements Design

BuildOperate

Need to address issues earlyEarly decisions have biggest impact

Illustration courtesy Bill Young©

Human Interactions

China Airlines 006• Autopilot compensates for single engine malfunction• Autopilot reaches max limits, aircraft turns slightly• Pilots not notified Autopilot at its limits• Pilots notice slight turn, disengage autopilot for manual control• Aircraft immediately nosedives

Pilot error or“Clumsy automation”?

30©

Operator Error: Old View
• Human error is cause of most incidents and accidents• So do something about human involved• Fire them
• Retrain them• Admonish them

• Rigidify their work with more rules and procedures• Or do something about humans in general• Marginalize them by putting in more automation(Leveson)34

Operator Error: Systems View
(Dekker, Rasmussen, Leveson, Woods, etc.)• Human error is a symptom, not a cause• All behavior affected by context (system) in which itoccurs• To understand human error, look at the system• System designs can make human error inevitable• When bad systems cause operator error, can we really blame theoperators rather than designers?

• To do something about operator error, look at:• Unintuitive equipment and system designs• Usefulness of procedures• Existence of goal conflicts and production pressures

Human error is a symptom of the system and its design
35

Most stove tops

Is this a design problem or just human error?
36*Image from D. Norman, 1988©

Natural Mapping

The right design will reduce human error
37*Image from D. Norman, 1988©

Toyota Unintended Acceleration• 2004: Push-button ignition• 2004-2009• 102 incidents of uncontrolled acceleration• Speeds exceed 100 mph despite stomping on
the brake• 30 crashes• 20 injuries• Today• Software fixes for pushbutton ignition, pedals

Software design problem or driver error?
http://www.reuters.com/article/2010/07/14/us-toyota-idUSTRE66D0FR20100714
http://www.statesman.com/business/u-s-toyota-cite-driver-error-in-many-803504.html

40©

Tesla Summon
Tesla:• "the incident occurred asa result of the driver not
being properly
attentive...“• Drivers must agree tolegal terms on their touch
screen before the featureis allowed
This feature will park Model S while the driver is outside the vehicle. Please note that thevehicle may not detect certain obstacles, including those that are very narrow (e.g., bikes),
lower than the fascia, or hanging from the ceiling. As such, Summon requires that you
continually monitor your vehicle's movement and surroundings while it is in progress andthat you remain prepared to stop the vehicle at any time using your key fob or mobile app orby pressing any door handle. You must maintain control and responsibility for your vehicle
when using this feature and should only use it on private property. “OKCANCEL42©

STAMP:
System Theoretic
Accident Model and Processes
Foundation of STPA47

Systems approach to safety engineering(STAMP)

STAMP Model• Treat accidents as a control problem,
not a failure problem• Accidents are more than a chain ofevents, they involve complex dynamicprocesses.
• Prevent accidents by enforcing
constraints on component behavior
and interactions• Captures many causes of accidents:– Component failure accidents– Unsafe interactions among components– Complex human, software behavior– Design errors– Flawed requirements• esp. software-related accidents
48(Leveson, 2012)©

Basic STAMP• Controllers use a process model todetermine control actions
Controller• Unanticipated behavior often occurs
when the process model is incorrect

Process
ModelControl
Actions

Feedback

Controlled Process• Four types of inadequate control
actions:
1) Control commands are not given
2) Inadequate commands are given
3) Potentially correct commands but too
early, too late4) Control action stops too soon or applied
too long

Tends to be a good model of both software and human behavior
49
Explains software errors, human errors, interaction accidents,…
(Leveson,(Leveson, 2012)2012)©

Basic STAMP
Controller
Process
Model
Control
Actions

Feedback

Controlled Process

50(Leveson, 2012)Basic STAMP
Controller
Process
ModelControl
Actions

Feedback

Controlled Process

51(Leveson, 2012)Basic STAMP
Controller
Process
Model
Control
Actions

Feedback

Controlled Process

52(Leveson, 2012)

Example
Control
Structure(Leveson, 2012)

STAMP and STPA

STAMP Model

Accidents arecaused byinadequate control
54©

STAMP and STPA
CAST
Accident
Analysis

STAMP Model

How do we findinadequate control
that caused an
accident?
Accidents arecaused byinadequate control55©

STAMP and STPA
CAST
AccidentAnalysis

STPAHazard
Analysis

STAMP Model

How do we findinadequate control
in a design?
Accidents arecaused byinadequate control
56©

STPA:
System Theoretic Process Analysis

57

STPA
(System-Theoretic Process Analysis)• System engineeringfoundation
STPA Hazard
Analysis

STAMP Model(Leveson, 2012)– Define accidents,system hazards,– Control structure• Step 1: Identify
unsafe control
actions• Step 2: Identify
accident causal
scenarios

Controller
Control
Actions

Feedback

Controlledprocess58©

Definitions• Accident (Loss)– An undesired or unplanned event that results in a loss,including loss of human life or human injury, property
damage, environmental pollution, mission loss, etc.
• Hazard– A system state or set of conditions that, together with aparticular set of worst-case environment conditions, willlead to an accident (loss).

Definitions from Engineering a Safer WorldExample System: Aviation

System-level Accident (Loss): ?

Example System: Aviation

System-level Accident (Loss): Two aircraft collide

System-level Accident (Loss): Two aircraft collide
System-level Hazard: ?

System-level Accident (Loss): Aircraft crashes
System-level Hazard: Two aircraft violate minimum
separation

Aviation Examples• System-level Accident (loss)– A-1: Two aircraft collide– A-2: Aircraft crashes into terrain / ocean• System-level Hazards– H-1: Two aircraft violate minimum separation– H-2: Aircraft enters unsafe atmospheric region– H-3: Aircraft enters uncontrolled state– H-4: Aircraft enters unsafe attitude– H-5: Aircraft enters prohibited area

Example system: Automotive vehicles• Accidents
and Hazards?

Image: http://whitneylawgroup.com/wp-content/uploads/2013/07/52-car-pile-up.jpg

STPA(System-Theoretic Process Analysis)• System engineeringfoundation
STPA HazardAnalysis

STAMP Model(Leveson, 2012)– Define accidents,
system hazards– Control structure• Step 1: Identify
unsafe control
actions• Step 2: Identify
accident causal
scenarios

Controller
Control
Actions

Feedback

Controlled
process74©

Control Structure Examples

Example
ControlStructure(Leveson, 2012)

Proton Therapy Machine
High-level Control Structure

GantryCyclotron

Beam path andcontrol elements©

Proton Therapy Machine
High-level Control Structure

Antoine PhD Thesis, 2012©

Managing complexity• Lesson from systems theory, cognitive science• Human minds manage complexity throughabstraction and hierarchy• Use top-down process– Start at a high abstract level– Iterate to drill down into more detail– Build hierarchical models of the system©

Proton Therapy Machine
Control Structure

Antoine PhD Thesis, 2012

Ballistic Missile
Defense System

Image from:http://www.mda.mil/global/images/system/aegis/FTM21_Missile%201_Bulkhead%20Center14_BN4H0939.jpg

Safeware Corporation

Adaptive Cruise Control

Image from: http://www.audi.com/etc/medialib/ngw/efficiency/video_assets/fallback_videos.Par.0002.Image.jpg

Qi Hommes

Automotive shift-by-wire

89
Image: http://www.toyota.com.au/prius/features/hybrid-performance©

Automotive Shift by WireYour turn:Control structure?

90“Application of STPA to a Shift by Wire System”, STPA workshop 2014©

Control structure for vehicle
DriverSteering, brake,accelerator(engine),ignition, othercontrols

Range
control

Current
range
indication

Shift Control
ModuleStatus information
Visual cues
Sensory feedback

Range
commands

Physical Vehicle*Similar for both mechanical/electrical implementations©

Automotive Shift by Wire92“Application of STPA to a Shift by Wire System”, STPA workshop 2014

STPA(System-Theoretic Process Analysis)• System engineeringfoundation– Define accidents,
system hazards– Control structure• Step 1: Identify
unsafe controlactions• Step 2: Identify
accident causal
scenarios(Leveson, 2012)

Controller
Control
Actions

Feedback

Controlled
process93©

STPA Step 1: Unsafe Control Actions (UCA)
4 ways unsafe control may occur:

Controller
ControlActions• A control action required for safety is not provided or is notfollowed• An unsafe control action is provided that leads to a hazard

Feedback• A potentially safe control action provided too late, too early,or out of sequence

Controlled
process

ShifterCommand• A safe control action is stopped too soon or applied too long(for a continuous or non-discrete control action)

Not providing
causes hazard

Providingcauses hazard

Incorrect
Timing/Order?

?

?

Stopped TooSoon /Applied toolong?
©

Structure of an Unsafe Control
ActionExample:“Computer provides open catalyst valve cmd while water valve is closed”TypeSource Controller

Context
Control Action

Four parts of an unsafe control action– Source Controller: the controller that can provide the control action– Type: whether the control action was provided or not provided– Control Action: the controller’s command that was provided /missing– Context: conditions for the hazard to occur95• (system or environmental state in which command is provided)©

UCAs  Safety Constraints
Unsafe Control Action

Safety Constraint©

STPA(System-Theoretic Process Analysis)• System engineeringfoundation– Define accidents,
system hazards– Control structure• Step 1: Identify
unsafe controlactions• Step 2: Identify
accident causal
scenarios(leveson, 2012)

Controller
Control
Actions

Feedback

Controlledprocess97©

STPA Step 2: Identify Causal Factors• Select an Unsafe Control Action
A. Identify what might cause it to happen– Develop accident scenarios– Identify controls and mitigations

B. Identify how control actions may not be
followed or executed properly– Develop causal accident scenarios– Identify controls and mitigations©

Step 2A: Potential causes of UCAs
Control input orexternal information
wrong or missing

UCA: Shift Control
Module provides
range command
without driver new
range selection

Controller
Inadequate
Procedures(Flaws in creation,process changes,incorrect
modification oradaptation)

Missing or wrong
communication
with another
Controller
controller

ProcessModel(inconsistent,incomplete,or incorrect)

Inadequate or
missing feedback
Feedback Delays

Actuator

SensorInadequate
operation

Inadequate
operation
Incorrect or no
information provided

Delayedoperation

Controller
Conflicting control actions

Measurement
inaccuracies

Controlled Process
Component failures
Changes over time

Process input missing or wrong

Feedback delaysUnidentified or
out-of-range
disturbance

Process output
contributes tosystem hazard©

STPA Step 2: Identify Causal Factors• Select an Unsafe Control Action
A. Identify what might cause it to happen– Develop accident scenarios– Identify controls and mitigations

B. Identify how control actions may not be
followed or executed properly– Develop causal accident scenarios– Identify controls and mitigations©

Step 2B: Potential control actions not followedControl input or
external information
wrong or missingController

Shift Control
Moduleprovides rangecommand

Inadequate
Procedures(Flaws in creation,process changes,incorrect
modification oradaptation)

Missing or wrong
communication
with anotherController
controller

ProcessModel(inconsistent,incomplete,or incorrect)

Inadequate or
missing feedback
Feedback Delays

Actuator

SensorInadequate
operation
Delayedoperation

Controller
Conflicting control actions

Inadequate
operation

Range is notengagedControlled Process

Measurement
inaccuracies
Feedback delays

Component failures
Changes over time

Process input missing or wrong

Incorrect or noinformation provided

Unidentified or
out-of-range
disturbance

Process output
contributes tosystem hazard©