Runway Safety Metric (RSM)

Firdu Bati, AJI-3

March, 2017

RSM Stakeholders

Presented to:

By:

Date:

Federal Aviation

Administration

Federal Aviation

Administration

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

1

Background

The primary goal of safety metrics is to gauge the safety performance of the National Airspace System (NAS), at the system and facility levels. 

Existing runway safety metrics focus solely on runway incursion type counts, not accidents and excursions.

2

RSM Goal: Develop a runway safety metric that includes accident, Runway Excursion (RE), Runway Incursion (RI), and Surface Incident (SI) data.

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Approach

Collect applicable accident and incident data.

National Transportation Safety Board  (NTSB), RI-SI, and RE data sources 

Use modeling to assign risk weights to all kinds of outcomes.

Weighted outcomes include injuries, damage, incursion types, surface incidents, and excursions

Weights are based on outcomes’ “proximities” to fatality and are ordered based on SME input [Injury → Damage → Incident]

For accidents, weighting gives some credit for saving lives and minimally-damaged aircraft

Aggregate all event weights for each fiscal year to get a FY score.

3

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Weighting Schematic

4

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

5

Combined Risk & Event Count

*Risk decreased, incident increased, accident relatively constant over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

6

Commercial Risk & Event Count

*Risk decreased, incident increased, accident relatively constant over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

7

Non-Commercial Risk & Event Count

*Risk decreased, incident increased, accident relatively constant over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

8

Commercial Risk & Ops (Per Million)

*Risk decreased, Ops slightly decreased over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

9

Non-Commercial Risk & Ops (Per Million)

*Risk increased and started decreasing, Ops decreased over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Conclusion

Systematic approach followed to develop Runway Safety metric

Inclusive of all relevant datasets

Quantitative technique to assign severity weights

Baseline/target is in development

Expected deployment date: FY 19

10

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Backup Slides

11

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Technical Approach

Prepare National Transportation Safety Board (NTSB), RI-SI Database, and RE Database data.

Assume the worst possible accident involves fatal injury.

Assign weights to accidents and incidents based on relative “distance” to fatality.

12

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Data Pre-Processing

NTSB data selection

Reports without phase codes relevant to the runway environment were removed

Remaining data classified as runway collision, taxiway collision, runway excursion by training a text mining model and validating results

Data merging

RE and RI-SI data have already been reviewed by domain expects and categorized

Some overlap between RE and NTSB datasets; duplicates were removed (NTSB records retained)

13

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

FY16 Dataset

14
|             | RunwayCollision | TaxiwayCollision | Runway Excursions |
|-------------|-----------------|------------------|-------------------|
| # of Events | 237             | 302              | 9,653             |
|             | A   | B   | C     | D     | E  | SI    |
|-------------|-----|-----|-------|-------|----|-------|
| # of Events | 235 | 319 | 5,472 | 7,967 | 12 | 8,936 |
|             | Runway Excursions |
|-------------|-------------------|
| # of Events | 1,276             |

NTSB (FY81 – FY16)

RI-SI (FY98 – FY16)

RE (FY12 – FY16)

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Event Count

15

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Three Step Weighting Process

Weights are assigned to types of accidents based on proximity to a fatal accident.

Appropriate domain expert assumptions are used to reorder the weights:

Injury

Damage

Incident without injury/damage

Incidents are assigned weights based on their corresponding accident types.

16

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Information Gain as a Measure of Uncertainty

17

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Information Gain as a Measure of Uncertainty

Bayesian Network employed

Accounts for correlation between different outcomes

Computes information gain

18

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

SME-Based Reordering Logic

Model output ranks some damage higher than injury.

Domain experts provided qualitative direction resulting in reordering.

Injury

Damage

Incident

19

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

SME-Based Reordering Logic

Compute information gain (model output)

Compute relative significance of highest gain

Normalize it so the relative gain sums to 1

Order gain using cumulative significance

Compute relative significance to new highest gain

Shift significance by maximum gain to compute relative significance to fatality 

20

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Entropy-Based Relative Weight

21

Commercial

Non-Commercial

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Weights – Penalty & Credit

Injuries

Penalty and credit terms for injured and non-injured, respectively

Damage

Penalty and credit terms for damaged and non-damaged, respectively

Incidents

Ratio of the weight of the corresponding accident type to frequency of incident

22

| , | 

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Credit – Penalty Correction Term

Example Two accidents 

A: 5/5 injured

B: 5/200 injured

Without credit term,

Both A & B: 5 * .75 = 3.75

With credit,

A: (5*.75) = 3.75

B; (5*.75) – (.975*.75) = 3.02

23

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Relative Weights – Commercial

24

Note: Y-Axis uses Logarithmic scale 

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Relative Weights – Non-Commercial

25

Note: Y-Axis uses Logarithmic scale 

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Application of Weights

Two potential applications: 

Aggregation: sum of all undesired outcome weights

Worst-outcome: weight of the worst outcome

*The choice makes little difference in the relative risk profile primarily due to the limiting effect of the worst-outcome [adding small numbers to a large number]

26

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

27

Combined Risk & Event Count

*Risk decreased, incident increased, accident relatively constant over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

28

Commercial Risk & Event Count

*Risk decreased, incident increased, accident relatively constant over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

29

Non-Commercial Risk & Event Count

*Risk decreased, incident increased, accident relatively constant over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

30

Commercial Risk & Ops (Per Million)

*Risk decreased, Ops slightly decreased over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

31

Non-Commercial Risk & Ops (Per Million)

*Risk increased and started decreasing, Ops decreased over time

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Assumptions

32

Assumed the following hierarchy of accident/incident severity based on domain expert input: fatality, injury, aircraft damage, incidents.

Incident to accident mapping assumption

RIs  Runway Collision

SIs  Taxiway Collision

RE Incidents  RE Accident

Assumed the following criteria should be used to split the dataset by Commercial and Non-Commercial

Part 121, 129, 135: Commercial

Others: General Aviation / Non-Commercial

Incident data are only available starting in 1997; however, the model used accident data since 1982.

This is more conservative

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

Conclusion

Systematic approach followed to develop Runway Safety metric

Inclusive of all relevant datasets

Quantitative technique to assign severity weights

Baseline/target is in development

33

Federal Aviation

Administration

<Presentation Title – Change on Master Slide>

<Date of Presentation – Change on Master Slide>

# image2.jpeg

# image1.wmf

# image3.png

# image4.png

# image5.png

# image6.png

# image7.png

# image8.png

# image9.png

# image10.png

# image50.png

# image60.png

# image70.png

# image80.png

# image11.png

# image12.png

# image13.png

# image120.png

# image130.png

# image14.png

# image15.png

# image16.png

# image17.png