May 2015 

NASA/TM–2015-218705 

Development and Execution of the RUNSAFE 

Runway Safety Bayesian Belief Network Model 

Lawrence L. Green 

Langley Research Center, Hampton, Virginia 
NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the 

advancement of aeronautics and space science. The 

NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

The NASA STI program operates under the auspices 

of the Agency Chief Information Officer. It collects, 

organizes, provides for archiving, and disseminates 

NASA’s STI. The NASA STI program provides access 

to the NTRS Registered and its public interface, the 

NASA Technical Reports Server, thus providing one 

of the largest collections of aeronautical and space 

science STI in the world. Results are published in both 

non-NASA channels and by NASA in the NASA STI 

Report Series, which includes the following report 

types: 

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase of 

research that present the results of NASA 

Programs and include extensive data or theoretical 

analysis. Includes compilations of significant 

scientific and technical data and information 

deemed to be of continuing reference value. 

NASA counter-part of peer-reviewed formal 

professional papers but has less stringent 

limitations on manuscript length and extent of 

graphic presentations. 

• TECHNICAL MEMORANDUM. 

Scientific and technical findings that are 

preliminary or of specialized interest, 

e.g., quick release reports, working 

papers, and bibliographies that contain minimal 

annotation. Does not contain extensive analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. 

Collected papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or 

co-sponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest. 

• TECHNICAL TRANSLATION. 

English-language translations of foreign 

scientific and technical material pertinent to 

NASA’s mission. 

Specialized services also include organizing 

and publishing research results, distributing 

specialized research announcements and feeds, 

providing information desk and personal search 

support, and enabling data exchange services. 

For more information about the NASA STI program, 

see the following: 

• Access the NASA STI program home page at 

http://www.sti.nasa.gov 

• E-mail your question to help@sti.nasa.gov 

• Phone the NASA STI Information Desk at 

757-864-9658 

• Write to: 

NASA STI Information Desk 

Mail Stop 148 

NASA Langley Research Center 

Hampton, VA 23681-2199 
National Aeronautics and 

Space Administration 

Langley Research Center 

Hampton, Virginia 23681-2199 

 

May 2015 

NASA/TM–2015-218705 

Development and Execution of the RUNSAFE 

Runway Safety Bayesian Belief Network Model 

Lawrence L. Green 

Langley Research Center, Hampton, Virginia 
Available from: 

NASA STI Program / Mail Stop 148 

NASA Langley Research Center 

Hampton, VA 23681-2199 

Fax: 757-864-6500 

The use of trademarks or names of manufacturers in this report is for accu rate reporting and does not constitute an 

official endorsement, either expressed or implied, of such products or manufacturers by the National Aeronautics and 

Space Administration. 
i 

Table of Contents 

1.0 List of Figures ........................................................................................................................ii

2.0 List of Tables .........................................................................................................................ii

3.0 Nomenclature .......................................................................................................................v

4.0 Abstract .............................................................................................................................. 1

5.0 Introduction......................................................................................................................... 2

5.1 Runway Incursion Events.................................................................................................... 3

5.2 Runway Excursion Events ................................................................................................... 7

5.3 Bayesian Belief Network Models ......................................................................................... 9

5.4 Methodology and Software ...............................................................................................11

6.0 Model Development ............................................................................................................13

6.1 Runway Incursion Model ...................................................................................................13

6.2 Runway Excursion Model ..................................................................................................21

7.0 Model Population ................................................................................................................24

7.1 Runway Incursion Model ...................................................................................................24

7.2 Runway Excursion Model ..................................................................................................43

8.0 Model Baseline Execution .....................................................................................................53

8.1 Runway Incursion Model ...................................................................................................53

8.2 Runway Excursion Model ..................................................................................................55

9.0 Model Sensitivity Analysis .....................................................................................................56

9.1 Runway Incursion Model ...................................................................................................56

9.2 Runway Excursion Model ..................................................................................................57

10.0 Conclusions.......................................................................................................................58

11.0 Recommendations .............................................................................................................59

12.0 References List ..................................................................................................................59
ii

1.0 List of Figures 

Figure 1. Runway Incursion Timeline Sketch.........................................................4

Figure 2. The Runway Incursion Bayesian Belief Network.......................................17

Figure 3. The Runway Excursion Bayesian Belief Network.......................................22

Figure 4. Sample RI Model Execution (Baseline Scenario).......................................54

Figure 5. Sample RI Model Execution (Worst-Case Scenario)....................................55 

Figure 6. Sample RE Model Execution (Average Probabilities)..................................56

Figure 7. Sample RI Model Execution Sensitivity Analysis.......................................57

Figure 8. Sample RE Model Execution Sensitivity Analysis......................................58

 

2.0 List of Tables 

Table 1. SME Probabilities and Confidence Bounds for the Node Airport Layout............24

Table 2. SME Probabilities and Confidence Bounds for the Node Signs, Markings and 

Equipment................................................................................................25 

Table 3. SME Probabilities and Confidence Bounds for the Node Airport Construction or 

RW/TW Closure..........................................................................................26 

Table 4. SME Probabilities and Confidence Bounds for the Node Contamination 

Control....................................................................................................26

Table 5. Conditional Probability Table for a Two-Factor Binary “Or” Node..................27

Table 6. Conditional Probability Table for the Node Airport Issues.............................27

Table 7. Conditional Probability Table Based on Rank Ordering of Contributing 

Factors.....................................................................................................29

Table 8. SME Probabilities and Confidence Bounds for the Node ATC Cert Training 

Issues......................................................................................................29 

Table 9. SME Probabilities and Confidence Bounds for the Node ATC OJT Issues...........29 

Table 10. SME Probabilities and Confidence Bounds for the Node ATC Mental or Physical 

State Issues...............................................................................................30 
iii

Table 11. Conditional Probability Table for the Node ATC HFACS Issues....................30

Table 12. SME Probabilities and Confidence Bounds for the Node Comm Content 

Issues......................................................................................................31 

Table 13. SME Probabilities and Confidence Bounds for the Node Comm Hardware 

Issues......................................................................................................31 

Table 14. Conditional Probability Table for the Node Two-Party Comm Issues...............31 

Table 15. SME Probabilities and Confidence Bounds for the Node Pilot Cert Training 

Issues......................................................................................................32

Table 16. SME Probabilities and Confidence Bounds for the Node Pilot OJT Issues.........32

Table 17. SME Probabilities and Confidence Bounds for the Node Pilot Mental or Physical 

State Issues...............................................................................................32 

Table 18. Conditional Probability Table for the Node Pilot HFACS Issues....................33

Table 19. SME Probabilities and Confidence Bounds for the Node Automation Interaction 

Issues (Pilot).............................................................................................33

Table 20. SME Probabilities and Confidence Bounds for the Node Inappropriate Aircraft 

Operations................................................................................................34

Table 21. SME Probabilities and Confidence Bounds for the Node Automation Interaction 

Issues (ATC).............................................................................................34

Table 22. SME Probabilities and Confidence Bounds for the Node Abnormal Air Traffic 

Volume or Complexity.................................................................................34

Table 23. SME Probabilities and Confidence Bounds for the Node Staffing or Procedural 

Issues......................................................................................................35

Table 24. Transposed Conditional Probability Table for the Node ATC Operational 

Issues......................................................................................................36 

Table 25. Transposed Conditional Probability Table for the Node Pilot Operational 

Issues......................................................................................................38

Table 26. SME Probabilities and Confidence Bounds for the Node Driver Training.........39

Table 27. Conditional Probability Table for the Node Driver Operational Issues.............39 

Table 28. SME Probabilities and Confidence Bounds for the Node Primary Error State.....40
iv

Table 29. SME Probabilities and Confidence Bounds for the Node Collision Scenarios.....41 

Table 30. SME Probabilities and Confidence Bounds for the Node Reaction Time...........41 

Table 31. Transposed Conditional Probability Table for the Node Potential RI Event 

Severity...................................................................................................42 

Table 32. SME Probabilities and Confidence Bounds for the Node Approach and Departure 

Constraints...............................................................................................43 

Table 33. SME Probabilities and Confidence Bounds for the Node Contamination 

Control....................................................................................................43

Table 34. SME Probabilities and Confidence Bounds for the Node Runway Length.........49

Table 35. Conditional Probability Table for the Node Airport Issues............................44 

Table 36. SME Probabilities and Confidence Bounds for the Node Runway Assignment...44

Table 37. SME Probabilities and Confidence Bounds for the Node Runway Collision 

Avoidance................................................................................................44 

Table 38. SME Probabilities and Confidence Bounds for the Node Contribution to 

Unstabilized Approach.................................................................................45 

Table 39. SME Probabilities and Confidence Bounds for the Node Lack of Current Weather 

Info........................................................................................................45

Table 40. Transposed Conditional Probability Table for the Node ATC Operational 

Issues.....................................................................................................45

Table 41. SME Probabilities and Confidence Bounds for the Node Unstabilized 

Approach.................................................................................................46 

Table 42. SME Probabilities and Confidence Bounds for the Node Aircraft Automation 

Issues......................................................................................................47 

Table 43. SME Probabilities and Confidence Bounds for the Node Inappropriate Aircraft 

Operations................................................................................................47

Table 44. Transposed Conditional Probability Table for the Node Pilot Operational 

Issues......................................................................................................48

Table 45. SME Multiplicative Factors and Confidence Bounds for HFACS Contributions to 

RE Events Relative to RI Events.....................................................................49

Table 46. RE / ATC / HFACS Contributions.......................................................49 
v 

Table 47. RE / Pilot / HFACS Contributions........................................................49

Table 48. Transposed Conditional Probability Table for the Node Pilot Error.................50 

Table 49. SME Probabilities and Confidence Bounds for the Node Mechanical Failure.....51

Table 50. SME Probabilities and Confidence Bounds for the Node RE Event Initiated.......52 

Table 51. SME Probabilities and Confidence Bounds for the Node Weather Issues..........53

3.0 Nomenclature 

AC Aircraft 

ANOVA Analysis of Variance 

AOSP Aviation Operations and Safety Program 

App Approach 

ARMD Aeronautics Research Mission Directorate 

ASIAS FAA Safety Information Analysis and Sharing 

ATADS FAA Air Traffic Activity Data System 

ATC Air Traffic Control 

Avg Average Value 

AvSP Aviation Safety Program 

BBN Bayesian Belief Network 

Cat RI Event Severity Category 

Cert Certification 

Comm Communication 

Cont Contamination Control 

Cntrb Contribution 

CPT Conditional Probability Table 
vi

Curr Current 

Dep Departure 

DX8 Design-Expert version 8 

DX9 Design-Expert version 9 

FAA Federal Aviation Administration 

HFACS Human Factors Analysis and Classification System 

Hi high bound for SME values 

LaRC Langley Research Center 

Lo low bound for SME values 

Max Maximum value 

Mech Mechanical 

Mfac Multiplicative Factor 

Min Minimum value 

NASA National Aeronautics and Space Administration 

NextGEN Next Generation Air Transportation System 

OJT On-the-Job-Training 

PED Pedestrian 

RE Runway Excursion 

RI Runway Incursion 

RUNSAFE Combined RI and RE event model 

RW Runway 

SME Subject Matter Expert 

StDev Standard Deviation 

TC Technical Challenge 
vii 

TW Taxiway 

Unstab Unstabilized 

VEH Vehicle 
1 

4.0 Abstract 

One focus area of the National Aeronautics and Space Administration (NASA) is to improve 

aviation safety. Runway safety is one such thrust of investigation and research. The two primary 

components of this runway safety research are in runway incursion (RI) and runway excursion 

(RE) events. These are adverse ground-based aviation incidents that endanger crew, passengers, 

aircraft and perhaps other nearby people or property. A runway incursion is the incorrect presence 

of an aircraft, vehicle or person on the protected area of a surface designated for the landing and 

take-off of aircraft; one class of RI events simultaneously involves two aircraft, such as one aircraft 

incorrectly landing on a runway while another aircraft is taking off from the same runway. A 

runway excursion is an incident involving only a single aircraft defined as a veer-off or overrun 

off the runway surface. 

Within the scope of this effort at NASA Langley Research Center (LaRC), generic RI, RE and 

combined (RI plus RE, or RUNSAFE) event models have each been developed and implemented 

as a Bayesian Belief Network (BBN). Descriptions of runway safety issues from the literature 

searches have been used to develop the BBN models. Numerous considerations surrounding the 

process of developing the event models have been documented in this report. The event models 

were then thoroughly reviewed by a Subject Matter Expert (SME) panel through multiple 

knowledge elicitation sessions. Numerous improvements to the model structure (definitions, node 

names, node states and the connecting link topology) were made by the SME panel. Sample 

executions of the final RUNSAFE model have been presented herein for baseline and worst-case 

scenarios. Finally, a parameter sensitivity analysis for a given scenario was performed to show the 

risk drivers. 

The NASA and LaRC research in runway safety event modeling through the use of BBN 

technology is important for several reasons. These include: 1) providing a means to clearly 

understand the cause and effect patterns leading to safety issues, incidents and accidents, 2) 

enabling the prioritization of specialty areas needing more attention to improve aviation safety, 

and 3) enabling the identification of gaps within NASA’s Aviation Safety funding portfolio. 
2 

5.0 Introduction 

One focus area of the National Aeronautics and Space Administration (NASA), enabled 

through the former Aviation Safety Program (AvSP), now the Airspace Operations and Safety 

Program (AOSP)1, of the NASA Aeronautics Research Mission Directorate (ARMD), and in 

cooperation with the Federal Aviation Administration (FAA), is to improve aviation safety. 

Specifically, this Program seeks to provide increasing capabilities to: 

• predict and prevent safety issues; 

• monitor for safety issues in-flight and lessen their impact should they occur; 

• analyze and design safety issues out of complex system behaviors; 

• analyze designs and operational data for potential hazards. 

The AvSP / AOSP explores hardware and software systems (technologies or products) that 

will operate in the Next Generation Air Transportation System (NextGEN)2. Runway safety is one 

thrust of investigation and research. The two primary components of runway safety are runway 

incursion (RI) and runway excursion (RE) events. 

A runway incursion is the incorrect presence of an aircraft, vehicle or person on the protected 

area of a surface designated for the landing and take-off of aircraft (as defined by the FAA Office 

of Runway Safety)3-4. A runway excursion is an incident involving only a single aircraft defined 

as a veer-off or overrun off the runway surface4. In short, RI and RE events are adverse groundbased aviation incidents that endanger crew, passengers, aircraft and perhaps other nearby people 

or property. Additional detail about RI and RE events is provided in subsequent sections of this 

document. 

Within the scope of this effort, statistical RI framework5, and generic RI, RE and combined 

(RI plus RE, or RUNSAFE) event models6-7 have each been developed and implemented as a 

Bayesian Belief Network (BBN)8-10. Data from the FAA and descriptions of issues from the 

literature searches, described subsequently, have been used to develop the BBN models. The 

development of the RUNSAFE (combined RI and RE) BBN model7, and some sample executions, 

are documented later in this report. Other similar BBN modeling efforts have been recently 

documented within a group working at NASA Langley Research Center (LaRC)8, 11-13

. More 

discussion about BBN models also follows subsequently. 
3 

The NASA Aviation Safety Program conducts cutting-edge research to produce innovative 

concepts, tools, and technologies that can improve the intrinsic safety attributes of current and 

future aircraft. The AvSP research centers have defined a set of Technical Challenge (TC)1 issues 

that are aligned with program goals and project objectives. These TC issues serve to focus research 

toward solving aviation safety problems and provide a consistent framework to focus, direct, plan, 

execute, manage, and communicate Center-distributed research. Among the TC issues relevant to 

this work are: 

• Assurance of Flight Critical Systems (air traffic control operations) 

• Discovery of Precursors to Safety Issues 

• Assuring Safe Human-Systems Integration 

• Improve Crew Decision-Making and Response in Complex Situations 

The NASA and LaRC research in event modeling through the use of BBN technology is 

important for several reasons. These reasons include: 1) a means to provide a clear understanding 

of the cause and effect patterns leading to safety issues, incidents and accidents, 2) to enable the 

prioritization of specialty areas needing more attention to improve aviation safety, and 3) to enable 

the identification of gaps within NASA’s Aviation Safety funding portfolio. 

5.1 Runway Incursion Events 

 Again, a runway incursion is the incorrect presence of an aircraft (AC), vehicle (VEH) or 

person (pedestrian or PED) on the protected area of a surface designated for the landing and takeoff of aircraft (as defined by the FAA Office of Runway Safety3-4). Generally, RI events are 

reported by air traffic control (ATC) personnel (this phrase is intended to include those personnel 

that may be controlling only ground traffic at airports) in one category (Cat) among several severity 

categories, originally defined as: 

• Cat A = an accident or near miss occurred

• Cat B = significant potential for collision existed

• Cat C = ample time and/or distance existed to avoid a collision 

• Cat D = an RI event with no immediate safety consequences

• Other = RI events that have not yet been properly classified
4 

 The FAA has recently revised these severity categories to be: Accident, Cat A (Near Miss), Cat 

B, Cat C, Cat D and Cat E (Other) as noted above3-4. Once initiated, RI events are short in duration 

and timing is critical; typical landing and takeoff times are 20 to 30 seconds and the event severity 

can easily escalate with just slightly different timing. 

 As shown in the timeline of Figure 1, an RI event consists of a development phase (blue box) 

and a mitigation phase (red). During the development phase, various circumstances (bad weather, 

poor airport layout, confusing communication with the ATC, etc.) contribute to the occurrence and 

possible severity of a future RI event; the development phase may take place over minutes or 

hours. During the mitigation phase, various actions may take place that can reduce the severity of 

the RI event; this phase may only be a few seconds. Three intermediate instances in time as marked 

in Fig. 1 are also relevant: 1) at some point in time, one or more of the people involved may realize 

that an RI is imminent but has not actually occurred yet, 2) at some point in time, due to the physics 

of the situation, the RI event becomes unavoidable, and 3) at some point in time, the RI event is 

actually initiated. Under some circumstances, the three instances in time (indicated as 1, 2 and 3 

in Figure 1) coalesce and there is no overlap between the development and mitigation phases. 

Figure 1. Runway Incursion timeline sketch. 

 One can imagine a situation where a pilot, on final approach for a landing, observes a VEH 

being driven at high speed toward the runway on which their aircraft has been cleared to land. An 

RI event has not yet been initiated but the pilot suspects that an RI event will occur. At this point 

(instance 1 in Figure 1), the pilot has ample time to initiate a go-around (whether self-initiated or 

directed by the ATC); the VEH may stop before entering the runway, and the RI event may never 

technically occur. Under slightly different circumstances, the pilot may only realize that an RI 

event will occur after it is too late to be avoided (i.e., the VEH will enter the runway without 
5 

authorization, instance 2 in Figure 1) and begin to initiate a go-around that results in a near miss 

with the VEH; without the go-around attempted, this event may have resulted an AC / VEH 

collision. Again, under slightly different circumstances, the pilot may only realize that an RI event 

has occurred after the VEH enters the runway without authorization (RI event initiated, instance 3 

in Figure 1); any attempted mitigation at this point could still result in an AC / VEH collision, a 

near miss, or the AC may have an accident on its own attempting to avoid the VEH. There are 

many complex situations to consider in this context. 

 In reviewing the literature on this topic, a recent NASA study on non-towered airports14

indicated that the number of RI events is increasing with time, with about half of the events being 

of low severity and the remainder being split among moderate, high, and severe RI events; among 

these events, intersecting runaways are noted as the highest contributing factor. A 2013 

presentation by the Boeing Company15 shows that flight hours, departures, and the size of the 

worldwide fleet have generally increased, while accident rates have remained essentially flat (but 

at a very low level) over the last 20 years; the same presentation points to about 61% of all fatal 

accidents and about 50% of onboard fatalities worldwide being associated with final approach, 

landing, takeoff and initial climb during the period 2004 through 2013. A recent U.S. Department 

of Transportation, Volpe Center16 report shows that the spacing of parallel runways has just a small 

effect (if any) on the number of RI events across all severity categories; the same reports illustrates 

that crossing the hold short line, entering the runway and crossing a runway as the most likely 

types of RI events. A recent journal article17 illustrates a dramatic increase in the number of RI 

reported in 2008 compared to previous years, with pilot deviations always being the largest source 

of these events. A recent FAA report18 described the strong correlation among airport geometry, 

complexity and various communication tools (including signage and runway markings) with RI 

events. A Pilots Association report19 illustrates an increase in RI events with air traffic, but with 

overall the RI event rate being less than 6 per million operations; this reports also points to major 

domestic airports (Chicago, Atlanta, Dallas/Fort Worth, Los Angeles, St. Louis and Philadelphia) 

as having the greatest number of RI events.

 With the goal of improving runway safety, a statistical analysis5 of the Runway Incursion (RI) 

Database20 from the FAA Safety Information Analysis and Sharing (ASIAS) website 21 and the 

FAA Air Traffic Activity Data System (ATADS)22, also from the FAA, was conducted to ascertain 

its relevance to the top ten challenges of AvSP. The information contained in the RI database was 
6 

found to contain data that may be relevant to several of the AvSP top ten challenges1 including: 1) 

the assurance of flight critical systems [i.e., airport operations], (2) the discovery of precursors to 

safety issues, and 3) improve crew decision-making and response in complex situations. Prior 

conference papers by this author fully documented the statistical analysis of RI data5 and the initial 

development of a BBN model for RI events6. A subsequent conference paper7 extended this BBN 

model to also include RE events; these three reports together serve as the basis for much of what 

is reported in this NASA TM.

 Some of the important findings of the statistical analysis5 were that: 

 while the number of RI events was found to be increasing over time, there has been 

essentially no change in the number of higher severity RI events over the time period 

examined (2001-2011)

 the assumed risk level was found to be increasing over time 

 part of the observed increase in the number of RI Events and the assumed risk level can be 

attributed to an RI definition change by the FAA between 2007 and 2008, described in a 

2010 paper by Chapman23

 Chapman also notes that pilots consistently rated RI events at higher severity than the FAA 

controllers that typically report the RI events23

 while a few airports, such as Winston / Salem, NC and Fort Wayne, IN had a number of 

RI events well above the mean value for all the airports considered, those airports with 

large traffic volume, such as O’Hare, Chicago, IL and Atlanta, GA clearly stand out with 

statistically significant high average risk sums 

 when the average risk sum is normalized by the air traffic volume associated with each 

airport, other smaller airports stand out with statistically significant high risk levels, 

meaning that a flyer’s actual risk of being involved in a RI event at high traffic volume 

airports may actually be significantly lower than the risk at other smaller volume airports 

 by far the most prevalent cause of RI events is pilot error (about 72% of all RI events) 

 an unauthorized person or vehicle account for about 19% of the RI events 

 among the pilot errors, two contributing factors were readily identified as major 

contributors: accidental use of the wrong runway or taxiway (about 25% of pilot errors), 
7 

and confusion about the extent of authority granted to the pilot at a specific time by the air 

/ ground / local traffic control personnel (about 20% of the pilot errors) 

 among weather factors, adverse lighting conditions may have the greatest overall and the 

most consistent contribution to severe runway events 

 snow (and other freezing conditions) are less of a potential contributor to runway events 

(only 2% to 9% of RI events cite these conditions) than wind, rain, visibility and lighting 

 surprisingly, only about 16% of the RI events examined included some form of 

intervention, where a corrective or mitigative action was taken 

 when an intervention or mitigation occurred, these actions were successful in reducing the 

RI event severity about 70% of the time 

 a “go-around” issued to incoming planes was the most common form of intervention 

5.2 Runway Excursion Events 

 The RE event rate is quoted in several references: about 1 to 2 per million flights for the period 

1990 through 200624, up to 16 accidents and incidents per year during the period 1978 to 2008, 

and 30 runway excursion accidents per year25. The approach and landing phases of flight have 

shown little improvement in safety over the last decade (up to 2008) and RE events are the third 

greatest source of aircraft crashes behind in flight loss of control and controlled flight into terrain26; 

according to this source, the frequency of runway incursions is about half that of runway 

excursions, which may amount to 10 to 20 overruns and veer-offs each year26; the data used in this 

report suggests that severe RI (Category A / B) are together about as common as RE. If the reader 

is concerned that the references cited here are out of date, prior versions of the 2013 Boeing 

report15, dating back to 2010 (and considering data back to 2004) show almost identical numbers 

for RI end RE events over a broad number of years. 

 According to one source24, landing RE events are the most common, representing about 77% 

of all RE events. Some contributing factors are shared across the various types of RE events. The 

most common contributing factor associated with landing overruns is wet/contaminated runways, 

with long landings being the second most common contributor. Several other contributing factors 

are also noted (incorrect decision to land, speed to high, late/incorrect use of brakes, late/incorrect 

use of thrust reversers, aquaplaning, tailwind and being too high on approach). For landing veeroffs, the most common contributing factors include crosswind, nose wheel steering problems, 
8 

collapsed landing gear, hard landing, tire failures and asymmetric power. The most common 

contributing factors to takeoff overruns include late abort/reject and an inaccurate estimation for 

takeoff mass. The most common contributing factors for takeoff veer-offs include inadequate 

supervision of the flight24. 

 Other additional causal and contributing factors for RE events exist, which are summarized 

here and noted again later in this report where they are particularly relevant. While the direct role 

of air traffic control (ATC) personnel in runway excursions was relatively small24, the ATC 

personnel may contribute to RE events by providing poor, complex or incomplete instructions. 

Several other contributing factors, including communication/coordination/planning, poor decision 

making processes about landing or takeoff under adverse circumstances and approach/takeoff 

procedures are important27. One contributing factor not previously mentioned is the inconsistent 

reporting of runway conditions and braking action at airports across the world26. Numerous 

challenges exist for improving runway safety for existing airports28 because, due to fixed runway 

layouts and surrounding populated areas, these facilities may lack the flexibility to implement 

recommended runway safety mitigation strategies. While contaminated runways (ice, snow, slush, 

wet or flooded) are a significant contributor to RE events, almost 47% of RE events occurred on 

dry runways29. Takeoff runway excursions were likewise predisposed by a number of factors30. 

 Some other reports discuss other model development and application efforts, also aimed at 

improving runway safety. For instance, one reference describes an analysis tool to quantify risk, 

support planning, and engineering decisions when determining runway safety area requirements 

for various types and sizes of airports27. The FAA is developing integrated risk models to forecast 

the risk and assess the impact of additional control measures at specific airports based on traffic 

volumes, complexity, and environmental factors31. Another study is taking a more novel and 

holistic approach to make sure that resources spent by airports to improve runway safety are 

actually used to address the most common types of RE events32. Yet another study employed 

human-in-the-loop simulations to evaluate traffic capacity at the Los Angeles International 

Airport33. An automated risk rating model for RI events was presented in another report34

. Another 

document by the same author explores what is known about the human errors and other factors 

that have been identified as contributing to runway incursions, and offers some error mitigation 

strategies16. An example of a BBN devoted to RI events is given in another source35. 
9 

5.3 Bayesian Belief Network Models 

 The modeling philosophy includes the use of a generic, high-level, system-integrated modeling 

with a systems level risk-based causal model. It should capture the multi-dependenc ies 

(interactions) of causal and contributing factors from various problem domains. However, the 

modeling should not be a representation of a specific accident/incident case, nor a detailed 

simulation analysis. 

 In general, the modeling steps undertaken include: 1) determining the causalities and cause-toeffect relations based on the historical risks and anticipated future risks from safety data/database 

and literature reviews, 2) constructing a baseline risk-based causal model as a BBN, 3) conducting 

Subject Matter Experts (SME) Knowledge Elicitation sessions to review the baseline model 

structure and to elicit the Conditional Probability Table (CPT) values for the baseline model 

without product insertions, and 4) inserting the NASA safety technologies/products into the model 

and eliciting CPT values with products included. The expected modeling results include 1) a 

quantification of the relative likelihood of concerned aviation risks, with technology products 

inserted and without, 2) an assessment of the direct risk mitigation effectiveness of the NASA 

safety technologies/products, 3) a portfolio gap analysis and 4) a sensitivity analysis for risk 

drivers. 

Several recent BBN modeling efforts6-7,9-13 have been undertaken to support the AvSP portfolio 

assessments and to determine if the AvSP technologies are addressing/mitigating aviation safety 

problems. The characteristics of issues selected for modeling are: 

• A significant accident category based on the historical data and/or future trend 

• Alignment with the focus and research areas of AvSP 

• Broad coverage on AvSP safety technology products 

• Many underlying causal/contributing factors that lead to aviation accidents 

• Suitability for a high-level system analysis and modeling 

 A typical BBN consists of the model structure and the model content. The model structure 

consists of a set of relevant definitions, as well as the node names, the node states, the ordering of 

the defined states for each node to facilitate SME comment, the connecting link topology and the 

connecting link priority as they enter specific nodes (again, to facilitate SME comment). The 

model content consists of the sets of marginal and Conditional Probability Table values. During 
10

the first phase of a typical BBN development cycle, NASA researchers develop (based upon 

database and literature search) and propose a model structure to an SME panel; the development 

step may take months to complete. Then, the various elements of the proposed structure are 

reviewed, modified and validated by the SME panel. Once the model structure has been agreed 

upon and validated by the SME panel, a CPT elicitation process (model population) is conducted 

by a facilitator on behalf of NASA to determine the appropriate model content. Some portions of 

the model review, modification, validation and population can be conducted in parallel. Once the 

model has been populated, it is executed to obtain a set of baseline results and a baseline sensitivity 

analysis. 

Although both RI and RE events may involve numerous contributing factors (e.g., airport 

layout, airport operations, weather, training and mechanical failures), RI events are more complex 

than RE events. RI events are “people intensive”, possibly involving multiple pilots, controllers, 

airport employees or contractors and perhaps other participants. There are also organizations 

(FAA, Airport Management, Aircraft Operators, etc.) that support each of the people directly 

involved in the runway safety events. RI events are also “communication (Comm) intensive”; 

several instances of two-party communications must simultaneously function properly in order to 

avoid problems. Two-party communications involve both the content and hardware transmission 

of information. Instances of two-party communications exist between all the people involved in 

the event (e.g., pilot to pilot, pilot to controller, controller to controller, controller to airport 

personnel). All runway safety events are short in duration and timing is critical; typical landing or 

takeoff reaction times are about 20 seconds (or less) and the event severity can easily escalate with 

just slightly different timing. The RI and RE models could be joined together through a set of 

common definitions for accident and incidents, based upon the level of aircraft damage and 

passenger injuries36 but this was deemed out of scope for the current models. 

An SME panel consisting of four consultants was assembled to review the model structure and 

to populate its content. The SME panel included two pilots and two other aviation expert 

consultants. The preliminary BBN RI event model discussed in the prior conference paper7

(describing the SME session of November 2013) was substantially modified and simplified during 

the second SME panel review (April 2014), as shown in Figure 2. The nodes in Figure 2 are color 

coded to indicate associations among the various groups. Generally, the flow of specific 

contributing factors through causal paths is from left to right in the figure. The SME panel 
11

validated many of the proposed definitions and most of the proposed model structure. However, 

the SME panel also provided significant clarification of several essential definitions within the RI 

event model. It is intended that the node names are suggestive of the states of each node; hence, 

limited clarifying information is presented about each of the nodes. Most of the nodes in the RI 

event model are binary, meaning they have only two possible states: the issue is present, or not, in 

the RI event under consideration. Where more than two states are present in a node, this will be 

made clear from the explanation of the node given subsequently. The goal of the SME elicitation 

is to provide probabilities for each of the possible states; for example, for the node “Airport 

Layout”, the SME goal is to determine the probability that the Airport Layout is an issue or not in 

the RI event. 

The SME panel unanimously agreed that developing a model for runway safety was much 

more difficult than developing one for in-flight operations. Two members of this SME panel have 

also participated in prior similar model reviews hosted by this NASA team for different 

applications. However, in this case, it was quite challenging to even achieve agreement on the 

basic structure of the BBN model among the NASA team and the SME panel. Numerous 

alternative models have been developed, discussed and discarded by the NASA team, either 

because they did not provide a satisfactory causal path, or because they were deemed to be too 

complex for use within the SME elicitation process for the purpose of portfolio assessment. The 

model proposed and discussed during the November 2013 session was significantly changed by 

the SME panel at that time and then significantly changed again by the same SME panel (April 

2014). Yet another SME review by the same panel (July 2104) altered the model structure further. 

However, through all the discussions and modifications, the runway safety model has been steadily 

improved and clarified. 

5.4 Methodology and Software 

 The scope of the work detailed in this document employed three commercially–distributed 

software products: Microsoft Excel37 and Design-Expert (versions 8 and 9, referred to herein as 

DX8 and DX9, respectively) from Stat-Ease, Inc38 and the Hugin Explorer software (version 8.1) 

from Hugin Expert A/S39

. The first two pieces of software (Microsoft Excel37 and DesignExpert38) were generally used during the data collection and analysis phase of the work5

, while the 

Hugin Explorer software was generally used during the BBN modeling phase of the work6, 7. The 

general workflow that was employed in this study was first to download the RI data set from the 
12

ASIAS web site. Then, the air traffic volume data set20 was downloaded from the FAA ATADS 

web site22. These datasets were downloaded in Microsoft Excel37 format and this software was 

used to sort and extract the information of interest in addition to some statistical processing. The 

intent of this data pre-processing was to develop representative marginal and conditional 

probabilities for use in the BBN modeling of specific events, causes, combinations of contributing 

factors, and the participant types (aircraft classes, and if vehicles or pedestrians were involved) of 

RI events that occurred. In this context, it is not necessary that these searches and sorts be 100% 

accurate, but merely that they provide reasonable guidance about the relative percentages. Having 

prepared the data set into suitable formats, the data was then imported into DX8 for the 

development of response surface (RS) models via the analysis of variance (ANOVA) technique, 

and for additional statistical processing with the software40-44

. Having identified primary and 

secondary causes and contributing factors, a series of increasingly more detailed BBN models were 

developed with the Hugin software and discussed among the team to determine which represented 

the best way to model the RI and RE event structures. These RI and RE event models were then 

present to, and discussed at length with, the SME panel; the SME panel made many additional 

clarifications, simplifications and structural changes to the BBN models. 

 The software choices noted above simply represent software currently available to the author, 

and software packages to which the author is quite familiar, but in no way represent an official 

federal government or NASA endorsement of these software packages. However, these software 

packages are known to include the desired capabilities for accomplishing the objectives of this 

study. 

 The original RI database that was used consists of 10459 records for RI events from October 1, 

2001 through September 30, 2011. The structure and use of this data set has been previously 

documented in detail5,20. 

 The primary Federal Air Regulations (FAR) aircraft categories of interest within this modeling 

effort are 121 (commercial), 135 (air taxi) and 91 (general aviation), but other categories of aircraft 

were included in the data set. In order to compare RI incident rates at various airports, the event 

data was combined with a data set of aircraft traffic volume. The air traffic volume data set 

provided quantitative measures of how many landings and takeoff (grouped together) occurred by 

year at each of over 400 domestic airports in several categories of aircraft. The total air traffic 

volume for each airport is also provided. These datasets were used together to investigate issues 
13

such as the percentage of the air traffic volume (total, or at a specific airport) that resulted in 

runway incursions over a given period of time, and the true, traffic–normalized risk level that is 

associated with those RI events. Again, the intent of the various data analysis operations was to 

develop representative marginal and conditional probabilities of specific events, causes, 

combinations of contributing factors, and participant the types of RI events that occurred. The data 

operations need only provide reasonable guidance about the relative percentages. All the 

quantitative data was then used as the basis for developing the BBN models. 

6.0 Model Development 

6.1 Runway Incursion Model 

 Not all possible combinations of these objects (AC, VEH and PED) are of interest to NASA, 

e.g., VEH in combination with VEH / PED is not a subject of this study. To avoid ambiguity for 

the most important Cat A events, the RI event severity rankings used henceforth in this report are: 

Accident, Near Miss, Cat B, Cat C and Other (including Cat D and Other from above, mentioned 

for completeness but this categorization will not be a subject of modeling or expert elicitation). 

For the purposes of this modeling effort, the scope of attention is restricted to aircraft involved, 

Cat C and above RI events, with movement restrictions to be defined subsequently. For the 

purposes of this study, only two types of RI events are considered: 1) AC and AC and 2) AC and 

VEH. RI events include at least two objects [aircraft (AC), vehicle (VEH) and/or 

person/pedestrian (PED)] with one of the objects being the aircraft. 

 As mentioned previously, the initial referenced RI data base includes 10459 RI events (with no 

narratives). Among these, seven were accidents, 110 were near misses, 114 were Cat B, 2014 were 

Cat C. Note that some RE events started as RI events and these were categorized as “not 

applicable” within the RI database; one such event occurred on August 27, 2006 where an aircraft 

crashed at the Lexington, KY airport resulting in 49 deaths45. The event was ultimately classified 

as an RE event, aircraft takeoff on wrong runway, but actually started as an RI event wherein the 

aircraft entered the takeoff runway (incorrect and too short) without authorization. This initial data 

set is useful for establishing overall probabilities related to the type of RI events that occur. The 

final data set consisted of 1596 RI events (Cat C and above, with brief narratives). Of these, there 

was just one accident (the others among the 10459 were excluded because no narrative was 
14

provided by the FAA), 30 were near misses, 20 were Cat B and 1545 were Cat C events. Of the 

1596 RI events, 1299 were caused by AC, 260 were caused by VEH and 37 were caused by PED. 

The complete data time frame ranges from 2001 through 2011, however, the narrative data time 

frame ranges from 2007 through 2011. The modeling time frame ranges from 2007 through 2014, 

or possibly 2015, at the latest. 

 It is important to realize that RI events are “people intensive”, involving possibly two pilots 

and possibly two controllers (when some form of split control such as air / ground is in effect). A 

VEH driver or PED could replace one of the pilots. There are also organizations (FAA, Airport 

Management, private air transport companies, etc.) behind each of the people directly involved in 

the RI event. The Human Factors Analysis and Classification System (HFACS) by Wiegmann and 

Shappell146 are frequently used to describe the organizational, supervisory and personal factors 

states that establish preconditions for human errors and violations. Unfortunately, the narratives 

provided for the RI events do not provide sufficient detail for a standard HFACS assessment. 

Hence, simplified HFACS models are used within this model, discussed subsequently. 

 Furthermore, RI events are also “communication (Comm) intensive”; several instances of twoparty communications (e.g., communications between pilot and ATC) must simultaneous ly 

function properly in order to avoid problems. Two-party communications involve both the content 

and transmission of information. The content must be correct and complete, timely and not too 

complex for the situation. The transmission must be accomplished without garbled or blocked 

information exchanges. Instances of two party communications exist between all the involved 

people in the RI event. Many times, communication frequencies are shared by numerous 

simultaneous two-party instances and confusion among all the parties can result when incorrect or 

incomplete information is transmitted correctly, or when correct information is not transmitted 

correctly. Taking this to the next level, split controllers are expected (by cockpit crews) to act as 

a unified controller and cockpit crews involving a pilot and co-pilot are expected (by control) to 

operate as a unified AC operator. Hence, any spilt entity needs adequate internal two party 

communications and adequate external two party communications must exist between the various 

entities. Failure of any part of this complex communication network results in deficient two party 

communications that can lead to confusion, a shared attribute among some or all of the participants. 
15

 The RI event severity rating is based strictly on the time / distance. The severity rating does not 

consider the object FAR vehicle classes, the potential for loss of life or damage to property, the 

causal path or error types, nor does it consider the HFACS states of the participants. In short, the 

existing FAA RI event definitions provide a very narrow way to examine runway safety, especially 

if the ultimate goal is to study the impact of technology injections. Many interesting questions can 

be posed and answered in the context of runway safety events that do not directly support RI event 

modeling by the strict FAA definitions; answers to these additional questions would provide 

significant insight into various aspects of possible technology injections and their effectiveness. 

During this work numerous alternative models have been developed, discussed and discarded 

either because they did not provide a satisfactory causal path, or because they were deemed to be 

too complex for use within the SME elicitation process. 

 As noted previously, an attempt has been made to restrict the RI event scenarios of interest 

within this study. Part of this reduced scope involves movement restrictions for the objects 

involved. Object 1 (AC, VEH or PED) initiates the RI event and must be on the runway (RW) at 

the start of the RI event, or as an AC on final approach to the runway, having crossed the runway 

threshold. As an AC, Object 1 arrived (or will shortly arrive) on the RW either by incorrectly 

landing on it, taxiing onto it, or (in the case of crossing runways) the AC may be landing or taking 

off on one RW, while a second AC (Object 2) is using the second RW. If Object 1 is VEH or PED, 

it is assumed to be an authorized agent of airport (an airport affiliated contractor or employee) that 

has moved onto the wrong runway or onto the correct runway but at the wrong time. Cases in 

which the VEH / PED arrived on the RW by uncontrolled, inappropriate runway access either 

directly via the airport perimeter, or indirectly through the airport terminal have been excluded 

from consideration. 

 An RI event perspective versus an aviation perspective has been adopted. This means that every 

situation considered herein is assumed to result in an aircraft involved, Cat C or above RI event. 

Only controlled US airports are considered. The RI event time frame is assumed in the range from 

seconds to minutes. The FAR aircraft types of interest are Part 121 (Commercial) and Part 135 

(Air Taxi). These are considered together due to presumed similar equipment levels; this 

assumption was validated by the SME panel. Another aircraft type of interest, and a major 

contributor to RI events, is Part 91 (General Aviation). Discussion with the SME panel revealed 

that virtually any type of aircraft may be operated as a Part 91 vehicle; thus, the Part 91 distinction 
16

is not very useful in this context. Other categories of AC (e.g., military, maintenance taxi) are 

included in the original data set, but these are not explicitly of interest in this study. The study 

considers pilot(s), controller(s), and relevant objects (vehicles and pedestrians) on the ground. The 

study also indirectly considers various airport geometries, various weather and visibility 

conditions, and various operating conditions. These factors are considered to be fixed during an 

RI event, whereas the participant HFACS states and two party communications are considered to 

be active during an RI event. Likewise, airport signs and markings are considered to be fixed 

mitigations during the RI event, whereas go-arounds, aborted takeoffs or other evasive maneuvers 

are considered to be active mitigating actions performed by the object operators. 

At this point, a preliminary BBN RI event model has been developed. An SME panel consisting 

of four consultants was assembled to review the model structure and to populate its content. The 

SME panel met over two days for about 14 hours of discussion about the complex RI event 

problem. A preliminary baseline RI model was agreed upon, as shown in Figure 2 and model 

population with SME likelihood belief values was also accomplished, discussed in a later section 

of this report. The nodes in Figure 2 are color coded to indicate associations among the various 

nodes. Generally, the flow of specific contributing factors through causal paths is from left to right 

in the figure. Many items funnel together through the two nodes identified as Fixed and Active 

Contributing Factors. The node identified as “RI Event Initiation” can be thought of as the start of 

the active mitigation phase of the RI event, which also ties back to the contributing factors. 
17

Figure 2. The Runway Incursion Bayesian Belief Network. 

The SME panel validated many of the proposed definitions and most of the proposed model 

structure. However, the SME panel also provided significant clarification of several essential 

definitions within the RI event model. Moreover, the SME panel suggested several structural 

changes to the model, especially as related to the best way to model the active mitigation phase of 

the RI events. The overall complexity of the proposed RI event model was reduced from 39 nodes 

and a combined conditional probability table (CPT) size of 1041 elements to one of 37 nodes and 

combined CPT size of 735 elements. The remainder of this section describes the current 

preliminary RI event model. The node name for each is presented along with some clarifying 

comments. Most of the nodes are binary, meaning they have only two possible states: yes or no; 

where more states are present in a node, this will be made clear from the explanation. The goal of 

the SME elicitation is to provide probabilities for each of the possible states; for example, for the 

node “Airport Layout”, the SME goal is to determine the probability that the Airport Layout is an 

issue or not in the RI event. 
18

 The reader should first observe the numbers in Figure 2, representing the three possible primary 

participants in an RI event: 1 is the pilot or pilots (orange node), 2 is the air traffic controller (ATC) 

or controllers (cyan node) and 3 is an airport or contractor vehicle driver (pink node). Each of these 

nodes has a number of other color-coded nodes with links pointing into these three primary nodes. 

Likewise, each of these three primary nodes have links pointing into the black node (Primary Error 

State). Starting with the green nodes (middle top), and moving counter-clockwise, the node 

descriptions of the RI event model follow: 

Airport Issues 

Airport Layout. The airport layout is an issue16,18. This may include potentially confusing 

elements such as parallel runways (with spacing of less than 1000 feet), intersecting runways, and 

taxiways parallel to and near runways, numerous taxiways crossing runways instead of perimeter 

taxiways. 

Signs, Markings and Lighting. The signs, markings and/or fixed equipment (e.g., lights) at the 

airport are deficient. This problem may be exacerbated under severe weather conditions when 

signs, etc. may be obscured from view20. 

Airport Construction or RW/TW Closure. Airport construction or runway/taxiway closure is 

an issue. 

Contamination Control. Contamination control, generally related to rain, snow or ice, is an 

issue18, 47

. 

Airport Issues. One or more of the issues within this grouping are present. 

ATC HFACS Issues 

Next are the purple nodes (upper left) which represent some of the most (as determined by the 

SME panel) Human Factors Analysis and Classification System46 issues. The reader will note that 

a similar purple HFACS group accompanies both the pilot and the vehicle driver, though in the 

latter case, the HFACS group only includes training. 

ATC Cert Training Issues. Certification training for the ATC involved is an issue. 

ATC OTJ Training Issues. On the job training for the ATC involved is an issue as a distraction13. 

ATC Mental or Physical State. The current physical or mental state of the ATC involved is an 

issue. 
19

ATC HFACS Issues. One or more of the HFACS issues within this grouping is an issue for the 

ATC involved46. 

ATC Operational Issues 

The cyan nodes (middle left) are other contributing factors that may influence the performance of 

the ATC. 

Automation Interaction Issues (ATC). Automation interaction is an issue for the ATC 

involved48. 

Abnormal Air Traffic Volume or Complexity. The traffic volume or complexity at this airport, 

at this time is an issue. For example, if the average traffic volume is high, it may cause a 

significantly increased work load for controllers and/or pilots; if low, it may result in extended 

periods of inactivity for controllers16. 

Staffing or Procedures Issues. The staffing level and/or work load management not appropriate 

for the situation is an issue, or the use of ambiguous or non-standardized ATC procedures is an 

issue16. 

ATC Operational Issues. One or more of the issues within this grouping are present. 

Two Party Communications 

The next group (blue nodes) describe the state of the system-wide two-party communications. 

Communication Content Issues. The completeness, correctness timeliness or complexity of 

communicated information is an issue. This may include the lack of a required usage for a call 

sign. Information may not have been transmitted at the appropriate time, i.e., it was not delayed17. 

Comm Hardware Error. Comm transmission is an issue. This may occur when the Comm system 

fails to operate as expected and may include blocked (“stepped on” communications where one 

party cuts off the communications of another), partially blocked (garbled or inaudible Comm 

transmission), hardware limitations / malfunctions and/or faulty headset jacks or connections17. 

Two Party Communication Issues. Comm Content Issues or a Communications Hardware Error 

has resulted in a Two Party Communications Error and is an issue. 
20

Pilot HFACS Issues 

Another group of HFACS contributing factors is present for the pilot. The grouping is the same as 

before, although the relevant probabilities of these factors being an issue in an RI event may be 

different than for the ATC. 

Pilot Cert Training Issues. Certification training for the pilot(s) involved is an issue. 

Pilot OTJ Training Issues. On the job training for the pilot(s) involved is an issue as a 

distraction13. 

Pilot Mental or Physical State. The current physical or mental state of the pilot(s) involved is an 

issue. 

Pilot HFACS Issues. One or more of the HFACS issues within this grouping is an issue for the 

pilot(s) involved46. 

Other Pilot Operational Issues 

Other contributing factors that may influence the performance of the pilot(s) involved are show in 

the orange nodes. 

Inappropriate Aircraft Operations: Pilot operations of the aircraft, outside of the flight 

operational manual guidelines, is an issue causing the RI event. 

Automation Interaction Issues (Pilot): Automation interaction is an issue for the pilot(s) 

involved48. 

Pilot Operational Issues: One or more of the issues within this grouping are present. 

Driver Operational Issues 

These issues include the two nodes below. (purple and pink nodes, top right) 

Driver Training: The training of airport vehicle drivers is an issue13. 

Driver Operational Issues: One or more of the HFACS issues within this grouping is an issue 

for the vehicle driver(s) involved46. 

 The preceding discussion covers all the nodes on the periphery of the left hand side of Figure 

2. These are all the issues potentially present that enable the RI event to occur. The nodes and 

states on the far right hand side of the figure generically define a specific RI event, of which 

numerous types and combinations may occur. NASA would hope to be in a position to broadly 
21

address many, if not all, of these specific RI event types with technology injections. Starting with 

the black node of Figure 2 (middle right), the node descriptions follow. 

Primary Error State: The primary error source is either Controller Error (typically, loss of 

oversight), Pilot Error (typically, failure to hold short of a runway without authorization), or Other 

(includes mechanical failure and Driver Error, i.e., a failure to hold short of a runway without 

authorization). The SME panel excluded from consideration in this model non-airport authorized 

vehicles and all pedestrians on the runway. 

Collision Scenarios. The SME panel identified the most common collision scenarios: crossing in 

front of an aircraft on departure, crossing in front of an aircraft on arrival, or intersection events 

(crossing active runways) and other (everything else that leads to an RI event)13. 

Reaction time. This node has two states defined by the SME panel, short (eight seconds or less) 

and long (nine seconds or more) 

Final RI Event Severity. The RI event severity as would be reported by the FAA, including the 

impact of Contributing Factors and Mitigating Actions is established here. The states enumerated 

by the SME panel are accident / near miss, or other. Although less severe RI event categories have 

been defined by the FAA, these were deemed out of scope for this BBN model because the SME 

panel could not provide sufficient discrimination among these less severe RI events. 

 This concludes the presentation of the RI event model. Likewise, the SME panel vetted many 

of the proposed definitions and most of the proposed model structure of the RE event model, shown 

in Figure 3. The SME panel again provided significant clarification of several essential definitions 

within the RE event model. Moreover, the SME panel suggested several simplifying structural 

changes to the model. The remainder of this section describes the current preliminary RE event 

model. The node name for each is presented along with some clarifying comments. Again, most 

of the nodes are binary with only two possible states (present as in issue in RE Events or not); 

where more states are present in a node, this will be made clear from the explanation of the node 

given subsequently. 

6.2 Runway Excursion Model 

 An overrun is an RE event in which the aircraft departs the end of a runway; a veer-off is an 

RE event in which an aircraft departs the side of a runway. As an RE event may occur on landing 
22

or takeoff, four types of RE events are possible: landing overrun, landing veer-off, takeoff overrun 

and takeoff veer-off. Some sources also include within RE events an aircraft attempting a landing 

that touches down in the undershoot area of the designated landing runway within the aerodrome 

perimeter49. 

Figure 3. The Runway Excursion Bayesian Belief Network. 

Airport Issues 

Beginning with green nodes in the upper right corner of Figure 3, and moving counter-clockwise, 

the nodes and states are below. 

Approach and Departure Constraints. The physical or regulatory constraints on approach or 

departure trajectories for the airport in question are an issue. 

Contamination Control. Contamination control (e.g., rain, snow or ice) for the airport in question 

is an issue. 

Runway Length. The runway length is an issue. This may be due to prevailing wind conditions, 

runway maintenance, or an aircraft landing on a runway that is too short for safe operations. 
23

Airport Issues: One or more of the issues within this grouping are present. 

ATC HFACS Issues 

As in the RI event network, the ATC HFACS Issues group (purple nodes) is repeated in the RE 

event network. The SME panel rated the relative importance of these contributing factors for the 

ATC as being of much less consequence for RE events than for RI events. Next, the cyan nodes 

are described: 

ATC Operational Issues 

Runway Assignment. The runway assignment provided by ATC is an issue. This may be due to 

prevailing wind conditions, runway maintenance, or unusual airport operations. 

Runway Collision Avoidance. An RI event (typically failure to hold short of an active runway) 

has precipitated an RE event. This was noted by the SME panel as being an extremely rare 

occurrence. 

Contribution to Unstabilized Approach. The ATC has provided instructions that contribute to 

an unstabilized approach. 

Lack of Current Weather Information. The ATC involved have provided non-current weather 

information that contributes to an RE event. 

ATC Operational Issues. One or more of the issues within this grouping are present. 

Pilot HFACS Issues 

As in the RI event network, the Pilot HFACS Issues is repeated in the RE event network. The SME 

panel rated the relative importance of these contributing factors for the pilot as being about equal 

for RE and RI events. 
24

Pilot Operational Issues and Remainder Nodes 

Inappropriate Aircraft Operations. Pilot operations of the aircraft, outside of the flight 

operational manual guidelines, is an issue causing the RE event. 

Unstabilized Approach. The pilot(s) involved have failed to perform a stabilized approach. 

Pilot Operational Issues. One or more of the issues within this grouping are present. 

Aircraft Automation Issues (blue node). Automation interaction is an issue for the pilot(s) 

involved49. 

Pilot Error. A pilot error has initiated an RE event. 

Weather Issues (yellow node). Weather issues have contributed to, or caused, an RE event. 

Mechanical Failure (pink node). Mechanical failure has contributed to, or caused, an RE event. 

RE Event Initiated: This node simply states whether an RE event has been initiated or not. 

7.0 Model Population 

7.1 Runway Incursion Model 

 The subject matter expert (SME) model elicitation for the node Airport Layout is summarized 

in Table 1. Four SMEs, identified as SME1, SME2, SME3 and SME4, were used for the 

probability elicitation. 

Table 1. SME Probabilities and Confidence Bounds for the Node Airport Layout.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 

Lo

95% 

Lo

90% 

Lo

90% 

Hi

95% 

Hi

99% 

Hi

issue 0.30 0.40 0.30 0.25 0.25 0.40 0.31 0.06 0.15 0.19 0.21 0.41 0.43 0.47

not 0.70 0.60 0.70 0.75 0.75 0.60 0.69 0.94 0.85 0.81 0.79 0.59 0.57 0.53

 The first column identifies the state of the node as being an issue for RI events or not. Columns 

two through five provide the SME probabilities for each state. In column six, the minimum (Min) 

of the four SME responses is computed. In column seven, the maximum (Max) of the four SME 

responses is computed. In column eight, the average (Avg) of the four SME responses is 
25

computed. In column nine, the standard deviation (StDev) of the four SME responses is computed. 

Columns 10 through 15 provide enclosed confidence bounds of the SME responses defined as: 

 Column 10: 99% Lo = Avg – 2.58*StDev 

 Column 11: 95% Lo = Avg – 1.96*StDev 

 Column 12: 90% Lo = Avg – 1.64*StDev 

 Column 13: 90% Hi = Avg + 1.64*StDev 

 Column 14: 95% Hi = Avg + 1.96*StDev 

 Column 15: 99% Hi = Avg + 2.58*StDev 

 The multiplicative factors above were derived for the confidence intervals were based upon the 

Microsoft Excel function NORMINV(Prob,0,1), where Prob = 0.950, 0.975 and 0.995 for the 90%, 

95% and 99% confidence bounds above, respectively. These provide some meaningful ranges of 

the SME inputs to desired levels of confidence that can be used in sensitivity analysis studies. 

 The SME model elicitation for the node Signs, Markings and Equipment is summarized in 

Table 2. 

Table 2. SME Probabilities and Confidence Bounds for the Node 

Signs, Markings and Equipment. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 

Lo

95% 

Lo

90% 

Lo

90% 

Hi

95% 

Hi

99% 

Hi

issue 0.15 0.2 0.15 0.15 0.15 0.2 0.16 0.02 0.1 0.11 0.12 0.2 0.21 0.22

not 0.85 0.8 0.85 0.85 0.85 0.8 0.84 0.98 0.9 0.89 0.88 0.8 0.79 0.78

 The SME model elicitation for the node Airport Construction or runway (RW) or taxiway (TW) 

Closure is summarized in Table 3. 
26

Table 3. SME Probabilities and Confidence Bounds for the Node 

Airport Construction or RW/TW Closure. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 

Lo

95% 

Lo

90% 

Lo

90% 

Hi

95% 

Hi

99% 

Hi

issue 0.10 0.10 0.10 0.15 0.10 0.15 0.11 0.02 0.05 0.06 0.07 0.15 0.16 0.17

not 0.90 0.90 0.90 0.85 0.90 0.85 0.89 0.98 0.95 0.94 0.93 0.85 0.84 0.83

 The SME model elicitation for the node Contamination Control is summarized in Table 4. In 

this case, the 99%, 95% and 90% Lo confidence bounds would be less than zero. However, the 

computed values have been constrained to be greater than or equal to zero and less than or equal 

to unity.

Table 4. SME Probabilities and Confidence Bounds for the Node 

Contamination Control. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 

Lo

95% 

Lo

90% 

Lo

90% 

Hi

95% 

Hi

99% 

Hi

issue 0.00 0.05 0.01 0.01 0.00 0.05 0.02 0.02 0.00 0.00 0.00 0.06 0.07 0.08

not 1.00 0.95 0.99 0.99 1.00 0.95 0.98 0.98 1.00 1.00 1.00 0.94 0.93 0.92

 The probabilities for the node Airport Issues are conditioned upon the presence or absence of 

the contributing factors in the leaf nodes which link into this node. That is to say the conditional 

probability table (CPT) in the node Airport Issues is conditioned upon the probabilities in the nodes 

1) Airport Layout, 2) Signs, Marking and Equipment, 3) Airport Construction or RW/TW Closure 

and 4) Contamination Control. Strictly speaking, the Airport Issues is an “or” node: if any one of 

the contributing factors is present, then the result should be that airport issues are present. Such a 

CPT for a binary node (two output states, issues are present or issues are not present, highlighted 

in yellow) with two binary input factors is shown in Table 5. The reader should notice that all the 

probabilities are set to either zero or unity (highlighted in green). 
27

Table 5. Conditional Probability Table for a Two-Factor Binary “or” Node.

Factor 1 Present Present Not Present Not Present

Factor 2 Present Not Present Present Not Present

Present 1 1 1 0

Not Present 0 0 0 1

 However, “or” nodes pose some difficulties for sensitivity analysis; hence a variety of other 

methods to populate the CPT for nodes like Airport Issues have been explored. In the current 

instance, the average SME probability for each of the four leaf nodes as being an issue that input 

to Airport Issues were summed, and then the probabilities were renormalized so as to sum to unity. 

That is, 0.31, 0.16, 0.11 and 0.02, for each of Airport Layout, Signs, Marking and Equipment, 

Airport Construction or RW/TW Closure and Contamination Control, respectively were summed 

(0.60) and then the probabilities were renormalized (0.52, 0.27, 0.18 and 0.03, respectively), so as 

to sum to unity. These renormalized probabilities were then summed to reflect the presence of the 

contributing factors. The resulting CPT for the Airport Issues node is shown in Table 6. A “yes” 

state indicates the factor is an issue, whereas a “no” state indicates that the factor is not an issue.

Table 6. Conditional Probability Table for the Node Airport Issues. 
28

 Other techniques for populating a CPT such as shown in Table 6 include: direct SME elicitation 

to determine combinatorial probabilities of the individual contributing factors, rank ordering, and 

weighting of the individual contributing factors. Direct elicitation of the SME to determine 

combinatorial probabilities (CPT values) of the individual contributing factors for a node like 

Airport Issues proved to be very difficult and time consuming for the RI model shown in Figure 

2. If the node under consideration has more than two states (such as the Primary Error State node, 

black, in Figure 2 with three states), this process became exponentially more tedious. What is 

desired is to get some measure of the relative importance of the individual contributing factors 

when considered in combination. The simplest approach is just to assume equal weighting for the 

contributing factors. Thus, for a node like Airport Issues, with four input nodes, the only possible 

values for the CPT are 0.00, 0.25, 0.50, 0.75 or 1.00, depending on how many of the four 

contributing factors are present. The next simplest approach may be just to have the SME panel 

rank order the contributing factors. For example the SME panel might (on average) rank the four 

contributing factors in this order: 

1. Sign, Marking and Equipment (most influential) 

2. Layout 

3. Construction or RW/TW Closure 

4. Contamination Control (least influential) 

 The sum of the rank orderings is 10 (1 + 2 + 3 + 4 = 10). The probabilities can then be 

renormalized and allocated in reverse order (0.4, 0.3, 0.2 and 0.1, respectively), so that the most 

influential contributing factor (Sign, Marking and Equipment) gets the greatest relative probability 

contribution (0.4). These relative importance probabilities can then be summed as described 

previously. This would lead to CPT values in the Airport Issues node having all the possible 

combinations of these values, as shown in Table 7. 
29

Table 7. Conditional Probability Table based on rank ordering of contributing factors.

 Finally, the SME panel can be asked to provide specific weightings for each of the 

contributing factors. If these weightings sum to unity, then no renormalization is necessary. 

 The SME model elicitation for the node Air Traffic Control (ATC) Certification (Cert) 

Training Issues is summarized in Table 8. 

Table 8. SME Probabilities and Confidence Bounds for the Node 

ATC Cert Training Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 

Lo

95% 

Lo

90% 

Lo

90% 

Hi

95% 

Hi

99% 

Hi

issue 0.01 0.02 0.01 0.01 0.01 0.02 0.01 0.01 0.00 0.00 0.00 0.02 0.02 0.02

not 0.99 0.98 0.99 0.99 0.99 0.98 0.99 1.00 1.00 1.00 1.00 0.98 0.98 0.98

 The SME model elicitation for the node ATC On-The-Job Training (OJT) Issues is 

summarized in Table 9. 

Table 9. SME Probabilities and Confidence Bounds for the Node 

ATC OJT Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 

Lo

95% 

Lo

90% 

Lo

90% 

Hi

95% 

Hi

99% 

Hi

issue 0.05 0.05 0.08 0.08 0.05 0.08 0.06 0.02 0.02 0.03 0.03 0.09 0.09 0.10

not 0.95 0.95 0.92 0.93 0.95 0.92 0.94 0.98 0.98 0.97 0.97 0.91 0.91 0.90
30

 The SME model elicitation for the node ATC Mental or Physical State Issues is summarized 

in Table 10. 

Table 10. SME Probabilities and Confidence Bounds for the Node 

ATC Mental or Physical State Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.10 0.02 0.10 0.02 0.10 0.07 0.04 0.00 0.00 0.01 0.13 0.15 0.17

not 0.95 0.90 0.98 0.90 0.98 0.90 0.93 0.96 1.00 1.00 0.99 0.87 0.85 0.83

 The same method, described above, used for the node Airport Issues was also used to 

construct the CPT for the node ATC Human Factors Analysis and Classification System 

(HFACS) Issues. That is, the average probabilities for the three input nodes were summed and 

renormalized; the resulting normalized probabilities were then summed again to reflect the 

presence of the three contributing factors, as shown in Table 11. 

Table 11. Conditional Probability Table for the Node ATC HFACS Issues.

state yes yes yes yes no no no no

ojt yes yes no no yes yes no no

cert yes no yes no yes no yes no

ATC 

HFACS 

Issues

1.00 0.93 0.57 0.50 0.50 0.43 0.07 0.00

No

ATC

HFACS

Issues

0.00 0.07 0.43 0.50 0.50 0.57 0.93 1.00

 The SME model elicitation for the node Communication (Comm) Content Issues is 

summarized in Table 12. 
31

Table 12. SME Probabilities and Confidence Bounds for the Node 

Comm Content Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.25 0.35 0.30 0.40 0.25 0.40 0.33 0.06 0.16 0.20 0.22 0.44 0.46 0.50

not 0.75 0.65 0.70 0.60 0.75 0.60 0.67 0.94 0.84 0.80 0.78 0.56 0.54 0.50

 The SME model elicitation for the node Comm Hardware Issues is summarized in Table 13.

Table 13. SME Probabilities and Confidence Bounds for the Node 

Comm Hardware Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.01 0.01 0.00 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.02 0.02 0.02

not 0.99 0.99 1.00 1.00 1.00 0.99 0.99 1.00 1.00 1.00 1.00 0.98 0.98 0.98

 The same method, described above, used for the node Airport Issues was also used to 

construct the CPT for the node Two-Party Comm Issues. That is, the average probabilities for 

the two input nodes were summed and renormalized; the resulting normalized probabilities were 

then summed again to reflect the presence of the two contributing factors, as shown in Table 14. 

Table 14. Conditional Probability Table for the Node Two-Party Comm Issues.

content yes yes no no

hardware yes no yes no

Two-Party 

Comm 

Issues

1.00 0.97 0.03 0.00

No

Issues

0.00 0.03 0.97 1.00
32

The SME model elicitation for the node Pilot Cert Training Issues is summarized in Table 15. 

Table 15. SME Probabilities and Confidence Bounds for the Node 

Pilot Cert Training Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.01 0.02 0.02 0.01 0.01 0.02 0.02 0.01 0.01 0.01 0.01 0.03 0.03 0.03

not 0.99 0.98 0.98 0.99 0.99 0.98 0.98 0.99 0.99 0.99 0.99 0.97 0.97 0.97

 The SME model elicitation for the node Pilot OJT Issues is summarized in Table 16. 

Table 16. SME Probabilities and Confidence Bounds for the Node 

Pilot OJT Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.01 0.01 0.03 0.01 0.05 0.03 0.02 0.00 0.00 0.00 0.06 0.07 0.08

not 0.95 0.99 0.99 0.97 0.99 0.95 0.97 0.98 1.00 1.00 1.00 0.94 0.93 0.92

 The SME model elicitation for the node Pilot Mental or Physical State Issues is summarized 

in Table 17. 

Table 17. SME Probabilities and Confidence Bounds for the Node 

Pilot Mental or Physical State Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.05 0.02 0.02 0.02 0.05 0.04 0.02 0.00 0.01 0.01 0.07 0.07 0.08

not 0.95 0.95 0.98 0.98 0.98 0.95 0.96 0.98 1.00 0.99 0.99 0.93 0.93 0.92

 The same method, described above, used for the node Airport Issues was also used to 

construct the CPT for the node Pilot HFACS Issues. That is, the average probabilities for the 
33

three input nodes were summed and renormalized; the resulting normalized probabilities were 

then summed again to reflect the presence of the three contributing factors, as shown in Table 18. 

Table 18. Conditional Probability Table for the Node Pilot HFACS Issues.

state yes yes yes yes no no no no

ojt yes yes no no yes yes no no

cert yes no yes no yes no yes no

Pilot 

HFACS 

Issues

1.00 0.78 0.67 0.44 0.56 0.33 0.22 0.00

No

Pilot

HFACS

Issues

0.00 0.22 0.33 0.56 0.44 0.67 0.78 1.00

 The SME model elicitation for the node Automation Interaction Issues (Pilot) is summarized 

in Table 19. 

Table 19. SME Probabilities and Confidence Bounds for the Node 

Automation Interaction Issues (Pilot). 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.03 0.01 0.05 0.01 0.01 0.05 0.02 0.02 0.00 0.00 0.00 0.05 0.06 0.07

not 0.97 0.99 0.95 1.00 1.00 0.95 0.98 0.98 1.00 1.00 1.00 0.95 0.94 0.93
34

 The SME model elicitation for the node Inappropriate Aircraft Operations is summarized in 

Table 20. 

Table 20. SME Probabilities and Confidence Bounds for the Node 

Inappropriate Aircraft Operations. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.10 0.10 0.01 0.10 0.01 0.10 0.08 0.05 0.00 0.00 0.01 0.15 0.17 0.20

not 0.90 0.90 0.99 0.90 0.99 0.90 0.92 0.96 1.00 1.00 0.99 0.85 0.83 0.80

 The SME model elicitation for the node Automation Interaction Issues (ATC) is summarized 

in Table 21. 

Table 21. SME Probabilities and Confidence Bounds for the Node 

Automation Interaction Issues (ATC). 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.00 0.05 0.05 0.10 0.00 0.10 0.05 0.04 0.00 0.00 0.00 0.12 0.13 0.15

not 1.00 0.95 0.95 0.90 1.00 0.90 0.95 0.96 1.00 1.00 1.00 0.88 0.87 0.85

 The SME model elicitation for the node Abnormal Air Traffic Volume or Complexity is 

summarized in Table 22.

Table 22. SME Probabilities and Confidence Bounds for the Node 

Abnormal Air Traffic Volume or Complexity. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.20 0.20 0.05 0.25 0.05 0.25 0.18 0.09 0.00 0.01 0.04 0.32 0.35 0.40

not 0.80 0.80 0.95 0.75 0.95 0.75 0.82 0.91 1.00 0.99 0.96 0.68 0.65 0.60
35

 The SME model elicitation for the node Staffing or Procedural Issues is summarized in 

Table 23. 

Table 23. SME Probabilities and Confidence Bounds for the Node 

Staffing or Procedural Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.15 0.20 0.25 0.05 0.25 0.16 0.09 0.00 0.00 0.02 0.30 0.33 0.38

not 0.95 0.85 0.80 0.75 0.95 0.75 0.84 0.91 1.00 1.00 0.98 0.70 0.67 0.62

 For the node ATC Operational Issues, a rank ordering of the relative effects of the six input 

nodes (Airport Issues, ATC HFACS Issues, Automation Interaction Issues (ATC), Abnormal Air 

Traffic Volume or Complexity, Staffing or Procedural Issues and Two-Party Comm Issues) was 

used. The average SME ranking of the six inputs was as follows: 

1. ATC HFACS Issues (most influential) 

2. Abnormal Air Traffic Volume or Complexity

3. Staffing or Procedural Issues and Two-Party Comm Issues (tied) 

4. Airport Issues 

5. Automation Interaction Issues (ATC) (least influential) 

 This Excel ranking (6, 5, 3.5, 3.5, 2 and 1, respectively) summed to 21, providing renormalized 

values of 0.29, 0.24, 0.17, 0.17, 0.10 and 0.05, respectively, highlighted in green in Table 24. The 

resulting 64 column CPT is too large to be legibly shown horizontally in this report. Hence, a 

transposed version of the CPT is shown in Table 24. The cells shown in green are the renormalized 

relative probabilities for the six input nodes that were summed in the column labeled “ATC Ops 

Issues” when a “yes” appeared in a row under the six input factors. 
36

Table 24. Transposed Conditional Probability Table for the Node ATC Operational Issues.

0.10 0.17 0.29 0.17 0.24 0.05 ATC Ops 

Issues

No ATC 

Ops 

Issues airport comm hfacs staffing volume auto

yes yes yes yes yes yes 1.00 0.00

yes yes yes yes yes no 0.95 0.05

yes yes yes yes no yes 0.76 0.24

yes yes yes yes no no 0.71 0.29

yes yes yes no yes yes 0.83 0.17

yes yes yes no yes no 0.79 0.21

yes yes yes no no yes 0.60 0.40

yes yes yes no no no 0.55 0.45

yes yes no yes yes yes 0.71 0.29

yes yes no yes yes no 0.67 0.33

yes yes no yes no yes 0.48 0.52

yes yes no yes no no 0.43 0.57

yes yes no no yes yes 0.55 0.45

yes yes no no yes no 0.50 0.50

yes yes no no no yes 0.31 0.69

yes yes no no no no 0.26 0.74

yes no yes yes yes yes 0.83 0.17

yes no yes yes yes no 0.79 0.21

yes no yes yes no yes 0.60 0.40

yes no yes yes no no 0.55 0.45

yes no yes no yes yes 0.67 0.33

yes no yes no yes no 0.62 0.38

yes no yes no no yes 0.43 0.57

yes no yes no no no 0.38 0.62

yes no no yes yes yes 0.55 0.45

yes no no yes yes no 0.50 0.50

yes no no yes no yes 0.31 0.69

yes no no yes no no 0.26 0.74

yes no no no yes yes 0.38 0.62

yes no no no yes no 0.33 0.67

yes no no no no yes 0.14 0.86

yes no no no no no 0.10 0.90

no yes yes yes yes yes 0.90 0.10

no yes yes yes yes no 0.86 0.14

no yes yes yes no yes 0.67 0.33

no yes yes yes no no 0.62 0.38
37

no yes yes no yes yes 0.74 0.26

no yes yes no yes no 0.69 0.31

no yes yes no no yes 0.50 0.50

no yes yes no no no 0.45 0.55

no yes no yes yes yes 0.62 0.38

no yes no yes yes no 0.57 0.43

no yes no yes no yes 0.38 0.62

no yes no yes no no 0.33 0.67

no yes no no yes yes 0.45 0.55

no yes no no yes no 0.40 0.60

no yes no no no yes 0.21 0.79

no yes no no no no 0.17 0.83

no no yes yes yes yes 0.74 0.26

no no yes yes yes no 0.69 0.31

no no yes yes no yes 0.50 0.50

no no yes yes no no 0.45 0.55

no no yes no yes yes 0.57 0.43

no no yes no yes no 0.52 0.48

no no yes no no yes 0.33 0.67

no no yes no no no 0.29 0.71

no no no yes yes yes 0.45 0.55

no no no yes yes no 0.40 0.60

no no no yes no yes 0.21 0.79

no no no yes no no 0.17 0.83

no no no no yes yes 0.29 0.71

no no no no yes no 0.24 0.76

no no no no no yes 0.05 0.95

no no no no no no 0.00 1.00

 For the node Pilot Operational Issues, a hybrid approach was used to populate the CPT. The 

average SME input for the Inappropriate Aircraft Operations (0.08) and Automation Interaction 

Issues (Pilot) (0.02) were used. The remaining three input nodes to Pilot Operational Issues (e.g.,

Airport Issues, Two-Party Comm Issues and Pilot HFACS Issues) were already conditional nodes 

and no ranking ordering of these was performed. Hence, the average CPT value (0.5) was used as 

the weighting value for each of these three inputs to the Pilot Operational Issues node. The 

resulting renormalization of the weighting values (0.5, 0.5, 0.5 0.08 and 0.02 renormalized to 0.31, 

0.31, 0.31, 0.05 and 0.01, highlighted in green in Table 25) gives a disproportionate weight to the 

five inputs. The resulting CPT is shown (transposed) in Table 25. In retrospect, eliciting a rank 

ordering, or even using equal weighting, of the five inputs would have been a preferred method. 
38

Table 25. Transposed Conditional Probability Table for the Node Pilot Operational Issues.

0.31 0.31 0.31 0.05 0.01 Pilot Ops 

Issues

No Pilot 

Ops 

Issues airport comm hfacs in ops auto

yes yes yes yes yes 1.00 0.00

yes yes yes yes no 0.99 0.01

yes yes yes no yes 0.95 0.05

yes yes yes no no 0.94 0.06

yes yes no yes yes 0.69 0.31

yes yes no yes no 0.68 0.33

yes yes no no yes 0.64 0.36

yes yes no no no 0.63 0.38

yes no yes yes yes 0.69 0.31

yes no yes yes no 0.68 0.33

yes no yes no yes 0.64 0.36

yes no yes no no 0.63 0.38

yes no no yes yes 0.38 0.63

yes no no yes no 0.36 0.64

yes no no no yes 0.33 0.68

yes no no no no 0.31 0.69

no yes yes yes yes 0.69 0.31

no yes yes yes no 0.68 0.33

no yes yes no yes 0.64 0.36

no yes yes no no 0.63 0.38

no yes no yes yes 0.38 0.63

no yes no yes no 0.36 0.64

no yes no no yes 0.33 0.68

no yes no no no 0.31 0.69

no no yes yes yes 0.38 0.63

no no yes yes no 0.36 0.64

no no yes no yes 0.33 0.68

no no yes no no 0.31 0.69

no no no yes yes 0.06 0.94

no no no yes no 0.05 0.95

no no no no yes 0.01 0.99

no no no no no 0.00 1.00

 
39

 The SME model elicitation for the node Driver Training is summarized in Table 26. 

Table 26. SME Probabilities and Confidence Bounds for the Node 

Driver Training. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.03 0.01 0.04 0.03 0.01 0.04 0.03 0.01 0.00 0.01 0.01 0.05 0.05 0.06

not 0.97 0.99 0.96 0.97 0.99 0.96 0.97 0.99 1.00 0.99 0.99 0.95 0.95 0.94

 For the node Driver Operational Issues, the hybrid approach (described above) was again used 

to populate the CPT. The average SME input (0.03) for the node Driver Training was used along 

with the average CPT value (0.5) for the nodes Airport Issues and Two-Party Comm Issues. The 

resulting renormalization of the weighting values (0.5, 0.5, and 0.03 renormalized to 0.49, 0.49 

and 0.03) gives a disproportionate weight to the three inputs. The resulting CPT is shown in Table 

27. In retrospect, eliciting a rank ordering, or even using equal weighting, of the five inputs would 

have been a preferred method. 

Table 27. Conditional Probability Table for the Node Driver Operational Issues. 

training yes yes yes yes no no no no

comm yes yes no no yes yes no no

airport yes no yes no yes no yes no

driver 

ops 

issues

1.00 0.51 0.51 0.03 0.97 0.49 0.49 0.00

no driver 

ops 

issues

0.00 0.49 0.49 0.97 0.03 0.51 0.51 1.00

 The node Primary Error State has three input links (Driver Operational Issues, ATC Operational 

Issues and Pilot Operational Issues) and three output states: Pilot Error, Controller Error and Other. 

Unfortunately, the SME panel provided probabilities for a different version of this table in which 

four output states were possible: Pilot Error, Controller Error, Driver Error and Other (mechanical 

failure); subsequent discussions removed the “Other” state from this node and removed from 
40

consideration any RI caused by mechanical failure. However, since the elicitation results did not 

align with the final version of table, a decision was made to combine the elicitation results for the 

Driver Error state with those for the Other state, and to rename the third output state as Other 

(includes Driver Error and mechanical failure). The resulting CPT has 3 rows and 8 columns. 

Each column describes the probabilities for a combination of Driver Operational Issues, Pilot 

Operational Issues and ATC Operational Issues; these combinations are highlighted in orange, 

yellow and green as shown in Table 28. 

Table 28. SME Probabilities and Confidence Bounds for the Node 

Primary Error State.

driver issues yes pilot issues yes controller issues yes

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.30 0.49 0.44 0.19 0.25 0.28 0.60 0.63 0.69

controller error 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20 0.20

other 0.31 0.50 0.36 0.11 0.17 0.20 0.52 0.55 0.61

driver issues yes pilot issues yes controller issues no

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.45 0.70 0.59 0.32 0.39 0.42 0.76 0.79 0.86

controller error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

other 0.30 0.55 0.42 0.13 0.20 0.23 0.61 0.64 0.71

driver issues yes pilot issues no controller issues yes

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

controller error 0.35 0.40 0.38 0.31 0.32 0.33 0.43 0.44 0.45

other 0.59 0.66 0.63 0.54 0.56 0.57 0.69 0.70 0.72

driver issues yes pilot issues no controller issues no

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

controller error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

other 0.99 1.00 1.00 0.99 0.99 0.99 1.00 1.00 1.00

driver issues no pilot issues yes controller issues yes

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.60 0.70 0.65 0.50 0.54 0.56 0.74 0.76 0.80

controller error 0.25 0.39 0.32 0.16 0.20 0.22 0.42 0.44 0.48

other 0.01 0.05 0.03 0.00 0.00 0.00 0.07 0.08 0.09

driver issues no pilot issues yes controller issues no

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.99 1.00 0.99 0.98 0.98 0.98 1.00 1.00 1.00

controller error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
41

other 0.00 0.01 0.01 0.00 0.00 0.00 0.02 0.02 0.02

driver issues no pilot issues no controller issues yes

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

controller error 0.99 1.00 0.99 0.98 0.98 0.98 1.00 1.00 1.00

other 0.00 0.01 0.01 0.00 0.00 0.00 0.02 0.02 0.02

driver issues no pilot issues no controller issues no

Min Max Avg 99% Lo 95% Lo 90% Lo 90% Hi 95% Hi 99% Hi

pilot error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

controller error 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

other 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

 

 The SME model elicitation for the node Collision Scenarios is summarized in Table 29. 

Table 29. SME Probabilities and Confidence Bounds for the Node 

Collision Scenarios. 

Collision 

Scenarios SME1 SME2 SME3 SME4 Min Max Avg StDev 99%Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

crossing / 

departure 0.10 0.15 0.20 0.15 0.10 0.20 0.15 0.04 0.04 0.07 0.08 0.22 0.23 0.26

crossing / 

arrival 0.10 0.15 0.10 0.20 0.10 0.20 0.14 0.05 0.02 0.05 0.06 0.22 0.23 0.26

intersecting 

runways

0.05 0.05 0.05 0.10 0.05 0.10 0.06 0.03 0.00 0.01 0.02 0.10 0.11 0.12

other 0.75 0.65 0.65 0.55 0.55 0.75 0.65 0.08 0.44 0.49 0.52 0.78 0.81 0.86

 The SME model elicitation for the node Reaction Time is summarized in Table 30. 

Table 30. SME Probabilities and Confidence Bounds for the Node 

Reaction Time. 

reaction 

time

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

short

(8 sec or 

less)

0.30 0.40 0.03 0.05 0.03 0.40 0.20 0.18 0.00 0.00 0.00 0.50 0.56 0.67
42

long

(9 sec or 

more)

0.70 0.60 0.97 0.95 0.97 0.60 0.80 0.18 0.33 0.44 0.50 1.00 1.00 1.00

 The node Potential RI Event Severity is conditioned upon the nodes Primary Error State, 

Collision Scenario and Reaction Time. A direct CPT elicitation was used and the average 

probabilities for the node are presented in Table 31. In Table 31, the following abbreviations are 

used: “cont” refers to controller error and “xrw” means crossing runways. 

Table 31. Transposed Conditional Probability Table for the Node Potential RI Event 

Severity. 

reaction 

time

collision 

scenario

error 

state

accident 

/ near 

miss

other

short depart cont 0.05 0.95

short depart pilot 0.08 0.92

short depart other 0.09 0.91

short arrive cont 0.05 0.95

short arrive pilot 0.07 0.93

short arrive other 0.03 0.97

short xrw cont 0.03 0.97

short xrw pilot 0.06 0.94

short xrw other 0.08 0.92

short other cont 0.01 0.99

short other pilot 0.02 0.98

short other other 0.02 0.98

long depart cont 0.038 0.962

long depart pilot 0.0608 0.9392

long depart other 0.0684 0.9316

long arrive cont 0.038 0.962

long arrive pilot 0.0532 0.9468

long arrive other 0.0228 0.9772

long xrw cont 0.0228 0.9772

long xrw pilot 0.0456 0.9544

long xrw other 0.0608 0.9392

long other cont 0.0076 0.9924

long other pilot 0.0152 0.9848

long other other 0.0152 0.9848
43

7.2 Runway Excursion Model 

 The SME model elicitation for the node Approach and Departure Constraints is summarized in 

Table 32. 

Table 32. SME Probabilities and Confidence Bounds for the Node 

Approach and Departure Constraints. 

SME1 SME2 SME2 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.10 0.05 0.03 0.03 0.10 0.06 0.03 0.00 0.00 0.00 0.11 0.12 0.14

not 0.95 0.90 0.95 0.98 0.98 0.90 0.94 0.97 1.00 1.00 1.00 0.89 0.88 0.86

 The SME model elicitation for the node Contamination Control is summarized in Table 33.

Table 33. SME Probabilities and Confidence Bounds for the Node 

Contamination Control. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.05 0.00 0.01 0.00 0.05 0.03 0.03 0.00 0.00 0.00 0.07 0.08 0.09

not 0.95 0.95 1.00 0.99 1.00 0.95 0.97 0.97 1.00 1.00 1.00 0.93 0.92 0.91

 The SME model elicitation for the node Runway Length is summarized in Table 34. 

Table 34. SME Probabilities and Confidence Bounds for the Node 

Runway Length. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.10 0.15 0.10 0.15 0.10 0.15 0.13 0.03 0.05 0.07 0.08 0.17 0.18 0.20

not 0.90 0.85 0.90 0.85 0.90 0.85 0.87 0.97 0.95 0.93 0.92 0.83 0.82 0.80
44

 The average SME input for node Airport Issues is shown in Table 34. The following 

abbreviations are used in Table 35: app =approach, dep = departure, cont = contamination 

control and rw = runway. 

Table 35. Conditional Probability Table for the node Airport Issues.

app/dep yes yes yes yes no no no no

cont yes yes no no yes yes no no

rw 

length

yes no yes no yes no yes no

issue 0.99 0.61 0.71 0.28 0.82 0.19 0.59 0.01

not 0.01 0.39 0.29 0.72 0.18 0.81 0.41 0.99

 The SME model elicitation for the node Runway Assignment is summarized in Table 36. 

Table 36. SME Probabilities and Confidence Bounds for the Node 

Runway Assignment. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.05 0.05 0.01 0.05 0.01 0.05 0.04 0.02 0.00 0.00 0.01 0.07 0.08 0.09

not 0.95 0.95 0.99 0.95 0.99 0.95 0.96 0.98 1.00 1.00 0.99 0.93 0.92 0.91

 The SME model elicitation for the node Runway Collision Avoidance is summarized in 

Table 37. 

Table 37. SME Probabilities and Confidence Bounds for the Node 

Runway Collision Avoidance. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.00 0.01 0.01 0.01 0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.01 0.02 0.02

not 1.00 0.99 1.00 0.99 1.00 0.99 0.99 1.00 1.00 1.00 1.00 0.99 0.98 0.98
45

 The SME model elicitation for the node Contribution to Unstabilized Approach is summarized 

in Table 38. 

Table 38. SME Probabilities and Confidence Bounds for the Node 

Contribution to Unstabilized Approach. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.10 0.10 0.20 0.25 0.10 0.25 0.16 0.08 0.00 0.02 0.04 0.29 0.31 0.36

not 0.90 0.90 0.80 0.75 0.90 0.75 0.84 0.93 1.00 0.98 0.96 0.71 0.69 0.64

 The SME model elicitation for the node Lack of Current Weather Information is summarized 

in Table 39. 

Table 39. SME Probabilities and Confidence Bounds for the Node 

Lack of Current Weather Info. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99% 95% 90% 90% 95% 99%

Lo Lo Lo Hi Hi Hi

issue 0.10 0.01 0.01 0.10 0.01 0.10 0.06 0.05 0.00 0.00 0.00 0.14 0.16 0.19

not 0.90 0.99 0.99 0.90 0.99 0.90 0.94 0.95 1.00 1.00 1.00 0.86 0.84 0.81

 The average SME model elicitation for the node ATC Operational Issues is summarized in 

Table 40. The HFACS contribution to the CPT are considered separately. The following 

abbreviations are used in Table 40: hfacs = human factors assessment and classification system, 

rw = runway, cntrb = contribution, unstab = unstabilized, app = approach, and curr = current. 

Table 40. Transposed Conditional Probability Table for the node 

ATC Operational Issues. 

hfacs rw assign coll avoid cntrb to 

unstab app

lack of curr 

weather

ATC Ops 

Issues

No ATC 

Ops Issues

yes yes yes yes yes 1.0000 0.0000

yes yes yes yes no 0.8705 0.1295

yes yes yes no yes 0.5702 0.4298
46

yes yes yes no no 0.4311 0.5689

yes yes no yes yes 0.9039 0.0961

yes yes no yes no 0.7370 0.2630

yes yes no no yes 0.5006 0.4994

yes yes no no no 0.1530 0.8470

yes no yes yes yes 0.8761 0.1239

yes no yes yes no 0.6536 0.3464

yes no yes no yes 0.4172 0.5828

yes no yes no no 0.2086 0.7914

yes no no yes yes 0.8066 0.1934

yes no no yes no 0.6258 0.3742

yes no no no yes 0.1418 0.8582

yes no no no no 0.0147 0.9853

no yes yes yes yes 0.9925 0.0075

no yes yes yes no 0.7825 0.2175

no yes yes no yes 0.5125 0.4875

no yes yes no no 0.3875 0.6125

no yes no yes yes 0.8125 0.1875

no yes no yes no 0.6625 0.3375

no yes no no yes 0.4500 0.5500

no yes no no no 0.1375 0.8625

no no yes yes yes 0.7875 0.2125

no no yes yes no 0.5875 0.4125

no no yes no yes 0.3750 0.6250

no no yes no no 0.1875 0.8125

no no no yes yes 0.7250 0.2750

no no no yes no 0.5625 0.4375

no no no no yes 0.1275 0.8725

no no no no no 0.0133 0.9868

 The SME model elicitation for the node Unstabilized Approach is summarized in Table 41. 

Table 41. SME Probabilities and Confidence Bounds for the Node 

Unstabilized Approach. 

with automation issue

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

issue 0.30 0.25 0.25 0.50 0.25 0.50 0.36 0.13 0.03 0.11 0.15 0.57 0.61 0.69

not 0.70 0.75 0.75 0.50 0.75 0.50 0.64 0.87 0.97 0.89 0.85 0.43 0.39 0.31
47

with no automation issue

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

issue 0.01 0.01 0.10 0.15 0.01 0.15 0.07 0.07 0.00 0.00 0.00 0.18 0.20 0.25

not 0.99 0.99 0.90 0.85 0.99 0.85 0.93 0.93 1.00 1.00 1.00 0.82 0.80 0.75

 The SME model elicitation for the node Aircraft Automation Issues is summarized in Table 42. 

Table 42. SME Probabilities and Confidence Bounds for the Node 

Aircraft Automation Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

issue 0.01 0.01 0.10 0.15 0.01 0.15 0.07 0.07 0.00 0.00 0.00 0.18 0.20 0.25

not 0.99 0.99 0.90 0.85 0.99 0.85 0.93 0.93 1.00 1.00 1.00 0.82 0.80 0.75

 The SME model elicitation for the node Inappropriate Aircraft Operations is summarized in 

Table 43. 

Table 43. SME Probabilities and Confidence Bounds for the Node 

Inappropriate Aircraft Operations. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

issue 0.30 0.10 0.10 0.30 0.10 0.30 0.20 0.12 0.00 0.00 0.01 0.39 0.43 0.50

not 0.70 0.90 0.90 0.70 0.90 0.70 0.80 0.88 1.00 1.00 0.99 0.61 0.57 0.50

 The average SME model elicitation for the node Pilot Operational Issues is summarized in 

Table 44. The HFACS contribution to the CPT is considered separately. 
48

Table 44. Transposed Conditional Probability Table for the node 

Pilot Operational Issues. 

unstab 

app

pilot 

hfacs

inapp ac 

ops

auto

Pilot Ops 

Issues

No Pilot 

Ops 

Issues

yes yes yes yes 0.99 0.01

yes yes yes no 0.83 0.18

yes yes no yes 0.79 0.21

yes yes no no 0.63 0.38

yes no yes yes 0.81 0.19

yes no yes no 0.74 0.26

yes no no yes 0.50 0.50

yes no no no 0.39 0.61

no yes yes yes 0.59 0.41

no yes yes no 0.45 0.55

no yes no yes 0.46 0.54

no yes no no 0.15 0.85

no no yes yes 0.55 0.45

no no yes no 0.39 0.61

no no no yes 0.23 0.78

no no no no 0.01 0.99

 The SME panel developed blanket multiplication factors (mfac) for the HFACS contributors 

for RE events relative to those previously recorded for RI events. The SME elicitation for the 

ATC and pilot multiplicative factors is summarized in Table 45. 
49

Table 45. SME multiplicative factors and confidence bounds for HFACS contributions 

to RE relative to RI events. 

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

atc

hfacs

mfac

0.10 0.20 0.10 0.20 0.10 0.20 0.15 0.06 0.00 0.04 0.06 0.24 0.26 0.30

pilot

hfacs

mfac

0.80 1.10 1.0 1.0 0.80 1.10 0.98 0.13 0.65 0.73 0.77 1.00 1.00 1.00

 The revised marginal probabilities for the RE / ATC / HFACS contributions are summarized 

in Table 46. 

Table 46. RE / ATC / HFACS contributions. 

RE 

HFACS

issue not

Avg 

Mfac

issue not

ATC Cert 0.0100 0.9900 0.1500 0.0015 0.9985

ATC OTJ 0.0600 0.9400 0.1500 0.0090 0.9910

ATC 

State

0.0700 0.9300 0.1500 0.0105 0.9895

 The revised marginal probabilities for the RE / Pilot / HFACS contributions are summarized 

in Table 47. 

Table 47. RE / Pilot / HFACS contributions. 

RE

HFACS issue not

Avg 

Mfac issue not

Pilot 

Cert 0.0200 0.9800 0.9800 0.0196 0.9804

Pilot OTJ 0.0300 0.9700 0.9800 0.0294 0.9706

Pilot 

State 0.0400 0.9600 0.9800 0.0392 0.9608
50

 The average SME model elicitation for the node Pilot Error is summarized in Table 48. 

Table 48. Transposed Conditional Probability Table for the node 

Pilot Error.

airport 

issues

weather 

issues

atc

issues

pilot

issues

Pilot

Error

No Pilot 

Error

yes yes yes yes 0.80 0.20

yes yes yes no 0.31 0.69

yes yes no yes 0.65 0.35

yes yes no no 0.19 0.81

yes no yes yes 0.60 0.40

yes no yes no 0.16 0.84

yes no no yes 0.38 0.62

yes no no no 0.08 0.92

no yes yes yes 0.71 0.29

no yes yes no 0.16 0.84

no yes no yes 0.63 0.37

no yes no no 0.07 0.93

no no yes yes 0.48 0.52

no no yes no 0.03 0.97

no no no yes 0.32 0.68

no no no no 0.00 1.00

 
51

 The SME model elicitation for the node Mechanical Failure is summarized in Table 49. 

Table 49. SME Probabilities and Confidence Bounds for the Node 

Mechanical Failure. 

automation yes

SME1 SME2 SME3 SME4 Min Max Avg StDev 99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

mech

failure 0.2 0.25 N/A 0.30 0.20 0.30 0.25 0.05 0.12 0.15 0.17 0.33 0.35 0.38

no

failure 0.8 0.75 N/A 0.70 0.80 0.70 0.75 0.95 0.88 0.85 0.83 0.67 0.65 0.62

automation no

SME1 SME2 SME3 SME4 Min Max Avg StDev 99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

mech

failure 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.00 0.05 0.05 0.05 0.05 0.05 0.05

no

failure 0.95 0.95 0.95 0.95 0.95 0.95 0.95 1.00 0.95 0.95 0.95 0.95 0.95 0.95
52

 The SME model elicitation for the node RE Event Initiated is summarized in Table 50. 

Table 50. SME Probabilities and Confidence Bounds for the Node 

RE Event Initiated. 

Mechanical Failure yes Pilot Error yes

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

RE Event 0.90 0.99 0.99 0.95 0.90 0.99 0.96 0.04 0.85 0.87 0.89 1.00 1.00 1.00

No 

Event

0.10 0.01 0.01 0.05 0.10 0.01 0.04 0.96 0.15 0.13 0.11 0.00 0.00 0.00

Mechanical Failure yes Pilot Error no

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

RE Event 0.70 0.55 0.75 0.70 0.55 0.75 0.68 0.09 0.45 0.51 0.53 0.82 0.84 0.90

No 

Event

0.30 0.45 0.25 0.30 0.45 0.25 0.33 0.91 0.55 0.49 0.47 0.18 0.16 0.10

Mechanical Failure no Pilot Error yes

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

RE Event 0.60 0.40 0.40 0.55 0.40 0.60 0.49 0.10 0.22 0.29 0.32 0.66 0.69 0.75

No 

Event

0.40 0.60 0.60 0.45 0.60 0.40 0.51 0.90 0.78 0.71 0.68 0.34 0.31 0.25

Mechanical Failure no Pilot Error no

SME1 SME2 SME3 SME4 Min Max Avg StDev

99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

RE Event 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.00 0.01 0.01 0.01 0.01 0.01 0.01

No 

Event

0.99 0.99 0.99 0.99 0.99 0.99 0.99 1.00 0.99 0.99 0.99 0.99 0.99 0.99
53

 The SME model elicitation for the node Weather Issues is summarized in Table 51. 

Table 51. SME Probabilities and Confidence Bounds for the Node 

Weather Issues. 

SME1 SME2 SME3 SME4 Min Max Avg StDev 99%

Lo

95%

Lo

90%

Lo

90%

Hi

95%

Hi

99%

Hi

Issue 0.85 0.7 0.75 0.7 0.70 0.85 0.75 0.07 0.57 0.61 0.63 0.87 0.89 0.93

Not 0.15 0.3 0.25 0.3 0.30 0.15 0.25 0.93 0.43 0.39 0.37 0.13 0.11 0.07

8.0 Model Baseline Execution 

8.1 Runway Incursion Model 

The most recent SME session (July 2014) resulted in a fully vetted BBN model for both RI 

and RE events. Moreover, the SME panel elicitation of marginal and conditional probabilities has 

also been completed. Thus, all the ingredients for a fully vetted and fully populated set of baseline 

models have been obtained. In order to demonstrate how a BBN model would function, random 

values for all of the conditional probability tables have been inserted into the final RI model so 

that model operations can be simulated as shown in Figure 4. The node probabilities are shown 

overlaying the model structure from Figure 2. In this case, the marginal (leaf node) probabilities 

for all of the contributing factors have been set to zero, and the reaction time has been set to “long” 

(i.e., more than 8 seconds) for the collision scenario “crossing in front of departure”. The model 

indicates that the probability of an accident or near miss (red node at middle right) is about 22% 

for this random scenario. 
54

Figure 4. Sample RI Model Execution (baseline scenario). 
55

Figure 5. Sample RI Model Execution (worst-case scenario). 

 In Figure 5, the same example has been shown again, but with all the leaf node marginal 

probabilities set to 1 (all the issue are a certainty) and the reaction time has been set to short for 

the same collision scenario. In this random, worst-case, the probability of an accident or near miss 

is increased to about 75%.

8.2 Runway Excursion Model 

 Figure 6 illustrates a sample execution of the RE model with average probabilities employed, 

as described in section 6 of this report. The node probabilities are again shown as an overlay to 

the model structure presented in Figure 3. The sample execution shown in Figure 6 indicates that 

under the assumptions used to construct this model, there is about a 10% chance of an RE event. 
56

Figure 6. Sample RE Model Execution (average probabilities). 

9.0 Model Sensitivity Analysis 

9.1 Runway Incursion Model 

 Figure 7 illustrates a typical sensitivity analysis for the RI model. In order to accomplish this 

sensitivity analysis, the Hugin tool parametrically varies the strength of the marginal and 

conditional probability table values. Then sensitivity values are determined at the maximum 

strength, the minimum strength and at the baseline (or average) CPT values. In figure 7 below 

those nodes where blue coloring is observed indicate response sensitivity at the maximum CPT 

values, those with red coloring indicate response sensitivity at the minimum CPT values, and 

those with green coloring indicate response sensitivity at the average (or baseline) CPT values. 

In each case, the intensity of the observed colors indicate the strength of the observed response 

sensitivity. Under the assumptions present in the model (i.e., nodes, links and CPT values as 

previously discussed), the node Potential RI Event Severity is, as expected, sensitive to the node 

Primary Error State; however, only very limited sensitivity discrimination among the other nodes 

to the left is possible. Yet, some degree of sensitivity must exist for every node in the diagram. 
57

 Thus, in order to examine more fully the sensitivities across the entire diagram, Figure 7 

illustrates only the sensitivities for the node Primary Error State. The strongest sensitivities for 

this node are with respect to the Pilot Operational Issues, the Pilot Physical or Mental State, 

Comm Content Issues, and the Airport Layout. 

Figure 7. Sample RI model execution sensitivity analysis. 

9.2 Runway Excursion Model 

 Figure 8 illustrates a typical sensitivity analysis for the RE model. Under the assumptions 

present in the model (nodes, links and CPT as previously discussed), Figure 8 illustrates that an 

RE Event Initiated is mostly dependent upon mechanical failure and then upon pilot error, 

runway length and automation issues.
58

Figure 8. Sample RE model execution sensitivity analysis. 

10.0 Conclusions 

 The RUNSAFE Bayesian Belief Network (BBN) model for Runway Incursion (RI) and 

Runway Excursion (RE) events has been presented. Numerous considerations surrounding the 

process of developing the RI and RE models have been documented in this report. The resulting 

RUNSAFE model (both RI and RE event models) has been thoroughly reviewed by a Subject 

Matter Expert (SME) panel through multiple SME knowledge elicitation sessions. Numerous 

improvements to the model structure (definitions, node names, node states and the connecting link 

topology) were made by the SME panel. The structural details of the resulting RUNSAFE BBN 

models for RI and RE events have been documented within this report. A few sample executions 

of the final RI and RE models, using random conditional probability tables have been presented 

for the baseline and worst-case scenarios; the resulting probability of an accident or near miss 
59

increases substantially for the worst-case scenario, compared with the baseline scenario. Finally, 

a parameter sensitivity analysis for a given scenario was performed to show the risk drivers. 

11.0 Recommendations 

It is recommended that the model structures presented herein and the CPT values developed 

by the SME panel be validated by comparison to available data , be expanded to include the 

injection of technology products intended to improve runway safety, and that SME input be used 

to characterize the impact of these technology products. It is also recommended that the resulting 

BBN for RI and RE events be used by NASA to generically model the causes of RI and RE events 

and to assess the effectiveness of technology products being developed under NASA funding. 

12.0 References List

1. Aviation Safety Program website, http://www.aeronautics.nasa.gov/programs_avsafe.htm 

(accessed 10/22/2014), now the Airspace Operations and Safety Program website, 

http://www.aeronautics.nasa.gov/programs-aosp.htm (accessed 3/6/2015). 

2. NextGEN, The Federal Aviation Administration, NextGEN website, 

http://www.faa.gov/nextgen/, (accessed 3/6/2015).

3. http://www.asias.faa.gov/pls/apex/f?p=100:32:0::NO::P32_REGION_VAR:1 – definition 

of RI event and severity categories (accessed 10/22/2014). 

4. Order 7050.1, Runway Safety Program, The Federal Aviation 

Administration,http://www.faa.gov/documentLibrary/media/Order/FAA_Order_7050.1B.

pdf (accessed 10/22/2014) – RI and RE event definitions, RI event severity classifications 

5. Green, Lawrence L., Analysis of Runway Incursion Data, AIAA 2013-4400, presented at 

13th the American Institute of Aeronautics and Astronautics (AIAA) / Aviation 

Technology, Integration, and Operations (ATIO) Conference, Los Angeles, CA, August 

12-14, 2013.

6. Green, Lawrence L., Development, Validation and Execution of a Bayesian Network for 

Runway Incursions, AIAA 2014-2158, presented at the 14th American Institute of 

Aeronautics and Astronautics (AIAA) / Aviation Technology, Integration, and 

Operations (ATIO) Conference, Atlanta, GA, June 16-21, 2014. 

7. Green, Lawrence L., Development of a Bayesian Belief Network Runway Incursion and 

Excursion Model, Proceedings of the American Society for Engineering Management 
60

2014 International Annual Conference, S. Long, E-H. Ng, and C. Downing eds., Virginia 

Beach, VA, October 16-17, 2014. 

8. Second Workshop on the Investigation and Reporting of Incidents and Accidents, IRIA 

2003, NASA/CP-2003-212642, Compiled by Kelly J. Hayhurst and C. Michael 

Holloway, Langley Research Center, Hampton, Virginia, September 2003. 

9. Luxhøj, J. T., Shih, A. T., Jones, S. M., Ancel, E., and Reveley, M. S., "Safety Risk 

Knowledge Elicitation in Support of Aeronautical R&D Portfolio Management: A Case 

Study," American Society for Engineering Management (ASEM) 33rd International 

Annual Conference, Virginia Beach, VA, Oct 17-20, 2012. 

10. Luxhøj, J. T., Jalil, M., and Jones, S. M., "A Risk-Based Decision Support Tool for 

Evaluating Aviation Technology Integration in the National Airspace System," 

Proceedings of the AIAA's 3rd Annual Aviation Technology, Integration and Operations 

(ATIO) Technical Forum, Denver, CO, November, 17-19. 2003. 

11. Luxhøj, J. T., "Probabilistic Causal Analysis for System Safety Risk Assessment in 

Commercial Air Transport," Proceedings of the Workshop on Investigating and 

Reporting of Incidents and Accidents (IRIA), Sept 16-19, Williamsburg, VA, p.17 

NASA/CP-2003-212642, 2003. 

12. Ancel, E., and Shih, A. T., "The Analysis of the Contribution of Human Factors to the Inflight Loss of Control Accidents," 12th AIAA Aviation Technology, Integration, and 

Operations (ATIO) Conference, Indianapolis, IN, Sept 17-19, 2012. American Institute of 

Aeronautics and Astronautics 9 

13. Shih, A. T., Ancel, E., and Jones, S. M., "Object-oriented Bayesian Networks (OOBN) 

for Aviation Accident Modeling and Technology Portfolio Impact Assessment," 

American Society for Engineering Management (ASEM) 33rd International Annual 

Conference, Oct 17-20, 2012. 

14. Runway Transgressions at Non-Towered and Tower Closed Airports, NASA Aviation 

Safety Reporting System (ASRS), Publication 61, July 2003. 

15. The Boeing Company, Statistical Summary of Commercial Jet Airplane Accidents, 2013. 

16. Cardosi, K., Chase, S., and Eon, D., “Runway Safety”, U.S. Department of 

Transportation, Volpe Center, Cambridge, MA, Feb. 12, 2010. 

17. Torres, K. R., Metscher, D. S., and Smith, M., “A Correlational Study of the Relationship 

between Human Factor Errors and the Occurrence of Runway Incursions”, International 

Journal of Professional Aviation Training & Testing Research, Vol. 5, (1) © 2011 

Published by Professional Aviation Board of Certification. 

18. The Federal Aviation Administration, “Wrong Runway Departures”, Aviation Safety 

Information Analysis and Sharing (ASIAS), Office of Aviation Safety, AVS–4 July 2007. 
61

19. Air Line Pilots Association International, “White Paper: Runway Incursions A Call to 

Action”, March 2007. 

20. Runway Incursion database website, The Federal Aviation Administration, 

http://www.asias.faa.gov/pls/apex/f?p=100:28:0::NO:28:: (accessed 10/22/2014). 

21. The FAA Aviation Safety Information Analysis and Sharing (ASIAS) System website, 

http://www.asias.faa.gov/pls/apex/f?p=100:1: (accessed 3/9/2015) 

22. Federal Aviation Administration Air Traffic Activity System (ATADS) website: 

https://aspm.faa.gov/opsnet/sys/main.asp (access Nov. 2014). 

23. Chapman, R., “Runway Incursions 2000-201: Is Safety Improving?” East Michigan 

University, Retrieved from http://www.miqrogroove.com/writing/runwayincursions/fulltext.pdf, Copyright © 2010 by Robert Chapin. 

24. A Study of Runway Excursions from a European Perspective, NLR-CR-2010-259, May 

2010, NLR Air Transport Safety Institute, Anthony Fokkerweg 2, 1059 CM Amsterdam, 

P.O. Box 90502, 1006 BM Amsterdam, The Netherlands, Telephone +31 20 511 35 00, 

Fax +31 20 511 32 10, Web site: http://www.nlr-atsi.nl. 

25. Hindsight 12, http://www.eurocontrol.int/sites/default/files/publication/files/101112-

HS12.pdf (accessed May 2014). 

26. Final Runway excursions Part 1: A worldwide review of commercial jet aircraft runway 

excursions, TRANSPORT SAFETY REPORT Aviation Research and Analysis Report –

AR-2008-018(1) Australian Transport Safety Board.

27. AIRPORT COOPERATIVE RESEARCH PROGRAM ACRP REPORT 3 Analysis of 

Aircraft Overruns and Undershoots for Runway Safety Areas, Transportation Research 

Board of the National Academies. 

28. Minimising the likelihood and consequences of runway excursions, TRANSPORT 

SAFETY REPORT Aviation Research and Analysis Report – AR-2008-018(1) Runway 

excursions Part 2: Australian Transport Safety Board. 

29. Running out of runway Analysis of 35 years of landing overrun accidents, National 

Aerospace Laboratory NLR-TP-2005-498.

30. Report on the Design and Analysis of a Runway Excursions Database, A Research 

Project Conducted for the Flight Safety Foundation by Safety Management Specialties, 

May 26, 2009, 5648 NE Caddis Drive, Hillsboro, Oregon 97124, (877) 767-7679.

31. The Strategic Runway Safety Plan, Report to Congress Pursuant to FAA Modernization 

and Reform Act of 2012 (Pub. L. No. 112-95, Section 314(a), U.S. Department of 

Transportation, FAA, November 2012. 
62

32. Runway Excursion Modeling and Analysis: A Web-Based Geographic Information 

System Solution.” Professor William Ziegler advised the student team, 

http://emerald.ts.odu.edu/Apps/FAAUDCA.nsf/Binghamton%20University.pdf?OpenFile

Resource, 2012,(accessed May 2014). 

33. Los Angeles International Airport North Airfield Safety Study, Final Report Prepared by: 

Dr. Arnold Barnett (Chairman), et. al. May 2010. 

34. A Method for Rating the Severity of Runway Incursions, Cardosi, Kim , Hannon, Daniel 

and Sheridan, Thomas, U.S. Department of Transportation, Volpe Center, 55 Broadway, 

Cambridge, MA. 

35. Goodheart, Benjamin Jeffery, Identification of Causal Paths and Prediction of Runway 

Incursion Risk Using Bayesian Belief Networks, ACRP 11-04 Graduate Research Award, 

Embry-Riddle Aeronautical University, Daytona Beach, FL October 2, 2013for TRB 

2014 Annual Meeting.

36. Reducing the Risk of Runway Excursions, Report of the Runway Safety Initiative, Flight 

Safety Foundation, MAY 2009. 

37. Microsoft Corporation, Excel website: http://products.office.com/en-us/excel (accessed 

Nov. 2014). 

38. The Stat-Ease, Inc website: http://www.statease.com/ (accessed 2/11/2013). 

(renumbered) 

39. HUGIN Expert A/S website: http://www.hugin.com/ (accessed Nov. 2014) 

40. Green, L. L., “Uncertainty Analysis of Historical Hurricane Data”, AIAA-2007-1101, 

2007.

41. Green, L. L.; Rickman, S. L.; Remark, B. J.; Vander Kam, J. C.; Kowal, J.; Johnson, K. 

L.; and Bouslog, S. A.: NASA/TM-2014-218262, May 2014. 

42. Green, L. L.: The Challenges of Credible Thermal Protection System Reliability 

Quantification, Paper #77, Session 8B in Cross-Cutting Technologies III, The 10th 

International Planetary Probe Workshop (IPPW-10), San Jose, CA June 2013. 

43. Green, L. L.: Uncertainty Analysis of Historical Hurricane Data. AIAA 2007-1101. 

Presented at the 45th AIAA Applied Aerospace Sciences Conference and Exhibit, Reno, 

NV, January 8-11, 2007. 

44. Green, L. L., Cruz, J.: Uncertainty Analysis for a Jet Flap Airfoil. Paper Number SISO06F-SIW-010. Published at the Simulation Interoperability Standards Organization, 2006 

Fall Simulation Interoperability Workshop, Orlando, FL, September 10-15, 2006 

45. “Attempted Takeoff From Wrong Runway Comair Flight 5191, Bombardier CL-600-

2B19, N431CA, Lexington, Kentucky, August 27, 2006” National Transportation Safety

Board ACCIDENT REPORT NTSB/AAR-07/05, PB2007-910406.
63

46. Wiegmann, Douglas A. and Shappell, Scott A., A Human Error Approach to Aviation 

Accident Analysis, Ashgate Publishing Limited, Burlington VT, 2003. (also, 

http://www.hfacs.com, accessed Nov. 14, 2013) 

47. Chow, Amy and Gulding, John. Capacity Variation Algorithms for Simulation Modeling 

and Performance Analysis, AIAA 2013-4355, presented at 13th the American Institute of 

Aeronautics and Astronautics (AIAA) / Aviation Technology, Integration, and 

Operations (ATIO) Conference, Los Angeles, CA, August 12-14, 2013. 

48. Rudisill, Marianne, “Line Pilots’ Attitudes About and Experience with Flight Deck 

Automation: Results of an International Survey and Proposed Guidelines, Published in: 

(1995) Proceedings of the Eighth International Symposium on Aviation Psychology. 

Columbus, OH: The Ohio State University Press. 

49. SKYbrary Runway Excursion website, 

http://www.skybrary.aero/index.php/Runway_Excursion, accessed Nov. 14, 2013. 
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Memorandum

 4. TITLE AND SUBTITLE

Development and Execution of the RUNSAFE Runway Safety Bayesian 

Belief Network Model 

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Green, Lawrence L.

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-20556

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 03

Availability: NASA STI Program (757) 864-9658

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

One focus area of the National Aeronautics and Space Administration (NASA) is to improve aviation safety. Runway safety is 

one such thrust of investigation and research. The two primary components of this runway safety research are in runway 

incursion (RI) and runway excursion (RE) events. These are adverse ground-based aviation incidents that endanger crew, 

passengers, aircraft and perhaps other nearby people or property. A runway incursion is the incorrect presence of an aircraft, 

vehicle or person on the protected area of a surface designated for the landing and take-off of aircraft; one class of RI events 

simultaneously involves two aircraft, such as one aircraft incorrectly landing on a runway while another aircraft is taking off

from the same runway. A runway excursion is an incident involving only a single aircraft defined as a veer-off or overrun off 

the runway surface. 

15. SUBJECT TERMS

Aviation safety; Bayesian belief network; Runway excursion; Runway incursion; Runway safety

18. NUMBER

 OF 

 PAGES

75

19b. TELEPHONE NUMBER (Include area code)

(757) 864-9658

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

 284848.02.01.07.04 

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA-TM-2015-218705

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 05 - 2015