January 2011 

NASA/TP–2011-217045 

Simulator Evaluation of Runway Incursion 

Prevention Technology for General Aviation 

Operations 

Denise R. Jones and Lawrence J. Prinzel III 

Langley Research Center, Hampton, Virginia 
NASA STI Program . . . in Profile

 Since its founding, NASA has been dedicated to 

the advancement of aeronautics and space science. 

The NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

 The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. It 

collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI program 

provides access to the NASA Aeronautics and Space 

Database and its public interface, the NASA Technical 

Report Server, thus providing one of the largest 

collections of aeronautical and space science STI in 

the world. Results are published in both non-NASA 

channels and by NASA in the NASA STI Report 

Series, which includes the following report types: 

 TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but having 

less stringent limitations on manuscript length 

and extent of graphic presentations. 

 TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis. 

 CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

 CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or co-sponsored by NASA. 

 SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest. 

 TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and 

technical material pertinent to NASA’s mission. 

 Specialized services also include creating custom 

thesauri, building customized databases, and 

organizing and publishing research results. 

 For more information about the NASA STI 

program, see the following: 

 Access the NASA STI program home page at 

http://www.sti.nasa.gov

 E-mail your question via the Internet to 

help@sti.nasa.gov

 Fax your question to the NASA STI Help Desk 

at 443-757-5803 

 Phone the NASA STI Help Desk at 

443-757-5802 

 Write to: 

 NASA STI Help Desk 

 NASA Center for AeroSpace Information 

 7115 Standard Drive 

 Hanover, MD 21076-1320
National Aeronautics and 

Space Administration 

Langley Research Center 

Hampton, Virginia 23681-2199 

 

January 2011 

NASA/TP–2011-217045 

Simulator Evaluation of Runway Incursion 

Prevention Technology for General Aviation 

Operations 

Denise R. Jones and Lawrence J. Prinzel III 

Langley Research Center, Hampton, Virginia 
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 

Acknowledgments 

Enter acknowledgments here, if applicable. 

The authors would like to express their appreciation to Lockheed Martin Mission 

Services for their support throughout all phases of testing and data analysis. 

The use of trademarks or names of
iii

Table of Contents 

Abstract ......................................................................................................................................................... 1

Introduction ................................................................................................................................................... 1

System Description ....................................................................................................................................... 2

Simulation Facility .................................................................................................................................... 2

Flight Deck Simulator ........................................................................................................................... 2

Air Traffic Control Simulation ............................................................................................................. 3

Research Displays ..................................................................................................................................... 3

Baseline (B) .......................................................................................................................................... 3

Baseline with Surface Map and Ownship (BMO) ................................................................................ 4

Baseline with Surface Map and Ownship and Traffic (BMOT) ........................................................... 4

Baseline with Audible Incursion Alerts (BA) ....................................................................................... 4

Baseline with Audible Incursion Alerts, Surface Map and Ownship (BAMO) .................................... 5

Baseline with Incursion Alerts, Surface Map and Ownship, and Traffic (BAMOT) ........................... 5

Baseline with Perspective Surface Map (BRIPS) ................................................................................. 5

Runway Incursion Alerting ....................................................................................................................... 6

Runway Safety Monitor ........................................................................................................................ 6

PathProxTM ............................................................................................................................................ 6

Alerting Displays .................................................................................................................................. 6

Test Method .................................................................................................................................................. 7

Test Matrix ................................................................................................................................................ 7

Rare Event Testing Phase ..................................................................................................................... 8

Usability Testing Phase ......................................................................................................................... 8

Runway Incursion Scenarios ..................................................................................................................... 9

Scenario 1 – arrival/takeoff hold ........................................................................................................... 9

Scenario 2 – departure/intersection departure ..................................................................................... 10

Scenario 3 – arrival/departure ............................................................................................................. 10

Scenario 4 – departure/departure ........................................................................................................ 11

Scenario 5 – taxi crossing/departure ................................................................................................... 11

Procedure ................................................................................................................................................ 12

Evaluation Pilots ..................................................................................................................................... 12

Results ......................................................................................................................................................... 13

Rare Event Testing Phase ....................................................................................................................... 13

Usability Study Phase ............................................................................................................................. 15

Quantitative Results ............................................................................................................................ 15

Qualitative Results .............................................................................................................................. 20

Conclusions ................................................................................................................................................. 21

References ................................................................................................................................................... 23

Appendix A: Rare Event Test Matrix ........................................................................................................ 24

Appendix B: Usability Study Test Matrix ................................................................................................. 27

Appendix C: Rare Event Case Lists ........................................................................................................... 29

Appendix D: Usability Study Case Lists ................................................................................................... 31

Appendix E: Post Run Questionnaires ....................................................................................................... 34

Appendix F. Rare Event Run Questionnaire Results ................................................................................. 38

Appendix G: Usability Study Display Evaluation Questionnaire Results ................................................. 45

Appendix H: Usability Study Detection Algorithm Evaluation Questionnaire Results ............................ 56

Appendix I: Final Semi-Structured Interview Questionnaire Results ........................................................ 59

Appendix J: Paired Comparisons Questionnaire ....................................................................................... 74
iv

Acronyms and Symbols

α Alpha, probability of Type 1 error 

ADS-B Automatic Dependent Surveillance-Broadcast 

AGL Above Ground Level 

AMASS Airport Movement Area Safety System 

ANOVA Analysis of Variance 

ASDE-3 Airport Surface Detection Equipment Model 3 

ASDE-X Airport Surface Detection Equipment Model X 

ATC Air Traffic Control 

B Baseline 

BA Baseline with Audible Incursion Alerts 

BAMO Baseline with Surface Map and Ownship and Audible Incursion Alerts 

BAMOT Baseline with Surface Map and Ownship, Traffic, and Incursion Alerts 

BMO Baseline with Surface Map and Ownship 

BMOT Baseline with Surface Map and Ownship and Traffic 

BRIPS Baseline with Perspective Surface Map 

C-206 Cessna 206 

EFB Electronic Flight Bag 

EP Evaluation Pilot 

FAA Federal Aviation Administration 

FBO Fixed Based Operator 

F F-ratio, variance ratio 

FY Fiscal Year 

GA General Aviation 

GS Ground Speed 

IFD Integration Flight Deck 

IMC Instrument Meteorological Conditions 

KRNO Reno/Tahoe International Airport 

LaRC Langley Research Center 

MANOVA Multivariate Analysis of Variance 

N/A Not Applicable 

NASA National Aeronautics and Space Administration 

NTSB National Transportation Safety Board 

OVR Overview 

p Probability of occurrence of an event 

RD Research Display 

RIPS Runway Incursion Prevention System 

RSM Runway Safety Monitor 

RWSL Runway Status Lights 

SA Situation Awareness 

SART Situation Awareness Rating Technique 

TA Terrain Awareness 

TCAS Traffic Alert and Collision Avoidance System 

TLX Task Load Index 

VFR Visual Flight Rules 

VMC Visual Meteorological Conditions 
1

Abstract 

A Runway Incursion Prevention System (RIPS) has been designed under previous research to 

enhance airport surface operations situation awareness and provide cockpit alerts of potential runway 

conflict, during transport aircraft category operations, in order to prevent runway incidents while also 

improving operations capability. This study investigated an adaptation of RIPS for low-end general 

aviation operations using a fixed-based simulator at the National Aeronautics and Space Administration 

(NASA) Langley Research Center (LaRC). The purpose of the study was to evaluate modified RIPS 

aircraft-based incursion detection algorithms and associated alerting and airport surface display concepts 

for low-end general aviation operations. This paper gives an overview of the system, simulation study, 

and test results. 

Introduction 

Runway incursions are a serious aviation safety hazard, particularly for general aviation (GA) 

operations. According to the Federal Aviation Administration (FAA) (FAA, 2007), during the four year 

period from fiscal year (FY) 2003 through FY 2006, there were approximately 250 million aircraft 

operations and 1,306 runway incursions reported at United States towered airports – approximately 5.2 

runway incursions for every one million operations. General aviation was involved in 72 percent of these 

incursions but only 55 percent of the operations. Eighty-two percent of the most severe incursions (98 of 

120 incursions) involved at least one GA aircraft. Four incursions resulted in collisions, with one of these 

collisions involving GA aircraft. These statistics do not consider incidents that occur at non-towered 

airports. 

Current FAA initiatives are targeting reductions in the severity and rate of runway incursions by 

implementing a combination of technology, infrastructure, procedural, and training interventions (FAA, 

2007). These solutions include Airport Surface Detection Equipment Model 3 (ASDE-3) and ASDE 

Model X (ASDE-X) radar; Airport Movement Area Safety System (AMASS); multilateration systems; 

in-pavement loops; Runway Status Lights (RWSL); enhanced controller training; airport surface 

operations advisory circulars; improved airport markings; improved pilot education, training, and 

awareness; and revised pilot/controller communications phraseology. These efforts target improved 

awareness and enhanced surveillance, but do not include on-board technology solutions for the flight 

deck. 

The National Transportation Safety Board (NTSB) considers runway incursions to be a serious 

aviation safety hazard, listing runway incursion prevention as one of their “most wanted” transportation 

safety improvements (NTSB, 2007). The NTSB specifically recommends that the FAA implement 

technology that “give immediate warnings of probable collisions/incursions directly to flight crews in the 

cockpit” (NTSB, 2000). In response to this recommendation, the FAA has begun to research the concept 

of transmitting ground-generated incursion alerts to aircraft and vehicles. 

NASA developed a Runway Incursion Prevention System (RIPS) for commercial and business 

transport aircraft operations to improve airport safety by providing surface situation awareness (SA) 

information and guidance cues, and alerts of runway conflicts and route deviations directly to the flight 

crew. The system was evaluated in several flight tests and simulation studies (Jones, Quach, and Young, 

2001; Jones, 2002; Jones, 2005). 

In the present work, RIPS was adapted for small-aircraft, GA operations. A piloted simulation study 

was conducted at NASA LaRC to evaluate RIPS for low-end GA operations that focused on evaluation of 

incursion detection algorithms and cockpit display concepts. This paper presents an overview of the 

system, description of the simulation study, and test results. 
2

System Description 

Simulation Facility 

Flight Deck Simulator 

NASA LaRC’s Integration Flight Deck (IFD), normally used as a transport-category, fixed-based, 

high-fidelity, flight simulator, was adapted for this study because its visual, tactile, and audio capabilities 

provided the highest level of fidelity at LaRC toward meeting the research objectives (e.g., visual traffic 

acquisition). The IFD was configured to emulate a Cessna 206 (C-206) GA aircraft (herein referred to as 

the ownship). A six-degree-of-freedom, non-linear, simulation model of the C-206 and representative 

control-force and braking models were used. The left throttle was utilized to control the aircraft power 

while the right throttle was utilized to control the fuel mixture. In order to avoid distractions, extraneous 

displays and gauges were covered or turned off during data collection. 

As shown in Figure 1, an electronic research display (RD) was installed on the instrument panel 

directly in front of the left seat and control yoke. The RD was composed of two 10.4.inch liquid crystal 

displays and was capable of displaying two separate digital displays, side-by-side. An electronic flight 

bag (EFB) display, located to the left of the RD, was used to present the airport surface map display 

concepts described below. This display measured 10.4 inches diagonally with a resolution of 1280 x 

1024 pixels. The collimated out-the-window scene provided a 200 degree horizontal by 40 degree 

vertical field of view at 26 pixels per degree resolution. 

This configuration was used in a previous simulation, and feedback from the subjects of that study 

indicated that the simulation fidelity and cockpit environment were sufficiently representative of a C-206 

aircraft (Bartolone et al, 2005). 

Traffic position data were “broadcast” at a 1 Hz rate. No additional latency or surveillance data 

inaccuracies were applied. Ownship position data were updated at 20 Hz. 

Figure 1. IFD simulator cockpit configuration and displays. 

Research 

Display 

EFB 
3

Air Traffic Control Simulation 

Approach and tower air traffic control (ATC) instructions and pilot requests and replies were used in 

the simulation to increase the simulation fidelity of the terminal area environment and provide normal 

pilot workload demands during the study. All ATC and other aircraft radio messages were pre-recorded 

using different voices. The messages were then played through the flight deck speaker system when the 

ownship and simulated traffic reached specified locations. The subject pilots were asked to provide radio 

replies, when requested by the pre-recorded ATC messages, as per normal operating procedures. 

Research Displays 

This simulation study was designed for low-end, GA aircraft; therefore, standard round dial 

instrumentation was used as the “baseline” display concept around which various airport surface map 

formats and alerting concepts were experimentally evaluated. The genesis and background for the RIPS 

concepts are described in detail from previous research (Jones, et al, 2001; Jones, 2002; and Jones, 2005). 

The map format and alerting concepts were intended to identify which RIPS elements from previous 

research, as well as new elements, are most applicable and necessary to prevent runway incursion 

accidents and incidents for low-end, GA aircraft operations. 

The round-dial displays were shown on the RD located in front of the evaluation pilot (EP) (see Flight 

Deck Simulator section). 

When dictated by experimental condition, an airport surface map was displayed on the EFB located to 

the left of the round dial display. The surface map was generated using a Reno/Tahoe International 

(KRNO) airport geographic database developed to RTCA standards (RTCA, 2001). The map scale was 

set to 2.5 nm for the airborne scenarios and 1.5 nm for the ground based scenarios and was not pilotselectable. Audible alerts sounded through the flight deck speaker system. 

Seven display conditions were evaluated during the course of the study as described below. 

Baseline (B) 

The Baseline (B) display condition consisted of a set of six instruments (airspeed indicator, attitude 

indicator, altimeter, vertical speed indicator, directional gyro, and turn coordinator) plus manifold 

pressure and instrument landing system data. All instruments were three inches in diameter and 

configured on the RD as shown in Figure 2. 

Figure 2. Baseline display condition. 
4

Baseline with Surface Map and Ownship (BMO) 

The BMO display condition consisted of the baseline round dials displayed on the RD with the 

addition of a plan view surface map displayed on the EFB (Figure 3). This version of the surface map 

displayed an airport layout along with ownship position. Neither traffic nor ATC instructions (such as 

assigned taxi route) were shown. Incursion alerting was not part of this condition. 

Baseline with Surface Map and Ownship and Traffic (BMOT) 

The BMOT display condition was equivalent to the BMO condition but with the addition of traffic on 

the surface map (Figure 4). It was assumed that traffic was reported by an Automatic Dependent 

Surveillance-Broadcast (ADS-B) system. All traffic positions were reported without any positional 

inaccuracy (albeit at only a 1 Hz update rate). No data drop-outs, false reporting, or un-equipped traffic 

were simulated. 

Traffic was displayed as dark blue chevrons when on the ground and cyan chevrons when airborne. 

The direction of travel was indicated by the pointed end of the chevron. A circular symbol was used 

when traffic was traveling less than six knots, since the direction of travel could not be reliably estimated 

from just the simulated broadcast of position. Incursion alerting was not part of this condition. 

Figure 3. BMO display condition. Figure 4. BMOT display condition. 

Baseline with Audible Incursion Alerts (BA) 

The BA display condition consisted of the baseline round dials displayed on the RD and audible 

runway incursion alerts that sounded over the flight deck speaker when a potential conflict was detected. 

The detection method used to generate the alert is described in detail in the Runway Incursion Alerting 

section. 
5

Baseline with Audible Incursion Alerts, Surface Map and Ownship (BAMO) 

The BAMO display condition was equivalent to the BMO condition with the addition of audible 

runway incursion alerts. 

Baseline with Incursion Alerts, Surface Map and Ownship, and Traffic (BAMOT) 

The BAMOT display condition was equivalent to the BMOT condition with the addition of both 

audible and visual runway incursion alerts (Figure 5). The alerting displays and detection method are 

described in detail in the Runway Incursion Alerting section. 

Baseline with Perspective Surface Map (BRIPS) 

The BRIPS display condition consisted of the baseline round dials displayed on the RD with a 

perspective surface map displayed on the EFB (Figure 6). The map graphically depicted a perspective, 

track-up airport layout with current ownship and traffic locations and incursion alerts, which are the same 

information presented with the plan view BAMOT condition. ATC instructions, including the approved 

taxi route and hold short locations, were sent via a simulated data-link and automatically loaded and 

depicted as a magenta route on the surface map and shown as an alpha-numeric string. Audible incursion 

alerts were also sounded. 

Audible route deviation and crossing hold alerts were also generated. Route deviation alerts were 

generated if ownship left its assigned path during taxi. Crossing hold alerts were generated if ownship 

crossed a hold line without clearance. 

 Figure 5. BAMOT display condition Figure 6. BRIPS display condition with 

 with warning alert. caution alert. 
6

Runway Incursion Alerting 

Two different incursion detection algorithms, Runway Safety Monitor and PathProxTM, were 

evaluated during the simulation study. Both algorithms were originally developed for large commercial 

and business transport aircraft operations and were modified for the low-end GA application reported 

herein. 

Runway Safety Monitor 

The Runway Safety Monitor (RSM) incursion detection algorithm (Green, 2006) uses a generic 

approach for detecting and generating incursion alerts. The RSM monitors traffic that enters a threedimensional virtual protection zone around the runway that is being used by ownship. Incursion detection 

is based on the operational state of ownship and traffic, as well as other criteria, including separation and 

closure rate, to avoid false alerts. Identification, position, and altitude data are used to track the traffic in 

the protection zone. Traffic data projections are calculated within RSM since, from flight test experience, 

reliable position updates are not received at consistent intervals. RSM generates a Warning alert, which 

occurs when a runway incursion is detected and evasive action is required to avoid a potential collision. 

Information provided with each alert includes identification of the incurring traffic and separation 

distance to potential conflict. 

Modifications were made to RSM for this study to enable incursion detection and alerting for low-end 

GA operations. Ownship parameters were defined for the specifications and dimensions of the C-206 

aircraft. Improvements and refinements were made to the RSM alerting criteria that consider the 

availability of data, aircraft characteristics, and specific separation requirements for GA. For example, 

different minimum separation distances and aircraft land/rollout distances typically apply for GA aircraft, 

representative of a C-206. Since C-206 aircraft may travel slower than larger commercial aircraft, the 

minimum alerting distance can be less. See Green, et al. 2009 for detailed specifications for RSM alerting 

criteria for both low-end GA and non-GA aircraft. 

PathProxTM 

The PathProxTM detection algorithm (Cassell et al, 2003) is designed to handle over 40 specific 

runway incursion scenarios. Alerts are issued based on the states of the ownship and traffic and on 

conditions including position, speed, and track angle. PathProxTM generates two types of alerts analogous 

to the Traffic Alert and Collision Avoidance System (TCAS) approach. A Caution alert informs the flight 

crew of a potential incursion or an incursion where the conflict does not yet require evasive action. The 

crew can take evasive action, however, at their discretion. PathProxTM also generates Warning alerts 

when immediate evasive action is required. Information provided with each alert includes identification 

of the incurring traffic, the associated runway, and separation distance between the traffic and ownship. 

Alerting Displays 

Incursion alerts could be presented to the flight crew visually on the surface map and/or audibly. The 

visual and audible alert phraseology were identical. The alert phrases were designed to provide 

descriptive information regarding the location of the conflicting traffic. It was postulated that providing 

more details about the location of the incurring traffic would provide additional situation awareness, 

particularly during the condition when only audible alerts were provided. The incursion alert phrases 

developed for this study are listed in Table 1. As shown in Figures 5 and 6, the textual forms of these 

alerts were presented on the surface map. Also, the traffic symbol representing the conflicting traffic was 

enlarged, changed color (yellow for Caution and red for Warning), and was highlighted by a target 

designator box. The identification tag was also highlighted. In the event that the incurring traffic symbol 

was not shown because of the display scale, a symbol was pegged on the edge of the display in the 

direction of the traffic on the perspective surface map only. The estimated distance to conflict (in feet) 
7

was shown beneath the ownship symbol. An example of a Warning alert for a scenario where both 

ownship and traffic are departing on intersecting runways is shown in Figure 5. An example of a Caution 

alert for a scenario where ownship is on approach to a runway with traffic in position and hold for 

departure is shown in Figure 6. 

Table 1. Incursion Alert Phrases 

Warning, Traffic 34R 

Caution, Traffic 34R 

Warning, Traffic 25 

Caution, Traffic 25 

Warning, Traffic Departing 34R 

Caution, Traffic Departing 34R 

Warning, Traffic Departing 25 

Caution, Traffic Departing 25 

Warning, Traffic Approaching 34R 

Caution, Traffic Approaching 34R 

Warning, Traffic Approaching 25 

Caution, Traffic Approaching 25 

Test Method 

The testing was conducted in two phases, rare event and usability. 

The rare event phase was designed to evaluate pilot reaction to a runway incursion event with a given 

display condition and incursion detection algorithm without expectation on the part of the subject 

(Newman and Foyle, 2003; Foyle and Hooey, 2003). This effect was created by flying 18 runs under 

“nominal” (i.e., no incursions) followed by the last run in this phase being a runway incursion event. The 

evaluation pilot (EP) was not told that this last run concluded the experiment phase or that a runway 

incursion was planned. 

Following the “rare event” phase, a usability phase was conducted. 

 Test Matrix 

The test matrix conditions are identified in Table 2. A full-factorial evaluation across 3 alerting 

conditions (no alerts, audible alerts, and audible plus visual alerts) and 4 display conditions (Baseline, 

plan view map, plan view map with traffic, and perspective map with traffic) was considered but rejected 

for practical and expediency reasons as follows: 

 Evaluating visual alerts in the baseline and baseline with plan view map (ownship only) 

conditions was rejected as not applicable (N/A) because the map did not contain any traffic 

information; therefore, the value of adding and testing visual alerting for these cases was 

considered to be small. 

 It was assumed that if traffic was provided on a surface map, both audible and visual alerts 

would be displayed; therefore, the conditions with audible-only alerts were not evaluated. 

 The experiment matrix then contains a 2x2 evaluation of the influence of a plan view map (on 

and off, with ownship position only) and audible alert (on and off). 
8

 The effect of a surface map displaying traffic in addition to ownship was evaluated with no 

incursion alerts and with combined audible and visual alerts. 

 Finally, the perspective surface map displaying ownship and traffic along with ATC 

instructions (such as the approved taxi route and hold short locations) was considered an 

advanced condition; therefore, only the most advanced display alerting condition (audible and 

visual alerts) was evaluated with this condition. 

Table 2. Display Test Conditions 

No Alert Audible Alert Audible & 

Visual Alert 

Baseline (B) B BA N/A 

B + plan view map with ownship BMO BAMO N/A 

B + plan view map with ownship & traffic BMOT Did not evaluate BAMOT 

B + perspective map with ownship & traffic Did not evaluate Did not evaluate BRIPS 

Rare Event Testing Phase 

Four display conditions (BMO, BMOT, BAMO, BAMOT) were evaluated across subjects. The EP 

flew only one display condition for 19 runs during the rare event testing phase. 

These four display conditions were chosen to evaluate the effects of (a) displaying traffic without 

alerting (BMO and BMOT) and (b) with alerting (BAMO and BAMOT). The evaluation of these four 

display conditions allowed for direct comparison of the efficacy of alerting without the display of traffic. 

The rare event testing matrix further enabled direct evaluation of the contribution of display of traffic, 

both with and without alerting, to determine whether just showing traffic is sufficient for situation 

awareness and runway incursion avoidance or whether alerting is required. 

Eighteen trials were randomly flown that consisted of six different approaches in varying day time 

visual meteorological conditions (VMC) and instrument meteorological conditions (IMC) visibility 

conditions (see Appendix A). The intent of these trials was to provide sufficient variety and task demands 

to hide the rare event, a runway incursion. Pilots were unaware of the total number of trials to be 

conducted. The final trial in the block was the runway incursion event (Scenario 1 as described below) 

conducted in marginal VMC conditions (3 nm visibility and 1000 ft ceiling). Scenario 1 – the 

arrival/takeoff hold incursion – was selected because of its prevalence in runway incursion incident and 

accident statistics. The RSM incursion detection algorithm was used as the alert source. A wind profile 

was introduced to add workload to the tasks (see Appendix A). 

Usability Testing Phase 

Following the rare event phase, a usability study phase evaluated the effectiveness of the display 

conditions for runway incursion prevention. Each EP evaluated all five incursion scenarios with the 

seven display conditions (see Appendix B). The first trial for each scenario group used the baseline 

display condition. All of the trials that provided alerting used the RSM incursion detection algorithm as 

the alert source. In this study phase, the subjects had an expectation for the study intent (i.e., runway 

incursions), but they did not know the scenario before the initial trial. Another purpose of the usability 

study was to evaluate the RSM and PathProxTM incursion detection algorithms for GA operations. Each 

EP evaluated both algorithms using all five incursion scenarios and the BRIPS display condition. All 

trials in the usability study were conducted in marginal VMC conditions (3 nm visibility and 1000 ft 

ceiling) without winds. 
9

Runway Incursion Scenarios 

Five incursion scenarios were developed. Within each scenario, an incursion was staged by a blunder 

from one of the participating other aircraft. 

A traffic pattern was established to create reasonable traffic flow at KRNO (Figure 7). Alternating 

arrivals and departures were simulated using Runway 34R with interleaving departures on Runway 25. 

Traffic traveled through the intersection of Runways 25 and 34R every minute. The incurring traffic was 

interleaved into this traffic flow. 

Every effort was made to produce similar timing for the scenarios; however, a certain amount of 

variability in the timing was naturally introduced due to the maneuvering conducted by the EP (i.e., 

approach speed, taxi speed, etc.). 

Scenario 1 – arrival/takeoff hold 

This scenario began with ownship approaching Runway 34R for landing, 3 nm from the threshold at 

1010 ft above ground level (AGL), and at an indicated airspeed of 90 kt. The EP was cleared to land by 

tower. Another aircraft was stopped and holding at the 34R hold line nearest to the runway threshold. 

Scenario 1 tested the incursion situation where an aircraft moves into position and holds for takeoff 

even though another aircraft was approaching the same runway for a landing (Figure 8). The traffic for 

this scenario, instead of holding short of the active runway, taxied into position without clearance when 

the ownship was 2 nm from the threshold and held in that location awaiting takeoff clearance. 

Figure 7. Traffic pattern. Figure 8. Scenario 1 configuration. 
10

Scenario 2 – departure/intersection departure 

This scenario began with ownship taxiing on Taxiway C at 8 kt toward Runway 34R threshold, 

approximately 500 ft from the threshold. The EP was cleared to taxi to Runway 34R via Taxiway C and 

hold short of Runway 34R. The traffic was cleared to taxi to Runway 34R via Taxiway L, holding short 

of Runway 34R for an intersection take-off. (Taxiway L is approximately 3000 ft from Runway 34R 

threshold.) Ownship was then cleared for takeoff. 

Scenario 2 tested the incursion situation where one aircraft is departing as another aircraft enters the 

runway for an intersection departure (Figure 9). The traffic for this scenario, instead of holding short of 

the active runway, taxied across the hold line and entered the runway without clearance once the ownship 

began its takeoff (i.e., on runway heading and traveling greater than 10 kt ground speed). 

Scenario 3 – arrival/departure 

This scenario began with ownship approaching Runway 34R for landing, 3nm from the threshold at 

1010 ft AGL, and at an indicated airspeed of 90 kt. Another aircraft was taxiing on Taxiway L near the 

Runway 25 hold line at the threshold. The other aircraft was cleared into position for departure on 

Runway 25. The ownship EP was then cleared to land on Runway 34R by tower. 

Scenario 3 tested the incursion situation where an aircraft was departing even though another aircraft 

was landing on an intersecting runway (Figure 10). The traffic for this scenario, instead of holding in 

position, began its takeoff on Runway 25 without clearance as the ownship crossed the threshold of 

Runway 34R. 

Figure 9. Scenario 2 configuration. Figure 10. Scenario 3 configuration. 
11

Scenario 4 – departure/departure 

This scenario began with ownship taxiing on Taxiway C at 8 kt toward Runway 34R threshold, 

approximately 500 ft from the threshold. The EP was cleared to taxi to Runway 34R via Taxiway C and 

hold short of Runway 34R. Another aircraft was taxiing on Taxiway L near the Runway 25 hold line at 

the threshold. The other aircraft was cleared into position for departure on Runway 25. The ownship EP 

was then cleared for takeoff on Runway 34R. 

Scenario 4 tested the incursion situation where an aircraft was departing even though another aircraft 

was departing on an intersection runway (Figure 11). The traffic for this scenario, instead of holding in 

position, began its takeoff on Runway 25 without clearance as the ownship began its takeoff (i.e., on 

runway heading and traveling greater than 10 kt ground speed). 

Figure 11. Scenario 4 configuration. Figure 12. Scenario 5 configuration. 

Scenario 5 – taxi crossing/departure 

This scenario began with ownship parked on the ramp at the Mercury Air Center (Fixed-Base 

Operator (FBO)) facing Taxiway L. Another aircraft was stopped on Taxiway L behind the Runway 25 

hold line at the runway threshold. The EP was cleared to taxi to Runway 34R via Taxiway C, cleared to 
12

cross Runway 25, and hold short of Runway 34R. The other aircraft was cleared into position for 

departure on Runway 25 while the ownship taxied out of the ramp. 

Scenario 5 tested the incursion situation where an aircraft taxis across a runway even though another 

aircraft is taking-off from the same runway (Figure 12). The traffic for this scenario, instead of holding in 

position, began its takeoff on Runway 25 without clearance as the ownship crossed the centerline of 

Taxiway L. 

Procedure 

Prior to the rare event testing phase, each EP participated in a briefing and training session. The 

training did not reveal the focus of the experiment (runway incursion prevention). 

The EP received training on the incursion alerting system prior to the rare event testing only if the 

display condition evaluated included alerting. The EP was trained to abort if a warning alert was given 

during departure, go-around if a warning alert was given on approach, and stop if a warning alert was 

given during taxi. The EP was not required to take evasive action when a caution alert was issued. 

Before each trial, the EP was briefed on the run conditions, e.g. approach or departure, visibility, and 

displays available. The case order list is shown in Appendix C. 

For the usability study phase, the EP received training on the incursion alerting system prior to data 

collection. Before each trial, the EP was briefed on the run conditions, e.g. approach or departure, 

visibility, alerting system selected, and displays available. The EP was asked to continue the maneuver 

until a warning alert was received for evaluation purposes. The case order list is shown in Appendix D. 

The test runs were documented via audio, video, and digital data recordings, and post-run, post-block, 

and post-test questionnaires (Appendices E through J). 

Evaluation Pilots 

Sixteen GA pilots served as EPs. The EPs were selected to create four cross-sections of flying 

experience representative of the Part 91 pilot population: low-time (≤400 hours) visual flight rules 

(VFR), high-time (>400 hours) VFR, low-time (≤1000 hours) instrument-rated, and high-time (>1000 

hours) instrument-rated (see Table 3). 

Table 3. EP Experience 

Low Time VFR 

(≤ 400 Hours) 

High Time VFR 

(> 400 Hours) 

Low Time InstrumentRated (≤ 1000 Hours) High Time InstrumentRated (> 1000 Hours) 

EP Flight 

Hours 

EP Flight 

Hours 

EP Flight 

Hours 

EP Flight 

Hours 

1 80 5 415 9 590 13 4170 

2 300 6 765 10 393 14 15000 

3 175 7 450 11 1000 15 5365 

4 160 8 401 12 510 16 3500 

Mean 178.8 Mean 507.8 Mean 623.3 Mean 7008.8 

St. Dev. 91.0 St. Dev. 172.7 St. Dev. 263.9 St. Dev. 5383.1 
13

Results 

A summary of quantitative and qualitative results is presented for the rare event testing and usability 

study phases. All data are referenced from the center of gravity of the aircraft. All statistically significant 

results are presented at the p < 0.05 level unless stated otherwise. 

Rare Event Testing Phase 

Each EP flew 18 various approach tasks before being presented with the runway incursion event 

(Scenario 1). 

The FAA runway incursion severity rating (FAA, 2007), described below, was used to categorize the 

runway incursion incident results. The data were independently evaluated by a subject matter expert for 

classification. 

Category A – Separation decreases, extreme action taken to narrowly avoid collision, or collision 

occurs; 

Category B – Separation decreases, significant potential for collision; 

Category C – Separation decreases, ample time and distance to avoid collision; 

Category D – Little or no chance of collision but meets definition of runway incursion. 

Fourteen of the 16 runway incursions resulted in the less hazardous Category C and D incursions, one 

resulted in a Category A incursion, and one resulted in a Category B incursion. (All trials generated at 

least a Category D rating because the scenario was designed to elicit a runway incursion situation.) 

The 14 less hazardous Category C and D incursions were mitigated by the EPs by conducting a goaround and gaining separation from the traffic. 

The Category A runway incursion occurred with the EP flying the BMOT display concept. Despite 

the traffic indications on the surface map and out-the-window visuals, the EP demonstrated no awareness 

of the runway traffic and over-flew the traffic and landed. 

The Category B incident occurred when the EP over-flew the runway traffic (at 146 ft AGL) before 

conducting a go-around. The EP was aware of the incursion after having received an audible alert 

(BAMO display concept) but continued to descend to visually acquire the traffic to confirm the alert. 

This incident would have been classified as a Category D incursion if the EP had initiated the go-around 

at first awareness of the alert. Post-experimental briefings confirmed that the pilot was aware of the 

traffic but, because of the information provided by the display concept, felt safe proceeding lower in 

altitude to visually confirm the hazard. 

No statistically significant differences were found between the display conditions for both the 

distance from ownship to the incurring traffic or for the difference in time (i.e.reaction time) from the 

incursion event to when a go around was initiated by the pilots, based on a Multivariate Analysis of 

Variance (MANOVA) (p > 0.05); therefore, separate ANOVAs were not conducted on these dependent 

variables. 

Table 4. Algorithm Performance for Rare Event Scenario. 

Distance to Traffic Time to Traffic 

Mean (ft) Std. Dev. (ft) Mean (sec) Std. Dev. (sec.) 

PathProxTM Caution 5836 348.5 35 2.0 

PathProxTM Warning 4234 268.0 25 2.0 

RSM Warning 4060 135.7 24 2.6 
14

Data on both incursion detection algorithms were collected during the incursion events; however, 

only the alerts generated by the RSM algorithm were displayed to the EP, when the display condition 

included alerting. The alerting algorithm performance for the rare event incursion scenarios trials are 

presented in Table 4. Data from two EPs were omitted as outliers from these means because the pilots 

were initially heading to the wrong runway and turned toward Runway 34R within 1.1 nm. Even though 

alerts were generated the results were skewed. The “Time to Traffic” in Table 4 is estimated based on the 

relative distance from the traffic and the relative ground speed of the ownship at the time of the alert. A 

MANOVA showed a significant effect between the incursion detection algorithms tested (F(4,42) = 

190.163). Subsequent Analysis of Variances (ANOVAs) on the dependent variables revealed that the 

PathProxTM Caution alert was generated significantly earlier (F(2,42) = 6.839) and at a greater distance 

from the incursion aircraft (F(2,42) = 7.302) than either the PathProxTM Warning alert or the RSM 

Warning alert. There were no significant differences between the PathProxTM Warning alert and RSM 

Warning alert. 

Of the 16 trials, 

 8 EPs initially became aware of the incursion traffic by viewing out the window, before the 

alert would have occurred; 

 5 visually acquired the traffic out the window after an incursion alert would have occurred if 

it were active; 

 1 saw the traffic on the surface map well before the alert occurred; and 

 2 EPs did not see the traffic at all (i.e., the Category A and B incursions described above). 

The number of EP’s for each initial traffic awareness location, categorized by display condition, are 

shown in Table 5. 

Table 5. Initial Traffic Awareness Location during Rare Event Scenarios (16 pilots). 

Display Condition Out the Window, Before 

Alert Threshold 

Out the Window, 

After Alert Threshold 

Surface Map, Before 

Alert Threshold 

Did Not See 

Traffic 

BMO 1 3 0 0 

BMOT 2 1 0 1 

BAMO 3 0 0 1 

BAMOT 2 1 1 0 

Total 8 5 1 2 

As shown in Table 6, the EP’s initial awareness of the traffic was not affected by display concept 

(i.e., differences were not statistically significant at the α = 0.05 level); however, the data indicates a trend 

(p = 0.09) where the incursion traffic was acquired sooner when the EP was provided with traffic on the 

surface map and/or incursion alerts. The data for the two EPs that did not acquire the traffic (BMOT and 

BAMO display conditions) and the two EPs that headed to the incorrect runway (BMO and BAMOT 

display conditions) were omitted from Table 6. 

Table 6. Initial Traffic Awareness Measurements during Rare Event Scenarios (12 pilots). 

Distance to Traffic Time to Traffic 

Display Condition Mean (ft) Std. Dev. (ft) Mean (sec) Std. Dev. (sec.) 

BMO 3927 575.6 24 1.7 

BMOT 4910 1171.3 29 7.6 

BAMO 5261 622.3 28 7.1 

BAMOT 4745 1598.1 28 9.5 
15

For those displays that had alerting (BAMO, BAMOT), there were no significant differences in 

qualitative measures of timeliness of the alerting in terms of being able to take evasive action. However, 

when pilots were asked to rate all four display concepts on the perceived efficacy of the alerts (F(3,16) = 

10.948) and the additional safety value added (F(3,16) = 8.814), an ANOVA revealed a significant effect 

between the displays (p < 0.05). Subsequent post-hoc Student Newman Kuels tests showed that pilots 

reported that the BMO display condition was significantly lower in perceived efficacy and safety value 

added than the other three display conditions. Comparisons between the remaining three display 

conditions did not yield any significant statistical differences in qualitative ratings for these dependent 

variables 

Usability Study Phase 

All test trials conducted during the usability study phase included incursion events. Although the EPs 

were aware that each trial would contain an incursion event, they were not told the type of incursion 

before the initial trial for each scenario. During each test trial, data were simultaneously collected on the 

performance of both incursion detection algorithms; however, only one method, when dictated by the 

experimental configuration, was chosen for flight deck alerting. 

The EPs evaluated each incursion scenario with the seven display conditions to determine the 

effectiveness of the display for runway incursion prevention. The initial trial evaluated the baseline (B) 

display condition. This was done to determine if the EP would visually acquire the incursion traffic out 

the window since the surface map and alerting were not available. The RSM algorithm was used as the 

alert source for display in these trials. The EPs then evaluated both the RSM and PathProxTM incursion 

detection algorithms using the five incursion scenarios and the BRIPS display condition. 

During the course of usability data collection, it became apparent that the BMO and BAMO display 

conditions provided nearly identical traffic awareness for the EPs as the B (no alerts) and BA (audible 

alerts only) display conditions, respectively; therefore, a limited number of BMO and BAMO test trials 

were conducted in the interest of time. 

 Quantitative Results 

The alerting algorithm performance is summarized in Table 7. A total of 602 test trials were 

completed. Data were not analyzed for 14 trials due to missing data files and one test trial was omitted 

due to unorthodox maneuvering by the EP, yielding a total of 587 trials. 

Alerts were required for display on 415 test trials. During these 415 trials, the RSM was the alert 

source 81 percent of the time (336 trials). 

RSM generated alerts on 524 of the 587 total trials. RSM did not alert on 60 trials due to the 

maneuver taken by the EP. For example, the EP may have acquired the traffic out the window and 

conducted a go-around before the alerting criteria were met. RSM did not alert on three trials due to the 

scenario timing (the EP maneuvered such that no incursion event occurred). 

Of the possible 587 test trials, PathProxTM generated warning alerts on 345 trials. The 242 trials in 

which PathProxTM did not alert were generally due to the maneuvering performed by the EP. For 

instance, when RSM alerting was provided, the pilot executed an avoidance maneuver and this generally 

occurred before the PathProxTM alerts were generated. Caution alerts were only possible on 232 trials 

(scenarios 1 and 5) and of these, alerts were generated on 154 trials. Detailed PathProxTM analysis was 

not possible because specific alerting criteria are proprietary. 
16

Table 7. Summary of Alerts Generated. 

Scenario Number of 

Trials 

RSM PathProx Caution PathProx Warning Missing 

File 

Alerts No Alert Alerts No Alert Alerts No Alert 

1 117 97 20 116 1 98 19 4 

2 117 113 4 N/A N/A 84 33 5 

3 121 116 5 N/A N/A 61 60 1 

4 117 109 8 N/A N/A 64 53 2 

5 115 89 26 38 77 38 77 3 

Total 587 524 63 154 78 345 242 15 

Scenario 1 arrival/takeoff hold results – For the Baseline display (B) condition, all 16 EPs acquired 

the incursion traffic out the window for Scenario 1 when approximately 1 nm from the runway threshold. 

As a result, the average go-around initiation point was 443 ft AGL and 5240 ft (approximately 28 

seconds) from the traffic. During the Baseline condition cases, data were still being collected on the 

conflict detection algorithms. On average, the EP initiated a go-around after the PathProxTM caution alert 

would have occurred (475 ft AGL and 5641 ft – approximately 34 seconds – from the traffic for the 

baseline condition), but before either the RSM (409 ft AGL and 4272 ft – approximately 24 seconds – 

from the traffic for the baseline condition) or PathProxTM warning alert (392 ft AGL and 4117 ft - 

approximately 24 seconds – from the traffic for the baseline condition) would have occurred. 

Algorithm performance analyses for all Scenario 1 test trials are shown in Table 8. The PathProxTM 

caution alert was generated much earlier than the warning alerts. Both the RSM and PathProxTM warning 

alerts were generated at essentially the same moment. It should be noted that during the test trials in 

which alerts were provided, the EP was asked to continue the maneuver until the alert was received for 

evaluation purposes. 

Table 8. Algorithm Performance for Scenario 1. 

AGL Distance to Traffic Time to Traffic 

Mean (ft) Std. Dev.(ft) Mean (ft) Std. Dev. (ft) Mean (sec) Std. Dev. (sec) 

PathProxTM Caution 613 98.0 5757 325.7 36 1.2 

PathProxTM Warning 496 88.7 4175 278.7 25 1.0 

RSM Warning 492 92.6 4115 114.7 25 2.1 

Scenario 2 departure/intersection departure results - Six of 16 EPs (37.5 percent) did not acquire the 

incursion traffic visually for Scenario 2, or saw the traffic too late to abort the departure and actually 

over-flew the runway traffic when using the Baseline (B) display condition. 

Data on the abort initiation location (identified when the throttles and\or ground speed were reduced) 

are shown in Table 9. The EPs aborted at a further distance from Taxiway L when provided with those 

display configurations having alerting (BA, BAMO, BAMOT, and BRIPS). The EPs also aborted the 

departure for all of the trials with alerting provided. Based on measured data, the departure was aborted 

later when alerting was not provided (B, BMO, and BMOT), although still with enough time to stop prior 

to reaching the traffic. Also, as noted above, the EP actually took off on six trials and over-flew the 

traffic, using the Baseline display. 

For this departure scenario, the RSM warning alert was generated before and at a lower ground speed 

than the PathProxTM warning alert (Table 10); however, both provided ample time to abort and stop 

before reaching the incurring traffic. PathProxTM caution alerts are not generated on departure. 
17

Table 9. Abort Initiation during Scenario 2 for Display Conditions. 

Nunber of Abort 

Occurrences

Distance to Taxiway L Time to Taxiway L 

Mean (ft) Std. Dev. (ft) Mean (sec) Std. Dev. (sec.) 

B 8 2177 485.6 18 1.9 

BMO 3 2004 595.3 15 5.7 

BMOT 12 1849 408.6 13 3.4 

BA 14 2493 108.1 19 1.1 

BAMO 5 2530 56.5 19 1.1 

BAMOT 15 2498 160.0 19 1.3 

BRIPS 16 2542 86.0 19 0.9 

Table 10. Algorithm Performance for Scenario 2. 

Ground Speed Distance from Twy L Time from Twy L 

Mean (kt) Std. Dev. 

(kt) 

Mean (ft) Std. Dev. 

(ft) 

Mean (sec) Std. Dev. 

(sec) 

PathProxTM Warning 41 4.4 2452 75.2 19 1.5 

RSM Warning 28 4.7 2637 102.9 21 1.4 

Scenario 3 arrival/departure results – For the Baseline (B) condition, 13 of 16 EPs (81 percent) did 

not acquire the incursion traffic visually for Scenario 3. Due to the scenario design; however, 14 EPs 

landed and stopped before reaching the crossing runway. One EP landed but taxied through the 

intersection as the traffic was departing from crossing Runway 25. One EP conducted a go-around. 

The algorithm performance for Scenario 3 is presented in Table 11. On average, the RSM Warning 

alert was generated slightly before the PathProxTM Warning alert. Both algorithms alerted at or slightly 

before touchdown. PathProxTM Caution alerts were not generated for this scenario. 

For all Scenario 3 trials (122 total), a go-around was conducted just before touchdown (7 ft AGL) on 

11 percent (14) of the trials. 

Table 11. Algorithm Performance for Scenario 3. 

Distance to Runway 34R/25 Intersection Time to Runway 34R/25 Intersection 

Mean (ft) Std. Dev. (ft) Mean (sec) Std. Dev. (sec) 

PathProxTM Warning 1190 306.4 9 2.8 

RSM Warning 1301 944.2 11 1.4 

Scenario 4 departure/departure results – Five of 16 EPs (31.3 percent) saw the incursion traffic 

visually and aborted the departure when using the Baseline (B) display condition. Nine of 16 EPs (62.5 

percent) did not see the incursion traffic visually, or saw the traffic too late to abort the departure. For 

these trials, the ownship came within an average distance of 458 ft (range from 179 ft to 795 ft) from the 

incursion traffic. Two of the trials resulted in no incursion event due to the scenario timing; therefore, 

these data are not included in the analysis. 

Generally, when alerts were provided (BA, BAMO, BAMOT, and BRIPS), the EPs were aware of the 

traffic conflict and initiated aborts earlier than without the alerts as shown in Table 12. For trials in which 
18

the EP visually acquired traffic - either out-the-window or using a map with traffic display - the departure 

was still aborted with enough time to stop before the crossing runway. 

The RSM warning alerts were generated before the PathProxTM warning alerts for this departure 

scenario. When RSM alerts were presented, the EP typically reacted based on those alerts; therefore, 

PathProxTM alerts were sometimes not generated or generated after initiation of the abort maneuver. For 

an accurate assessment of PathProxTM performance, only the results from the trials in which the 

PathProxTM alerts were shown to the EP are presented (Table 13). Again, the RSM warning alert was 

generated before and at a lower ground speed than the PathProxTM warning alert; however, both still 

provided ample time to abort and stop before reaching the incurring traffic. PathProxTM caution alerts are 

not generated on departure. 

Table 12 . Abort Initiation during Scenario 4 for Display Conditions. 

Distance to Runway 25 Time to Runway 25 

Mean (ft) Std. Dev. (ft) Mean (sec) Std. Dev. (sec.) 

B 1194 799.7 9 6.2 

BMO 1928 293.4 15 2.7 

BMOT 1743 531.6 13 4.6 

BA 2091 65.8 17 0.6 

BAMO 2056 151.6 16 2.1 

BAMOT 2107 69.4 17 0.9 

BRIPS 2096 79.6 17 1.2 

Table 13. Algorithm Performance for Scenario 4. 

Ground Speed Distance from Rwy 25 Time from Rwy 25 

Mean (kt) Std. Dev. 

(kt) 

Mean (ft) Std. Dev. 

(ft) 

Mean (sec) Std. Dev. 

(sec) 

PathProxTM Warning 48 8.5 1891 115.2 15 1.8 

RSM Warning 29 4.8 2216 35.3 18 0.9 

Scenario 5 taxi crossing/departure results - Fourteen of 16 EPs (87.5 percent) acquired the incursion 

traffic out the window when using the Baseline (B) condition and stopped before reaching Runway 25. 

One EP did not see the traffic and actually crossed Runway 25 in front of the departing traffic. One EP 

saw the departing traffic out the window but chose to cross Runway 25 anyway. 

The RSM algorithm uses predictive alerting, triggered on the aircraft ground speed, in an attempt to 

keep the aircraft clear of a runway, behind the hold line. If the ownship is traveling 8 kt or greater and is 

not slowing down, the alert will be generated before the aircraft reaches the hold line, providing sufficient 

distance to stop before crossing the hold line. As the taxi speed increases, the alert is generated when the 

ownship is a farther distance from the hold line. However, the alert is not generated until after the 

ownship crosses the hold line when the ownship is traveling less than 8 kt. The 8 kt threshold was used to 

minimize false or nuisance alerts as the ownship taxis toward a hold line. Details on the PathProxTM 

implementation are proprietary. 

The effect of predictive alerting with the RSM alert using an 8 kt ground speed trigger is shown in 

Table 14. If the ground speed was above 8 kt approaching the hold line, the RSM alert was generated 

before reaching the hold line (positive value). A negative value indicates the aircraft crossed over the 

hold line before the alert was generated. Note that the hold line was 170 ft from the edge of the runway 
19

for this scenario so in all cases the alert was still generated well before the aircraft entered the runway. 

The proactive RSM alerting trigger speed may need to be reassessed to keep a higher percentage of taxi 

blunders from crossing hold lines. These data indicate that almost 40% of the time ownship crossed the 

hold line (i.e., technically, a runway incursion). 

As with Scenario 4, the RSM alerts were generated before the PathProxTM alerts. Therefore, for an 

accurate assessment of PathProxTM, only the results from trials in which the PathProxTM alerts were given 

to the EP are presented (Table 15). The data shows that PathProx does not use predictive alerting. All 

warning and cautions occurred after the ownship crossed the hold line, but were still triggered 153 and 

130 ft, respectively, prior to the runway. Assuming a reasonable taxi speed, ownship should have been 

able to react and stop prior to the runway. 

Table 14. Scenario 5 RSM Alert Generation 

EP Ground 

Speed (kt) 

Distance to hold 

line (feet) 

1 9.7 59 

2 9.7 48 

3 9.2 59 

4 7.9 -12 

5 7.0 -6 

6 8.4 56 

7 6.0 -7 

8 13.4 17 

9 10.3 70 

10 8.4 48 

11 8.3 48 

12 7.5 -17 

13 7.4 -14 

15 6.5 -7 

16 10.6 8 

Table 15. Algorithm Performance for Scenario 5. 

Distance Before/Past Hold Line Distance from Runway 25 Edge 

Mean (ft) Std. Dev. (ft) Mean (ft) Std. Dev. (ft) 

PathProxTM Caution 17 past 3.8 153 3.8 

PathProxTM Warning 41 past 19.5 130 19.5 

RSM Warning 23 before 32.6 193 32.6 
20

Qualitative Results 

Post-run (Appendices E and F), post-block (Appendices G and H), and post-test (Appendices I and J) 

questionnaires were administered. These data are discussed in the following. Ratings for most of the 

questions were generally based on a scale of 1 (low) to 10 (high). The result values shown in this section 

are the mean values based on the 1 to 10 scale, unless stated otherwise. 

Moving Map Display – Statistical analyses were not conducted on the qualitative data on pilot ratings 

of the moving map display because of limited power, due to the low number of observations, for the nonparametric test; therefore, descriptive data are reported below. When the EPs rated the effectiveness of 

the perspective surface map relative to the plan view map for prevention of runway incursions, the 

perspective map was rated slightly more effective (2.88 on a 10-point scale). However, the location of the 

surface map was deemed suboptimal and should be positioned closer to the pilot’s head-up field of view. 

The traffic presentation on the surface map was considered easily discernable (8.38). Most EPs (14 of 16) 

considered traffic presentation necessary to prevent runway incursions. The addition of traffic was rated 

to provide increased (8.56) SA over a surface map with only ownship location. Over half of the EPs (10 

of 16) indicated visual presentation of alerts on the surface map was necessary to prevent runway 

incursions. The addition of visual alerts was rated to provide increased (6.27) SA. The EPs considered 

the terms used for the incursion alerts (e.g. “Warning, Traffic Departing 25”) to be very effective (8.88). 

For display conditions with alerting available, the EPs indicated that the incursion event would most 

likely be brought to their attention first through audible alerting, then on the surface map, and lastly out 

the window. Nine of 16 EPs indicated that an audible alert alone would be the minimum necessary for an 

effective incursion prevention display, while five of 16 indicated a surface map with ownship and traffic 

but without alerting would be the minimum to be effective. However, all 16 EPs indicated a surface map 

with ownship and traffic in conjunction with an audible alert would be an optimal incursion prevention 

display. According to averaged EP ratings, for all alerting display conditions, the caution and warning 

alerting system was perceived to provide the greatest amount of runway incursion awareness (see Table 

16).    Based on the descriptive results, there was no consistency across the EPs in terms of alerting 

preference for incursion awareness and perceived SA improvement. For each of the four display 

conditions (BA, BAMO, BAMOT, BRIPS), the majority of EPs rated the “caution and warning” alerting 

higher for incursion awareness. However, for each display condition, at least 6 of 16 EPs rated the 

“warning only” alerting as preferable for incursion awareness and, for these EPs, the average SA rating 

was 6.5 to 7.7 in perceived SA improvement. 

Table 16. Alerting Preference for Incursion Awareness and Perceived SA Improvement 

Alerting Type Warning Only Caution & Warning 

Number of EPs SA 

Rating 

Number of EPs SA 

Rating 

BA 6 6.5 10 4.2 

BAMO 6 6.7 10 4.3 

BAMOT 7 7.7 9 5 

BRIPS 7 7.6 9 5.2 

Note: “Number of EPs” column represents the number of EPs that indicated preference for alerting 

condition. “SA Rating” column represents the amount of SA improvement (0 – 10 scale) for the alerting 

condition indicated by that EP. 

Algorithm Alerting - All EPs indicated that both the RSM and PathProxTM alerting provided 

sufficient time to avoid a potential conflict. Only six of 16 EPs (37.5 percent) thought providing both 
21

caution and warning alerts was more effective in preventing runway incidents than a single warning alert. 

However, nine of 16 EPs indicated that the caution and warning system provided greater (4.39) SA and 

provided more reaction time. Eleven of 16 EPs (68.8 percent) liked the idea of having a caution alert in 

conjunction with a warning to provide more evaluation and reaction time, i.e. a greater comfort level. For 

the scenarios evaluated, the EPs generally felt that providing caution and warning alerts on approach was 

most effective, while a warning alert alone was sufficient when on the airport surface (during departure 

and taxi). 

The EPs were asked if resolutions or maneuver guidance should be provided in addition to runway 

incursion alerts for various operations. The majority of EPs would like to be provided with maneuver 

guidance for conflict resolution on final approach (12 of 16) and when taxiing across a runway (nine of 

16). Half of the EPs would like maneuver guidance on departure. 

Surface Operations Safety - In general, the EPs felt safer during runway incursion encounters when 

alerting was provided (B=2.06, BMO=2.75, BMOT= 5.38, BA=7.19, BAMO=7.44, BAMOT=9.31, and 

BRIPS=9.19). Analysis of pilot responses to their perceived safety and runway incursion prevention 

effectiveness support this conclusion with significant main effects found (i.e., for perceived safety 

(F(6,90) = 857.390) and runway incursion prevention effectiveness (F(6, 90) = 188.793)). The addition 

of traffic was marginally beneficial when presented on a moving map display and was only effective 

when alerting was provided. A possible cause may be that pilots ARE out-the-window and CANNOT BE 

focused on a head-down display during this critical phase of flight, especially in VMC flight conditions. 

When alerted, the pilot is provided a cue that it might be beneficial for them to direct focus and attention 

to the head-down display to locate the incurring traffic. In fact, pilots rated having audible alerts (BA) 

and having alerts with a map with ownship but no traffic (BAMO) similarly for runway incursion 

prevention on almost all dependent variables measured. For the experimental scenarios tested, the 

moving map display appeared beneficial in preventing incursions only when both traffic AND alerting 

were included. Ideally, a system should have the ability to alert the pilot to incurring events and cue them 

where to look without having to go head-down, into the cockpit. Head-up and head-worn displays 

provide this type of capability but are to date, not generally installed in low-end GA aircraft. 

Conclusions 

A Runway Incursion Prevention System (RIPS), developed for commercial transport flight decks and 

adapted for general aviation operations, was evaluated in a piloted simulation study. The purpose of the 

study was to evaluate the aircraft-based incursion detection algorithms and associated alerting and airport 

surface display concepts for general aviation operations using a rare event study followed by a usability 

study. 

The results indicate that, during the rare event testing, for the scenario evaluated, most pilots were 

able to acquire the incurring traffic by looking out the cockpit windows (in VMC conditions), even before 

incursion alerting was activated. In the usability study, for some scenarios, pilots were generally unable 

to detect the incurring traffic out-the-window. This emphasized the importance of providing alerts of 

potential runway conflicts to the pilot. 

The surface map showing ownship (without other traffic) was rated as being significantly inferior to a 

surface map showing traffic and/or incursion alerting for perceived safety value added. However, one 

pilot experienced a severe runway incursion and risk for collision despite having traffic displayed on a 

surface map. The addition of traffic was marginally beneficial when presented on a surface map display 

and was only effective when alerting was provided. A possible cause may be that pilots had transitioned 

heads-up out the window and were not focused on the head-down display to locate the incurring traffic. 

Pilots reported that the utility of the surface map would be significantly more effective if located higher 

on the instrument panel closer to the pilot’s head-up field of view. 

The results generally match past research on commercial operations in that the incursion alerts 

provided sufficient time to avoid a potential incursion conflict. Departures were generally aborted sooner 
22

when alerts were provided, resulting in greater safety margins. A surface map showing ownship and 

traffic along with audible alerts was considered an optimal incursion prevention display, while an audible 

alert alone was considered a minimally effective display. Over half of the pilots would like maneuver 

guidance for conflict resolution in conjunction with incursion alerting. In general, the pilots felt 

substantially safer during potential runway incursion incidents with alerting onboard. 
23

References 

Bartolone, A.P., Glaab, L.J., Hughes, M.F., and Parrish, R.V. (2005), Initial Development of a Metric to 

Describe the Level of Safety Associated with Piloting an Aircraft with Synthetic Vision 

Systems (SVS) Displays, Proceedings of the SPIE Defense & Security Symposium. 

Enhanced and Synthetic Vision 2005. Editor(s): Jacques G. Verly. Vol. 5802, April 2005, pp. 

112-126. 

Cassell, R., Evers, C., and Esche, J. (2003), Safety Benefits of PathProx – A Runway Incursion Alerting 

System, Proceedings of the AIAA/IEEE 22nd Digital Avionics Systems Conference. 

Indianapolis, IN, 12-16 Oct. 2003. 

FAA (2004), Air Traffic Control, Department of Transportation, Federal Aviation Administration Order 

7110.65P, Washington DC, 2004. 

FAA (2007), FAA Runway Safety Report. Department of Transportation, Federal Aviation 

Administration, Office of Runway Safety, Washington DC, September 2007. 

Foyle, D.C. and Hooey, B.L. (2003), Improving Evaluation and System Design Through the Use of OffNominal Testing: A Methodology for Scenario Development, Proceedings of the 12th

International Symposium on Aviation Psychology, Wright State University, Dayton, OH. pp. 

397-402. 

Green, D.F. (2006), Runway Safety Monitor Algorithm for Single and Crossing Runway Incursion 

Detection and Alerting, NASA Langley Research Center, Hampton, VA, NASA CR-2006-

214275. February 2006. 

Green, D.F., Otero, S.D., Barker, G.D., and Jones, D.R. (2009), Initial Concept for Terminal Area 

Conflict Detection, Alerting, and Resolution Capability On or Near the Airport Surface, 

NASA Langley Research Center, Hampton, VA, NASA TM-2009-215696, March 2009. 

Jones, D.R., Quach, C.C., and Young, S.D. (2001), Runway Incursion Prevention System – 

Demonstration and Testing at the Dallas/Fort Worth International Airport, Proceedings of 

the 20th Digital Avionics Systems Conference, Daytona Beach, FL, 14-18 Oct. 2001. 

Jones, D.R. (2002), Runway Incursion Prevention System Simulation Evaluation, Proceedings of the 

AIAA/IEEE 21st Digital Avionics Systems Conference. Irvine, CA, 27-31 Oct. 2002. 

Jones, D.R. (2005), Runway Incursion Prevention System Testing at the Wallops Flight Facility, 

Proceedings of the SPIE Defense & Security Symposium. Enhanced and Synthetic Vision 

2005. Editor(s): Jacques G. Verly. Vol. 5802, April 2005, pp. 47-58. 

Newman, R.L. and Foyle, D.C. (2003), Test Scenarios for Rare Events, Proceedings of the 12th

International Symposium on Aviation Psychology. Wright State University, Dayton, OH, pp. 

873-882. 

NTSB (2000), Safety Recommendation, Letter to the FAA Administrator, A-00-66, National 

Transportation Safety Board, Washington, DC, July 2000. 

NTSB (2007), Most Wanted List - Transportation Safety Improvements, National Transportation Safety 

Board, Washington, DC, www.ntsb.gov/recs/brochure/mostwanted_2008.pdf, November 

2007. 

RTCA (2001), User Requirements for Aerodrome Mapping Information, RTCA DO-272, Washington, 

DC, November 2001. 
24

Appendix A: Rare Event Test Matrix 

Evaluation Pilot EPn (n=1 to 16) 

RIPS Scenario Scenario 1 – arrival / takeoff hold (see Figure 8) 

Meteorological VMC1 = 3 nm, day, 1000 ft ceiling 

 VMC2 = 3 nm, day, 2000 ft ceiling 

 IMC1 = 1 nm, day, 1000 ft ceiling 

 IMC2 = 1 nm, day, 400 ft ceiling 

 IMC3 = 2 nm, day, 1000 ft ceiling 

Altitude (ft) 4412 

(0 AGL) 

4420 

(8 AGL) 

5420 

(1008 AGL) 

7100+ 

(2688 AGL) 

Wind Speed (kt) 10 10 20 30 

Wind Direction (deg) 260 260 260 360 

 Note: gradual shift of wind speed and direction 

Display Concept BMO = Baseline + surface map and ownship 

 BMOT = Baseline + surface map and ownship and traffic 

 BAMO = BMO + audible incursion alerts 

 BAMOT = BMOT + incursion alerts (audible & visual) 

Note: IC = Initial Condition 

Approach S34RL = Straight-in approach to 34R and land. IC is 3nm out, 1010 ft AGL, 90 kt 

 S34RW = Straight-in approach to 34R then waveoff at 200 ft AGL. IC is 3 nm out, 

 1010 ft AGL, 90 kt 

 S34LSS= Straight-in approach, lined up with 34L, sidestep to 34R and land. IC is 3nm 

 out, 1010 ft AGL, 90 kt 

 S25L = Straight-in approach to 25 and land. IC is 3 nm out, 1800 ft AGL 

 C-long = Non-precision approach – initial approach to 34R from base leg ending in 

 Circling approach to 25 and land. IC is 9.56 nm out from 34R on left 

 base (85 degree heading & 1.6 nm left offset), 2737.5 ft AGL 

 S-long = ILS approach – approach to 34R from base leg, follow approach path and go 

 around at 200 ft. IC is 9.56 nm from 34R on left base (85 degree heading & 1.6 

 nm left offset), 2737.5 ft AGL 
25

Rare Event Test Matrix 

 

EP Case No. RIPS 

Scenario 

Meteorological 

Alert 

Source 

Display 

Concept 

Approach 

1,5,9,13 1 None VMC1 N/A BMO S34RL 

 2 Scenario 1 VMC1 N/A BMO S34RL 

 3 None IMC2 N/A BMO S34RL 

 4 None IMC2 N/A BMO S34RL 

 5 None VMC1 N/A BMO S34RW 

 6 None VMC1 N/A BMO S34RW 

 7 None IMC2 N/A BMO S34RW 

 8 None IMC2 N/A BMO S34RW 

 9 None VMC1 N/A BMO S34LSS 

 10 None VMC1 N/A BMO S34LSS 

 11 None VMC1 N/A BMO S34LSS 

 12 None VMC1 N/A BMO S34LSS 

 13 None VMC2 N/A BMO S25L 

 14 None VMC2 N/A BMO S25L 

 15 None VMC2 N/A BMO S25L 

 16 None VMC2 N/A BMO S25L 

 17 None IMC3 N/A BMO C-long 

 18 None IMC3 N/A BMO C-long 

 19 None IMC2 N/A BMO S-long 

2,6,10,14 20 None VMC1 N/A BMOT S34RL 

 21 Scenario 1 VMC1 N/A BMOT S34RL 

 22 None IMC2 N/A BMOT S34RL 

 23 None IMC2 N/A BMOT S34RL 

 24 None VMC1 N/A BMOT S34RW 

 25 None VMC1 N/A BMOT S34RW 

 26 None IMC2 N/A BMOT S34RW 

 27 None IMC2 N/A BMOT S34RW 

 28 None VMC1 N/A BMOT S34LSS 

 29 None VMC1 N/A BMOT S34LSS 

 30 None VMC1 N/A BMOT S34LSS 

 31 None VMC1 N/A BMOT S34LSS 

 32 None VMC2 N/A BMOT S25L 

 33 None VMC2 N/A BMOT S25L 

 34 None VMC2 N/A BMOT S25L 

 35 None VMC2 N/A BMOT S25L 

 36 None IMC3 N/A BMOT C-long 

 37 None IMC3 N/A BMOT C-long 
26

 38 None IMC2 N/A BMOT S-long 

3,7,11,15 39 None VMC1 N/A BAMO S34RL 

 40 Scenario 1 VMC1 RSM BAMO S34RL 

 41 None IMC2 N/A BAMO S34RL 

 42 None IMC2 N/A BAMO S34RL 

 43 None VMC1 N/A BAMO S34RW 

 44 None VMC1 N/A BAMO S34RW 

 45 None IMC2 N/A BAMO S34RW 

 46 None IMC2 N/A BAMO S34RW 

 47 None VMC1 N/A BAMO S34LSS 

 48 None VMC1 N/A BAMO S34LSS 

 49 None VMC1 N/A BAMO S34LSS 

 50 None VMC1 N/A BAMO S34LSS 

 51 None VMC2 N/A BAMO S25L 

 52 None VMC2 N/A BAMO S25L 

 53 None VMC2 N/A BAMO S25L 

 54 None VMC2 N/A BAMO S25L 

 55 None IMC3 N/A BAMO C-long 

 56 None IMC3 N/A BAMO C-long 

 57 None IMC2 N/A BAMO S-long 

4,8,12,16 58 None VMC1 N/A BAMOT S34RL 

 59 Scenario 1 VMC1 RSM BAMOT S34RL 

 60 None IMC2 N/A BAMOT S34RL 

 61 None IMC2 N/A BAMOT S34RL 

 62 None VMC1 N/A BAMOT S34RW 

 63 None VMC1 N/A BAMOT S34RW 

 64 None IMC2 N/A BAMOT S34RW 

 65 None IMC2 N/A BAMOT S34RW 

 66 None VMC1 N/A BAMOT S34LSS 

 67 None VMC1 N/A BAMOT S34LSS 

 68 None VMC1 N/A BAMOT S34LSS 

 69 None VMC1 N/A BAMOT S34LSS 

 70 None VMC2 N/A BAMOT S25L 

 71 None VMC2 N/A BAMOT S25L 

 72 None VMC2 N/A BAMOT S25L 

 73 None VMC2 N/A BAMOT S25L 

 74 None IMC3 N/A BAMOT C-long 

 75 None IMC3 N/A BAMOT C-long 

 76 None IMC2 N/A BAMOT S-long 
27

Appendix B: Usability Study Test Matrix 

RIPS Scenario 1 = Arrival / takeoff hold 

 2 = Departure / intersection departure 

 3 = Arrival / departure 

 4 = Departure / departure 

 5 = Taxi crossing / departure 

Display Concept B = Baseline round dials 

 BMO = B + surface map and ownship 

 BMOT = B + surface map and ownship and traffic 

 BA = B + audible incursion alerts 

 BAMO = BMO + audible incursion alerts 

 BAMOT = BMOT + incursion alerts (audible & visual) 

 BRIPS = B + perspective surface map 

All runs conducted under VMC1 (3nm, day, 1000 ft ceiling, no wind) 

Table B.1: Display Concept Evaluation for Runway Incursion Prevention 

Case No. RIPS 

Scenario 

Alert 

Source 

Display 

Concept 

77 1 N/A B 

78 1 N/A BMO 

79 1 N/A BMOT 

80 1 RSM BA 

81 1 RSM BAMO 

82 1 RSM BAMOT 

83 1 RSM BRIPS 

84 2 N/A B 

85 2 N/A BMO 

86 2 N/A BMOT 

87 2 RSM BA 

88 2 RSM BAMO 

89 2 RSM BAMOT 

90 2 RSM BRIPS 

91 3 N/A B 

92 3 N/A BMO 

93 3 N/A BMOT 

94 3 RSM BA 

95 3 RSM BAMO 

96 3 RSM BAMOT 

97 3 RSM BRIPS 

98 4 N/A B 

99 4 N/A BMO 
28

100 4 N/A BMOT 

101 4 RSM BA 

102 4 RSM BAMO 

103 4 RSM BAMOT 

104 4 RSM BRIPS 

105 5 N/A B 

106 5 N/A BMO 

107 5 N/A BMOT 

108 5 RSM BA 

109 5 RSM BAMO 

110 5 RSM BAMOT 

111 5 RSM BRIPS 

Table B.2: RSM and PathProxTM Incursion Detection Algorithm Evaluation 

Case No. RIPS 

Scenario 

Alert Source Display 

Concept 

148 1 RSM BRIPS 

150 1 PathProxTM BRIPS 

152 2 RSM BRIPS 

154 2 PathProxTM BRIPS 

156 3 RSM BRIPS 

158 3 PathProxTM BRIPS 

160 4 RSM BRIPS 

162 4 PathProxTM BRIPS 

164 5 RSM BRIPS 

166 5 PathProxTM BRIPS 
29

Appendix C: Rare Event Case Lists 

Tables C.1 and C.2 lists the experimental case order for each evaluation pilot for the rare event 

portion of the testing. Refer to Appendix A for a detailed description of each case. 

Table C.1. Rare Event Case Order for Evaluation Pilots 1 through 8 

 EP1 EP2 EP3 EP4 EP5 EP6 EP7 EP8 

Run 1 17 36 55 74 17 36 55 74 

Run 2 16 33 49 65 15 31 49 60 

Run 3 10 25 47 62 11 24 51 64 

Run 4 6 30 48 61 7 33 44 68 

Run 5 11 23 54 73 4 28 48 71 

Run 6 13 35 46 71 14 23 41 62 

Run 7 7 34 43 64 13 32 50 66 

Run 8 8 27 45 69 1 22 39 58 

Run 9 12 26 42 67 10 35 47 61 

Run 10 15 29 52 60 5 25 45 70 

Run 11 19 38 57 76 19 38 57 76 

Run 12 3 31 44 66 16 26 46 73 

Run 13 1 22 51 70 12 20 54 65 

Run 14 9 24 53 72 6 29 43 72 

Run 15 14 28 39 68 8 34 52 69 

Run 16 4 20 41 58 3 27 42 63 

Run 17 5 32 50 63 9 30 53 67 

Run 18 18 37 56 75 18 37 56 75 

Run 19 2 21 40 59 2 21 40 59 
30

Table C.2. Rare Event Case List for Evaluation Pilots 9 through 16 

 EP9 EP10 EP11 EP12 EP13 EP14 EP15 EP16 

Run 1 17 36 55 74 17 36 55 74 

Run 2 3 28 44 68 12 34 54 67 

Run 3 11 32 53 71 8 27 47 60 

Run 4 1 35 39 69 3 23 41 63 

Run 5 13 23 42 72 1 25 46 64 

Run 6 16 31 47 58 14 33 52 70 

Run 7 8 33 49 62 5 32 43 68 

Run 8 12 30 50 63 4 22 42 69 

Run 9 14 29 51 65 6 35 39 61 

Run 10 9 34 48 73 13 29 50 65 

Run 11 19 38 41 76 19 38 57 76 

Run 12 6 20 57 66 15 31 49 58 

Run 13 10 26 54 61 7 28 44 71 

Run 14 15 24 43 60 9 24 51 73 

Run 15 5 27 45 64 16 20 45 66 

Run 16 7 22 52 67 10 26 48 62 

Run 17 4 25 46 70 11 30 53 72 

Run 18 18 37 56 75 18 37 56 75 

Run 19 2 21 40 59 2 21 40 59 
31

Appendix D: Usability Study Case Lists 

Tables D.1 through D.4 lists the experimental case order for each evaluation pilot for the usability 

study portion of the testing. Refer to Appendix B for a detailed description of each case. In Tables D1 

and D2, the cases shown with ‘skip’ were not conducted. 

Part 1: Display Concept Evaluation 

Table D.1. Case List for Display Concept Evaluation for Pilots 1 through 8 

EP1 EP2 EP3 EP4 EP5 EP6 EP7 EP8 

77 77 84 84 91 91 98 98 

78 83 90 85 92 97 104 99 

79 82 89 86 93 96 103 100 

80 81 88 87 94 95 102 101 

81 80 87 88 95 94 101 102 

82 79 86 89 96 93 100 103 

83 78 85 90 97 92 99 105 

84 84 91 91 98 98 105 105 

90 85-skip 92-skip 97 104 99-skip 106-skip 111 

89 86 93 96 103 100 107 110 

88 87 94 95-skip 102 101 108 109-skip 

87 88-skip 95-skip 94 101 102-skip 109-skip 108 

86 89 96 93 100 103 110 107 

85 90 97 92-skip 99-skip 104 111 106-skip 

91 91 98 98 105 105 77 77 

92 97 104 99-skip 106-skip 111 83 78-skip 

93 96 103 100 107 110 82 79 

94 95-skip 102-skip 101 108 109-skip 81-skip 80 

95 94 101 102-skip 109-skip 108 80 81-skip 

96 93 100 103 110 107 79 82 

97 92-skip 99-skip 104 111 106-skip 78-skip 83 

98 98 105 105 77 77 84 84 

104 99-skip 106-skip 111 83 78-skip 85-skip 90 

103 100 107 110 82 79 86 89 

102 101 108 109-skip 81-skip 80 87 88-skip 

101 102-skip 109-skip 108 80 81-skip 88-skip 87 

100 103 110 107 79 82 89 86 

99 104 111 106-skip 78-skip 83 90 85-skip 

105 105 77 77 84 84 91 91 

106 111 83 78-skip 85-skip 90 97 92-skip 

107 110 82 79 86 89 96 93 

108 109-skip 81-skip 80 87 88-skip 95-skip 94 

109 108 80 81-skip 88-skip 87 94 95-skip 

110 107 79 82 89 86 93 96 

111 106-skip 78-skip 83 90 85-skip 92-skip 97 
32

Table D.2. Case List for Display Concept Evaluation for Pilots 9 through 16 

EP9 EP10 EP11 EP12 EP13 EP14 EP15 EP16 

105 105 77 77 84 84 91 91 

106 111 78 83 85 90 92 97 

107 110 79 82 86 89 93 96 

108 109 80 81 87 88 94 95 

109 108 81 80 88 87 95 94 

110 107 82 79 89 86 96 93 

111 106 83 78 90 85 97 92 

77 77 91 91 98 98 105 105 

83 78-skip 97 92-skip 104 99-skip 111 106-skip 

82 79 96 93 103 100 110 107 

81 80 95-skip 94 102-skip 101 109-skip 108 

80 81-skip 94 95-skip 101 102-skip 108 109-skip 

79 82 93 96 100 103 107 110 

78-skip 83 92-skip 97 99-skip 104 106-skip 111 

84 84 105 105 77 77 84 84 

85-skip 90 106-skip 111 78-skip 83 85-skip 90 

86 89 107 110 79 82 86 89 

87 88-skip 108 109-skip 80 81-skip 87 88-skip 

88-skip 87 109-skip 108 81-skip 80 88-skip 87 

89 86 110 107 82 79 89 86 

90 85-skip 111 106-skip 83 78-skip 90 85-skip 

91 91 84 84 91 91 98 98 

97 92-skip 90 85-skip 97 92-skip 104 99-skip 

96 93 89 86 96 93 103 100 

95-skip 94 88-skip 87 95-skip 94 102-skip 101 

94 95-skip 87 88-skip 94 95-skip 101 102-skip 

93 96 86 89 93 96 100 103 

92-skip 97 85-skip 90 92-skip 97 99-skip 104 

99 98 98 98 105 105 77 77 

99-skip 104 99-skip 104 106-skip 111 78-skip 83 

100 103 100 103 107 110 79 82 

101 102-skip 101 102-skip 108 109-skip 80 81-skip 

102-skip 101 102-skip 101 109-skip 108 81-skip 80 

103 100 103 100 110 107 82 79 

104 99-skip 104 99-skip 111 106-skip 83 78-skip 
33

Part 2: Incursion Detection Algorithm Evaluation 

Table D.3. Case List for Incursion Detection Algorithm Evaluation for Pilots 1 through 8 

EP1 EP2 EP3 EP4 EP5 EP6 EP7 EP8 

148 150 148 150 152 154 152 154 

150 148 150 148 154 152 154 152 

154 152 154 152 162 160 162 160 

152 154 152 154 160 162 160 162 

156 158 156 158 148 150 148 150 

158 156 158 156 150 148 150 148 

162 160 162 160 158 156 158 156 

160 162 160 162 156 158 156 158 

164 166 164 166 164 166 164 166 

166 164 166 164 166 164 166 164 

Table D.4. Case List for Incursion Detection Algorithm Evaluation for Pilots 9 through 16 

EP9 EP10 EP11 EP12 EP13 EP14 EP15 EP16 

164 166 164 166 148 150 148 150 

166 164 166 164 150 148 150 148 

162 160 162 160 158 156 158 156 

160 162 160 162 156 158 156 158 

156 158 156 158 164 166 164 166 

158 156 158 156 166 164 166 164 

154 152 154 152 154 152 154 152 

152 154 152 154 152 154 152 154 

148 150 148 150 160 162 160 162 

150 148 150 148 162 160 162 160 
34

Appendix E: Post Run Questionnaires 

The evaluation pilot completed a questionnaire (see below) at the end of each evaluation run. Mental 

Demand, Physical Demand, Temporal Demand, Performance (Pilot), Effort, and Frustration were all Task 

Load Index (TLX) measures that evaluated workload. Demand on and Supply of Attentional Resources 

and Understanding of Situation values were combined to derive the Situation Awareness Rating 

Technique (SART) rating. Level of Terrain Awareness and Stress were two other independent measures 

that were collected. 

MENTAL DEMAND DEMAND ON ATTENTIONAL RESOURCES 

 Low High Low High 

PHYSICAL DEMAND SUPPLY OF ATTENTIONAL RESOURCES 

 Low High Low High 

TEMPORAL DEMAND UNDERSTANDING OF THE SITUATION 

 Low High Low High 

PERFORMANCE LEVEL OF TERRAIN AWARENESS 

 Good Poor Low High

EFFORT STRESS 

 Low High Low High 

FRUSTRATION 

 Low High

 Workload: Situational Awareness: 

 TLX - Mental Demand SART - Demand on Attentional Resources 

 Physical Demand Supply of Attentional Resources 

 Temporal Demand Understanding of the Situation 

 Performance 

 Effort 

 Frustration 
35

Run Questionnaire Key: 

Title Descriptions 

MENTAL DEMAND How much mental and perceptual activity 

was required (e.g., thinking, deciding, 

calculating, remembering, looking, 

searching, etc.)? Was the task easy or 

demanding, simple or complex, exacting or 

forgiving? 

PHYSICAL DEMAND How much physical activity was required 

(e.g., pushing, pulling, turning, controlling, 

activating, etc.)? Was the task easy or 

demanding, slow or brisk, slack or 

strenuous, restful or laborious? 

TEMPORAL DEMAND How much time pressure did you feel due to 

the rate or pace at which the tasks or task 

elements occurred? Was the pace slow and 

leisurely or rapid and frantic? 

PERFORMANCE How successful do you think you were in 

accomplishing the goals of the task set by 

the experimenter (or yourself)? How 

satisfied were you with your performance in 

accomplishing these goals? 

EFFORT How hard did you have to work (mentally 

and physically) to accomplish your level of 

performance? 

FRUSTRATION LEVEL How insecure, discouraged, irritated, 

stressed and annoyed versus secure, 

gratified, content, relaxed and complacent 

did you feel during the task? 
36

Please rate your overall impression of the scenario in terms of how much attention and effort was required 

to perform the scenario successfully. Things to consider are the degree of instability, complexity, and 

variability that you perceived while flying the scenario. 

DEMAND ON ATTENTIONAL RESOURCES 

Low High

Please rate your overall impression of the scenario in terms of the amount of “spare” attention to give to 

other tasks. Was 100% of your attention directed towards successfully completing the scenario? Or, 

could you have completed other sub-tasks while flying the scenario? Things to consider include your 

level of arousal, level of concentration, and if you attention was divided across many sub-tasks. 

SUPPLY OF ATTENTIONAL RESOURCES 

Low High

Please rate your overall understanding of what was happening with the aircraft. Mark on the line below 

the degree to which you felt confident that you were aware of the elements in your environment. Things 

to consider include the level of information quantity and quality as well as familiarity that you felt you 

had with what was taking place during the scenario. 

UNDERSTANDING OF THE SITUATION 

Low High

Please rate you overall understanding of the terrain environment you were operating within. Things to 

consider for this response are how comfortable were you with your terrain awareness. 

LEVEL OF TERRAIN AWARNESS 

Low High

Please rate your overall level of stress that you experienced while completing the experimental test run. 

STRESS 
37

1. Please rate your level of traffic awareness experienced with the display concept during the approach. 

1 = Low; 10 = High 

2. Please rate your level of awareness of where you were (ownship position awareness) using the display 

concept during the approach. 1 = Low; 10 = High 

3. Please rate your ability to stay on path (flight path awareness) using the display concept during the 

approach. 1 = Low; 10 = High 
38

Appendix F. Rare Event Run Questionnaire Results 

The following questionnaire was administered to the EP at the conclusion of the rare event runway 

incursion run. Four different display conditions were used for the rare event testing as follows: 

EP # Display Condition 

1, 5, 9, 13 BMO 

2, 6, 10, 14 BMOT 

3, 7, 11, 15 BAMO 

4, 8, 12, 16 BAMOT 

Only half of the EPs received incursion alerts for the rare event test run. 

1. If applicable (BAMO and BAMOT only), please rate the timeliness of the alert (did the alert happen 

in a timely manner) in terms of providing enough time to take evasive action. 

1 = Poor; 10 = Excellent 

EP BAMO EP BAMOT 

3 8 4 7 

7 5 8 8 

11 9 12 10 

15 9 16 9 

Mean 7.75 Mean 8.5 

Std. Dev. 1.89 Std. Dev. 1.29 

2. Upon receiving an incursion alert, was your immediate reaction to : 

Confirm the hazard; Take evasive action; Other: Explain Reason 

EP # Confirm Hazard Take Evasive Action Other Reason 

1 N/A 

2 N/A 

3  Would have queried ATC with warning 

4  Already confirmed traffic prior to alert 

5 N/A 

6 N/A 

7   Need radio response. Initiate each, can 

continue if valid or realign if not. 

8  

9 N/A 

10 N/A 

11  
39

12  Unfamiliar with warning 

13 N/A 

14 N/A 

15  Saw first – alert confirmed visual 

16  

Total 5 4 

3. Please indicate where you first perceived the location of the incursion aircraft. 

Window; Surface Map (if applicable); Audible Alert (if applicable) 

If applicable, upon seeing the incursion aircraft, where did you then focus your attention? 

Window; Surface Map (if applicable); Audible Alert (if applicable) 

If applicable, please estimate how useful each of the following were in helping you to detect and avoid 

the runway incursion situation (total 100%) 

Window; Surface Map (if applicable); Audible Alert (if applicable) 

First Perceived Location Focus Attention Detection Usefulness (%) 

EP # Window Map Audible Window Map Audible Window Map Audible 

1  N/A N/A  N/A 100 0 N/A 

2  N/A  N/A 100 0 N/A 

3 N/A   10 0 90 

4   40 60 0 

5  N/A N/A  N/A 100 0 N/A 

6  N/A  N/A 80 20 N/A 

7  N/A  95 0 5 

8   50 50 0 

9  N/A N/A  N/A 100 0 N/A 

10 - - N/A - - N/A - - N/A 

11  N/A  100 0 0 

12   0 40 60 

13  N/A N/A  N/A 100 0 N/A 

14  N/A - - N/A - - N/A 

15  N/A  80 0 20 

16   33 33 34 

Total 12 1 2 10 3 1 70.6 14.5 26.1 
40

4. If applicable, how did you use the surface map to detect and locate the incursion aircraft?

EP 1 – N/A 

EP 2 – N/A 

EP 3 – N/A 

EP 4 – Saw traffic location on the map and then confirmed the traffic by looking out the window. Looked 

at map afterwards for additional confirmation. 

EP 5 – N/A 

EP 6 – N/A 

EP 7 – N/A 

EP 8 – I was looking at the display to confirm my visual out-the-window look when it lit up red. 

EP 9 – N/A 

EP 10 – N/A 

EP 11 – N/A 

EP 12 – Upon audible, I confirmed on map then visually. 

EP 13 – None 

EP 14 – No response 

EP 15 – Out of field of view. 

EP 16 – Confirm visual. 

If applicable, please indicate whether the surface map alone was sufficient to help you detect and 

locate the intrusion aircraft? Yes; No; Likelihood of detection (%) 

EP # Yes No Likelihood of Detection (%) 

1 

2  0 

3 N/A 

4  75 

5 N/A 

6 N/A 

7 N/A 

8  100 

9 

10 

11 

12 

13 N/A 

14 

15 

16 

Total 1 10 
41

What enhancements could be made to the surface map to raise the likelihood? 

EP 2 – More attention getting symbol, location of display is too far from site picture 

EP 4 – Highlight traffic before alert generated. A longer warning on map with audible warning meaning 

imminent danger. 

EP 6 – Location 

EP 7 – Traffic of hazard to this aircraft, should have bright/flash indication of hazard with warning 

EP 11 – Adding traffic 

EP 12 – Positioning 

EP 14 – Placement 

EP 15 – Place within field of view. 

EP 16 – Too cluttered. 

5. If applicable, please indicate whether the audible alert alone was sufficient to help you detect and 

locate the intrusion aircraft? Yes; No; Likelihood (%) 

EP # Yes No Likelihood of Detection (%) 

1 N/A 

2 N/A 

3  90 

4 No answer 

5 N/A 

6 N/A 

7  25 

8 

9 N/A 

10 N/A 

11  100 

12  

13 N/A 

14 N/A 

15  

16 

Total 5 2 

What enhancements could be made to the audible alert (e.g. 3-D Audio) to raise the likelihood? 

EP 3 – Position 

EP 7 – Need clear beep/squawk to alert about the alert. Plus voice distinct from ATC. 

EP 11 – Audio is OK 

EP 12 – None 

EP 15 – Tone 

EP 16 – Female with British accent. 
42

6. Please indicate your rating of how much aid the display concept provided in detection of the 

incursion aircraft. 1 = None; 10 = Exceptional 

EP BMO EP BMOT EP BAMO EP BAMOT 

1 1 2 1 3 8 4 8 

5 1 6 4 7 2 8 10 

9 1 10 1 11 8 12 9 

13 1 14 2 15 7 16 8 

Mean 1 Mean 2 Mean 6.25 Mean 8.75 

Std. Dev. 0 Std. Dev. 1.41 Std. Dev. 2.87 Std. Dev. 0.96 

Collective 

EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

 1 1 8 8 1 4 2 10 1 1 8 9 1 2 7 8 

Mean 4.5 

Std. Dev. 3.58 

7. Please indicate the likelihood that the display concept would prevent similar runway incursions in 

the real world (based on your experiences). 1 = Low; 10 = High 

EP BMO EP BMOT EP BAMO EP BAMOT 

1 1 2 10 3 9 4 9 

5 1 6 9 7 4 8 10 

9 5 10 8 11 8 12 10 

13 1 14 5 15 10 16 10 

Mean 2 Mean 8 Mean 7.75 Mean 9.75 

Std. Dev. 2 Std. Dev. 2.16 Std. Dev. 2.63 Std. Dev. 0.5 

Collective 

EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

 1 10 9 9 1 9 4 10 5 8 8 10 1 5 10 10 

Mean 6.88 

Std. Dev. 3.5 

8. Please rate your level of perceived safety you believe you would experience if you had this system 

onboard your aircraft during a similar runway incursion event. 1 = Low; 10 = High 

EP BMO EP BMOT EP BAMO EP BAMOT 

1 1 2 10 3 7 4 8 

5 1 6 9 7 7 8 10 

9 5 10 7 11 6 12 10 

13 5 14 5 15 10 16 9 

Mean 3 Mean 7.75 Mean 7.5 Mean 9.25 

Std. Dev. 2.31 Std. Dev. 2.22 Std. Dev. 1.73 Std. Dev. 0.96 
43

Collective 

EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

 1 10 7 8 1 9 7 10 5 7 6 10 5 5 10 9 

Mean 6.88 

Std. Dev. 2.94 

9. Please rate (if applicable) the effectiveness of the display concept compared to what is currently 

available onboard Part 23 aircraft for prevention of runway incursions. 1 = None; 10 = Exceptional 

EP BMO EP BMOT EP BAMO EP BAMOT 

1 1 2 10 3 9 4 10 

5 1 6 10 7 4 8 10 

9 1 10 7 11 8 12 10 

13 5 14 5 15 8 16 9 

Mean 2 Mean 8 Mean 7.25 Mean 9.75 

Std. Dev. 2 Std. Dev. 2.5 Std. Dev. 2.22 Std. Dev. 0.5 

Collective 

EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

 1 10 9 10 1 10 4 10 1 7 8 10 5 5 8 9 

Mean 6.75 

Std. Dev. 3.45 

Please indicate the reason for your rating. 

EP 1 – The display concept did not show aircraft on it. 

EP 2 – No airport display / map in current aircraft. 

EP 3 – One more piece of information. 

EP 4 – No current indications of traffic available. This display concept gives exceptional traffic 

information. 

EP 5 – System is not indicating traffic. 

EP 6 – Like having co-pilot and second pair of eyes. 

EP 7 – Definite potential for improvement. 

EP 10 – Enhances traffic awareness if used the technology & knew how to use it & part of scan 

EP 11 – Less subjective type of information 

EP 12 – Only visual on Part 23 in low visibility conditions. Audible and surface map make a big 

difference. 

EP 13 – No display of traffic. 

EP 15 – Focus attention where many tasks may be involved 

EP 16 – Gives more information than just visual – gets pilot’s attention. 
44

10. Please provide any other comments or suggestions for improvement of the efficacy of the display 

concept for prevention of runway incursions. 

EP 1 – Needs to show aircraft on it. 

EP 2 – Ability to use software to make numbers, dials, letters, symbols bigger/smaller, brighter/dimmer. 

EP 6 – Automatic range scaling. 

EP 8 – Because of current vs prior experience. 

EP 10 – On instrument approaches would rely more on ATC. 
45

Appendix G: Usability Study Display Evaluation Questionnaire Results 

The EPs evaluated the seven display conditions after each test scenario. The acronyms for the display 

conditions are defined as follows: 

B Baseline 

BMO Baseline with Surface Map and Ownship 

BMOT Baseline with Surface Map and Ownship and Traffic 

BA Baseline with Audible Incursion Alerts 

BAMO Baseline with Surface Map and Ownship and Audible Incursion Alerts 

BAMOT Baseline with Surface Map and Ownship, Traffic, and Incursion Alerts 

BRIPS Baseline with Perspective Surface Map 

1. Please rate the likelihood that the display configuration would prevent the runway incursion 

scenario evaluated. 1 = low; 10 = high 

Scenario 1: arrival / takeoff hold

EP # B BMO BMOT BA BAMO BAMOT BRIPS 

1 6 6 8 7 7 10 9 

2 1 1 5 5 5 8 8 

3 1 1 4 6 6 7 8 

4 3 3 4 7 7 10 10 

5 1 1 8 10 10 10 10 

6 0 0 3 8 8 10 8 

7 1 4 5 4 5 8 9 

8 1 1 4 10 10 10 10 

9 1 1 6 7 7 9 9 

10 1 1 5 7 7 8 8 

11 1 1 5 8 8 10 10 

12 1 1 5 8 8 10 10 

13 1 2 5 4 6 8 9 

14 1 1 2 7 7 9 8 

15 1 1 3 7 7 8 9 

16 1 1 3 7 7 9 9 

Mean 1.38 1.63 4.69 7 7.19 9 9 

Std. Dev. 1.36 1.5 1.66 1.71 1.42 1.03 0.82 
46

Scenario 2: departure / intersection departure

EP # B BMO BMOT BA BAMO BAMOT BRIPS 

1 6 6 8 7 7 10 10 

2 1 2 6 6 6 8 9 

3 1 1 3 6 6 8 8 

4 2 2 7 7 7 10 10 

5 1 1 3 9 9 9 10 

6 0 0 3 8 8 10 8 

7 1 3 4 4 5 9 8 

8 1 1 4 10 10 10 10 

9 1 3 6 7 8 9 9 

10 1 1 5 6 6 9 9 

11 1 1 6 8 8 10 10 

12 1 1 5 10 10 10 10 

13 1 2 5 4 7 9 8 

14 1 2 2 7 7 9 8 

15 1 1 2 7 7 7 8 

16 1 2 3 7 7 9 9 

Mean 1.31 1.81 4.5 7.06 7.38 9.13 9 

Std. Dev. 1.3 1.38 1.79 1.73 1.41 0.89 0.89 

Scenario 3: arrival / departure

EP # B BMO BMOT BA BAMO BAMOT BRIPS 

1 4 4 5 6 6 10 9 

2 1 2 5 7 8 9 10 

3 1 1 4 7 7 8 8 

4 2 2 4 7 7 10 10 

5 1 1 3 8 9 10 10 

6 1 3 6 8 8 10 10 

7 1 3 5 2 5 8 9 

8 1 1 1 10 10 10 10 

9 1 1 4 8 8 8 8 

10 1 1 5 7 7 8 8 

11 1 1 5 8 8 10 10 

12 1 1 4 8 8 10 10 

13 1 1 5 5 5 8 8 

14 1 1 1 2 2 3 3 

15 1 1 2 8 8 9 10 

16 1 1 3 7 8 9 9 

Mean 1.25 1.56 3.88 6.75 7.13 8.75 8.88 

Std. Dev. 0.77 0.96 1.5 2.14 1.89 1.77 1.78 
47

Scenario 4: departure / departure

EP # B BMO BMOT BA BAMO BAMOT BRIPS 

1 4 4 6 6 6 10 10 

2 1 2 6 8 8 9 10 

3 1 1 4 7 7 8 8 

4 1 2 5 6 6 9 9 

5 0 1 3 10 10 10 10 

6 0 2 6 8 8 10 8 

7 1 4 5 5 6 9 9 

8 1 1 3 10 10 10 10 

9 1 1 4 8 8 8 8 

10 1 1 6 7 8 8 8 

11 1 1 7 8 8 10 10 

12 1 1 4 10 10 10 10 

13 1 3 5 5 7 9 8 

14 1 1 3 7 7 9 8 

15 1 1 3 7 7 8 8 

16 1 1 3 7 7 9 9 

Mean 1.06 1.69 4.56 7.44 7.69 9.13 8.94 

Std. Dev. 0.85 1.08 1.36 1.59 1.35 0.81 0.93 

Scenario 5: taxi crossing / departure

EP # B BMO BMOT BA BAMO BAMOT BRIPS 

1 4 4 6 7 7 10 10 

2 1 3 6 7 8 9 10 

3 1 1 4 6 7 7 8 

4 2 2 7 5 5 9 9 

5 1 1 8 10 10 10 10 

6 0 3 6 8 8 10 8 

7 1 2 6 4 5 9 8 

8 1 1 10 10 10 10 10 

9 1 1 6 7 7 9 8 

10 1 1 5 7 7 10 10 

11 1 1 8 8 8 10 10 

12 1 1 6 8 8 10 10 

13 1 3 5 6 7 9 8 

14 1 1 5 7 7 9 8 

15 1 1 2 8 8 9 9 

16 1 2 2 5 7 9 9 

Mean 1.19 1.75 5.75 7.06 7.44 9.31 9.06 

Std. Dev. 0.83 1.0 2.05 1.65 1.36 0.79 0.93 
48

2. Please rate the efficacy of the display concept for runway incursion prevention with and without 

alerting for the following display concepts. 1 = low; 10 = high 

Scenario 1 – arrival / takeoff hold EP # B + no alert B + audible alert BMO + no alert BMO + audible alert 

BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 

1 5 5 5 6 6 6 6 8 9 10 7 7 8 9 

2 1 6 1 5 4 6 5 6 5 8 5 5 5 8 

3 1 5 1 6 4 4 4 7 6 7 4 8 7 8 

4 3 7 3 7 4 8 8 9 8 10 8 9 8 10 

5 1 10 1 10 8 10 8 10 8 10 8 10 8 10 

6 0 8 0 8 3 8 3 8 8 10 3 8 8 8 

7 1 4 4 5 5 7 5 7 6 8 6 8 7 9 

8 1 10 1 10 4 10 4 10 4 10 4 10 4 10 

9 1 7 1 7 6 7 6 9 6 9 6 9 7 9 

10 1 7 1 7 6 7 5 7 6 8 5 7 6 8 

11 1 8 1 8 6 8 5 8 6 10 5 8 6 10 

12 1 8 1 8 8 10 5 10 8 10 5 10 8 10 

13 1 4 2 6 3 7 5 7 6 8 5 6 6 9 

14 1 7 1 7 1 7 2 7 2 9 2 7 2 8 

15 1 7 1 7 4 7 3 8 6 8 4 9 7 9 

16 1 7 1 7 5 7 3 9 7 9 3 9 7 9 

Mean 1.31 6.94 1.56 7.13 4.81 7.44 4.81 8.13 6.31 9.0 5.0 8.13 6.5 9.0 

Std. 

Dev. 

1.14 1.73 1.31 1.45 1.83 1.59 1.68 1.26 1.74 1.03 1.71 1.45 1.67 0.82 
49

Scenario 2 – departure / intersection departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert BMO + audible & 

visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 

1 6 7 6 7 7 7 8 9 9 10 8 9 9 10 

2 1 6 2 6 5 5 6 6 5 8 6 8 7 9 

3 1 6 1 6 4 7 3 6 4 8 4 6 4 8 

4 2 7 2 7 4 8 7 9 8 10 7 9 8 10 

5 1 10 1 10 8 10 7 10 8 10 8 10 9 10 

6 0 8 0 8 3 8 3 8 6 10 3 8 6 8 

7 1 4 3 5 4 6 4 8 7 9 5 7 6 8 

8 1 10 1 10 1 10 4 10 4 10 4 10 4 10 

9 1 7 3 8 4 8 6 8 6 9 5 9 6 9 

10 1 6 1 6 5 6 5 6 5 9 5 6 5 9 

11 1 8 1 8 7 9 6 10 9 10 7 10 9 10 

12 1 10 1 10 8 10 1 10 8 10 1 10 8 10 

13 1 4 2 7 5 7 5 7 7 9 4 7 7 8 

14 1 7 2 7 2 7 2 7 2 9 2 7 2 8 

15 1 7 1 7 2 6 2 7 3 7 3 7 3 8 

16 1 7 2 7 5 8 3 9 7 9 3 9 7 9 

Mean 1.31 7.13 1.81 7.44 4.63 7.63 4.5 8.13 6.13 9.19 4.69 8.25 6.25 9.0 

Std. 

Dev. 

1.3 1.82 1.38 1.5 2.09 1.54 2.07 1.5 2.13 0.91 2.09 1.44 2.18 0.89 
50

Scenario 3 – arrival / departure EP # B + no alert B + audible alert BMO + no alert BMO + audible 

alert BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 

1 5 6 5 6 6 6 8 9 9 10 8 9 9 10 

2 1 7 2 8 7 8 5 8 7 9 6 7 7 10 

3 1 7 1 7 4 7 4 7 6 8 3 7 4 8 

4 2 7 2 7 3 8 4 10 4 10 4 10 4 10 

5 1 8 1 9 2 9 3 10 2 10 3 10 3 10 

6 1 8 3 8 6 10 6 8 6 10 6 8 6 10 

7 1 2 3 5 4 6 5 6 7 8 5 8 7 9 

8 1 10 1 10 1 10 1 10 1 10 1 10 1 10 

9 1 8 1 8 4 8 4 8 4 8 4 8 4 8 

10 1 7 1 7 6 8 5 7 7 8 5 7 7 8 

11 1 8 1 8 6 9 5 9 7 10 6 9 7 10 

12 1 8 1 8 4 10 4 8 8 10 8 10 8 10 

13 1 5 1 5 4 7 5 7 6 8 5 7 7 8 

14 1 2 1 2 2 2 1 3 2 3 1 3 2 3 

15 1 8 1 8 3 8 2 9 4 9 2 9 4 10 

16 1 7 1 8 4 9 3 7 4 9 3 7 4 9 

Mean 1.31 6.75 1.63 7.13 4.13 7.81 4.06 7.88 5.25 8.75 4.38 8.06 5.25 8.94 

Std. 

Dev. 

1.01 2.14 1.15 1.89 1.71 2.01 1.81 1.78 2.32 1.77 2.16 1.81 2.29 1.81 
51

Scenario 4 – departure / departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert 

BMO + visual alert BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 

1 5 6 5 6 6 6 8 9 9 10 8 9 9 10 

2 1 8 2 8 7 8 6 8 8 9 7 8 7 10 

3 1 7 1 7 6 8 4 7 6 8 4 8 5 8 

4 1 6 2 6 5 7 5 7 5 9 5 7 5 9 

5 1 10 1 10 3 10 3 10 3 10 3 10 3 10 

6 0 8 2 8 2 8 6 8 6 10 2 8 6 10 

7 1 5 4 6 5 7 5 7 6 9 5 7 6 9 

8 1 10 1 10 5 10 3 10 5 10 2 10 2 10 

9 1 8 1 8 4 8 4 8 4 8 4 8 4 8 

10 1 7 1 8 6 8 6 8 6 8 6 8 6 8 

11 1 8 1 8 8 10 7 10 10 10 7 10 10 10 

12 1 10 1 10 4 10 4 10 4 10 4 10 4 10 

13 1 5 3 7 5 7 5 7 6 9 3 6 5 8 

14 1 7 1 7 2 7 3 7 3 9 2 7 2 8 

15 1 7 1 7 3 7 3 7 4 8 3 7 4 8 

16 1 7 1 7 5 7 3 7 5 9 5 9 7 9 

Mean 1.19 7.44 1.75 7.69 4.75 8 4.69 8.13 5.63 9.13 4.38 8.25 5.31 9.06 

Std. 

Dev. 

1.05 1.59 1.24 1.35 1.69 1.32 1.58 1.26 2.00 0.81 1.89 1.29 2.24 0.93 
52

Scenario 5 – taxi crossing / departure EP # B + no alert B + audible alert BMO + no alert BMO + audible alert BMO + visual alert 

BMO + audible & visual alert BMOT + no alert BMOT + audible alert BMOT + visual alert BMOT + audible & visual alert BRIPS + no alert BRIPS + audible alert BRIPS + visual alert BRIPS + audible & visual alert 

1 4 6 4 6 5 6 6 8 7 10 6 8 7 10 

2 1 7 3 8 7 8 6 8 7 9 6 8 8 10 

3 1 6 1 5 4 7 4 7 6 7 5 7 6 8 

4 2 5 2 5 5 6 7 9 7 9 6 9 6 9 

5 1 10 1 10 9 10 8 10 9 10 8 10 9 10 

6 0 8 3 8 6 8 6 8 6 10 8 8 6 8 

7 1 4 2 5 4 6 6 8 7 9 5 7 6 8 

8 1 10 1 10 5 10 10 10 10 10 10 10 10 10 

9 1 7 1 7 5 7 6 9 6 9 6 8 6 8 

10 1 7 1 7 4 8 5 7 6 10 5 7 6 10 

11 1 8 1 8 9 9 8 10 9 10 8 10 9 10 

12 1 8 1 8 6 8 6 10 9 10 6 10 9 10 

13 1 6 3 7 6 8 5 8 8 9 6 7 7 8 

14 1 7 1 7 2 7 5 8 5 9 5 8 4 8 

15 1 8 1 8 7 8 2 8 7 9 2 8 7 9 

16 1 5 2 7 5 9 2 9 5 9 9 9 9 9 

Mean 1.19 7.00 1.75 7.25 5.56 7.81 5.75 8.56 7.13 9.31 6.31 8.38 7.19 9.06 

Std. 

Dev. 

0.83 1.67 1.00 1.53 1.82 1.28 2.05 1.03 1.5 0.79 1.92 1.15 1.64 0.93 
53

3. Please rate surface traffic awareness for the following display concepts. 1 = low; 10 = high 

Scenario 1 – arrival / takeoff hold

EP # B BMO BMOT BRIPS 

1 4 4 8 10 

2 1 2 8 8 

3 1 1 6 7 

4 1 1 10 10 

5 1 1 10 10 

6 0 0 8 6 

7 1 5 8 9 

8 1 1 10 10 

9 1 1 6 6 

10 1 1 9 9 

11 1 1 8 8 

12 1 1 8 8 

13 1 1 6 5 

14 1 1 9 8 

15 1 1 3 4 

16 1 2 7 7 

Mean 1.13 1.5 7.75 7.81 

Std. Dev 0.81 1.26 1.84 1.87 

Scenario 2 – departure / intersection departure

EP # B BMO BMOT BRIPS 

1 4 4 9 10 

2 1 2 8 9 

3 1 1 6 6 

4 1 1 9 9 

5 1 1 8 10 

6 0 0 8 6 

7 1 5 9 8 

8 1 1 10 10 

9 1 3 6 8 

10 1 1 9 9 

11 1 1 8 8 

12 1 1 6 6 

13 1 3 7 6 

14 1 1 6 5 

15 1 1 4 5 

16 1 2 5 5 

Mean 1.13 1.75 7.38 7.5 

Std. Dev. 0.81 1.34 1.71 1.90 
54

Scenario 3 – arrival / departure

EP # B BMO BMOT BRIPS 

1 4 4 10 10 

2 1 2 9 10 

3 1 1 6 6 

4 1 1 8 8 

5 0 0 8 8 

6 0 3 8 6 

7 1 6 8 9 

8 10 10 10 10 

9 1 1 4 4 

10 1 1 9 9 

11 1 1 8 8 

12 1 1 5 6 

13 1 1 5 5 

14 1 1 3 3 

15 1 1 4 4 

16 1 1 5 5 

Mean 1.63 2.19 6.88 6.94 

Std. Dev. 2.39 2.56 2.28 2.35 

Scenario 4 – departure / departure

EP # B BMO BMOT BRIPS 

1 5 5 9 10 

2 1 2 8 10 

3 1 1 6 6 

4 1 1 7 7 

5 1 1 3 3 

6 0 0 6 6 

7 1 2 9 9 

8 1 1 10 10 

9 1 3 4 4 

10 1 1 9 9 

11 1 1 10 10 

12 1 1 4 4 

13 1 3 8 7 

14 1 1 3 2 

15 1 1 3 4 

16 1 1 7 7 

Mean 1.19 1.56 6.63 6.75 

Std. Dev. 1.05 1.21 2.55 2.74 
55

Scenario 5 – taxi crossing / departure

EP # B BMO BMOT BRIPS 

1 5 5 10 10 

2 1 3 9 10 

3 1 1 6 7 

4 1 1 8 7 

5 1 1 10 10 

6 0 0 10 8 

7 1 4 9 8 

8 1 1 10 10 

9 1 1 8 8 

10 1 1 9 9 

11 1 1 8 8 

12 1 1 6 6 

13 1 3 6 5 

14 1 1 8 7 

15 1 1 2 2 

16 1 1 9 9 

Mean 1.19 1.63 8 7.75 

Std. Dev. 1.0 5 1.36 2.13 2.14 
56

Appendix H: Usability Study Detection Algorithm Evaluation Questionnaire 

Results 

The EPs evaluated the two incursion detection algorithms after each test scenario. The scenarios are 

defined as follows: 

Scenario 1 arrival / takeoff hold 

Scenario 2 departure / intersection departure 

Scenario 3 arrival / departure 

Scenario 4 departure / departure 

Scenario 5 taxi crossing / departure 

1. Please rate the effectiveness of the alerting for preventing runway incursions for each display 

concept (based on the runway incursion scenario experienced). 1 = low; 10 = high 

Note: All evaluations were made using the BRIPS display concept (Baseline with perspective map). 

Scenario 1 Scenario 2 Scenario 3 Scenario 4 Scenario 5 

EP # Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

1 10 10 10 9 10 6 10 7 10 5 

2 9 10 10 10 10 10 10 10 9 10 

3 7 8 7 7 7 8 7 8 7 

4 8 9 9 9 7 7 8 7 7 4 

5 10 10 10 10 3 10 10 10 10 3 

6 7 10 10 10 10 10 10 9 10 9 

7 8 9 9 9 8 9 9 8 9 8 

8 10 6 10 8 10 10 10 10 10 10 

9 8 9 8 8 8 8 8 8 8 

10 8 9 8 8 8 8 8 8 8 9 

11 9 10 10 10 10 10 10 10 10 10 

12 9 10 10 9 9 10 10 10 10 

13 8 9 9 8 9 9 9 9 8 

14 8 9 8 8 8 8 8 7 8 8 

15 10 9 10 10 9 10 4 10 8 

16 9 9 9 9 9 9 9 9 9 

Mean 8.63 9.13 9.19 8.88 8.44 8.8 9.19 8.31 9.06 7.88 

Std. Dev. 1.02 1.02 0.98 0.96 1.79 1.48 0.91 1.66 1.00 2.16 

Comments:

EP1 Scenario 2 two stage – alerted later 

EP4 Scenario 1 two stage – more time, chance to evaluate 

EP7 Scenario 1 two stage faster 

 Scenario 3 based on how landed, thought two stage alerted sooner 
57

 Scenario 4 single stage faster 

 Scenario 5 single stage faster 

EP9 Scenario 5 caution didn’t buy anything 

EP13 Scenario 5 caution not beneficial – too close to warning 

EP14 Scenario 5 single stage – did not perceive difference in alerting 

 two stage – not useful as implemented 

EP15 Scenario 5 two stage – no need for two alerts in this case 

2. In general, which type of alerting is better for single pilot general aviation operations? 

Single stage alerting or two stage alerting 

For the choice selected, how much more effective is that alerting type for runway incursion 

prevention? 

1 = low; 10 = high 

Please provide a reason for your answer. 

EP # Single 

Stage 

Effectiveness Two 

Stage 

Effectiveness Reason 

1 1 It gave more time to react with single stage in 

sim. I’d rather have two stage alerting with the 

alerting time of single stage. 

2 1 2 Both are excellent, two stage provides a higher 

comfort level. 

3 1 4 It is better to have the caution which seems to 

give you more time to evaluate the situation and 

react accordingly. 

4 1 4 1 6 Two stage for approaches only. Single stage 

alerting allowed more time for response during 

ground ops. The two stage, however, allows the 

pilot to know when to start looking on approach 

for other aircraft when concentrating on other 

tasks. 

5 1 3 Seems redundant but a second warning may help 

if the first is not heard. 

6 1 8 With one you first react. With two your mind 

makes you analyze what is happening. 

7 1 8 Two stage seems to provide earlier alerting of 

hazards, enabling pilot attention and decision 

making – proactive responses. Like selective on 

stage in certain cases (takeoff ...) 

8 1 8 (no comment) 

9 1 2 Earlier notification 

10 1 1 The two stage gave me a chance to look at the 

display prior to the warning. Better SA. 

11 1 3 Single is very good, but two stage offers a little 

more cushion. 

12 1 3 Caution can be confusing 

13 1 4 More time to prepare before acting 
58

14 1 3 Early alert for more reaction time 

15 1 9 Late alert for two stage 

16 1 4 Couldn’t tell a big difference 

Total 6 6.4 11 3.64 

3. With respect to latency, were the incursion alerts provided in a timely manner allowing sufficient 

time to react to the potential conflict? 

All 16 EPs answered YES 

Did either single stage or two stage alerting provide more time for response to the RI? 

Please estimate how much more time the alert provided. 

If not, when should the alert have occurred? 

EP # Single 

Stage 

Two Stage Seconds Comment 

1  2 

2  5 to 10 

3   3 to 4 Single stage for taxi and departure, two stage for approach 

4  5 to 10 Two stage taxi (scenario 5) not enough time. Two stage more time 

on approach. 

5 Same If any thing, alert should have occurred when the other plane 

begins takeoff roll. Two stage for scenario 3 & 5 was late. 

6  1 to 1.5 Except for departure, alerts should be sooner. 

7  1 to 4 Potentially a lifetime. 

8 Equal 

9  2 Most times single stage is all that is needed. Two stage beneficial 

on approach only. 

10 Neither 

11  2 to 3 

12  1 

13  3 

14   2 Single stage for takeoff, two stage for landing 

15  3 

16  10 to 15 Two stage on approach 

Total 8 7 
59

Appendix I: Final Semi-Structured Interview Questionnaire Results 

A post-test questionnaire was administered to each EP. The acronyms for the display conditions are 

defined as follows: 

B Baseline 

BMO Baseline with Surface Map and Ownship 

BMOT Baseline with Surface Map and Ownship and Traffic 

BA Baseline with Audible Incursion Alerts 

BAMO Baseline with Audible Incursion Alerts, Surface Map and Ownship 

BAMOT Baseline with Incursion Alerts, Surface Map and Ownship, Traffic 

BRIPS Baseline with Perspective Surface Map 

I. RIPS General Safety 

1. In general, please rate the level of safety felt during runway incursion incidents using the following 

equipage. 1 = Not Safe; 10 = Completely Safe 

EP # B BMO BMOT BA BAMO BAMOT BRIPS 

1 4 4 5 6 6 10 10 

2 5 6 7 8 9 10 10 

3 1 2 6 6 6 7 8 

4 2 2 5 5 5 10 10 

5 1 1 7 8 8 9 10 

6 0 0 3 8 8 10 8 

7 1 5 6 6 7 9 8 

8 5 5 8 10 10 10 10 

9 1 3 4 7 7 9 9 

10 6 6 9 10 10 10 10 

11 1 1 8 8 8 10 10 

12 1 1 5 10 10 10 10 

13 1 3 4 5 6 9 8 

14 2 3 3 5 5 9 8 

15 1 1 3 7 7 8 9 

16 1 1 3 6 7 9 9 

Mean 2.06 2.75 5.38 7.19 7.44 9.31 9.19 

Std. Dev. 1.84 1.95 2.00 1.76 1.67 0.87 0.9.1 
60

II. RIPS General Advantages and Disadvantages 

2a. What do you consider the best feature of the RIPS displays/technology you evaluated today? 

Can the feature be improved? If so, how? 

EP # Best Feature Improvement 

1 

Showed movement of other aircraft on airport 

Liked perspective map better; however, the traffic 

icons were harder to see 

2 Provides enhanced situational awareness, 

ownship location, airport environment & traffic 

position & direction moving Capability to vary font size 

3 Audible alert "Warning, traffic 34R" not descriptive enough 

4 Airport layout system for increased situational 

awareness Text clutter adjustments (ID labels) 

5 Audio alerts Little sooner to match the graphic takeoff roll 

6 Top down view, audible & traffic N# no help 

7 Dynamic and real-time situational awareness 

and hazard warning (no comment) 

8 N/A N/A 

9 Audible alert (no comment) 

10 Traffic Warning late sometimes / latency 

11 Traffic + audible altering (no comment) 

12 Audible (no comment) 

13 Combination visual & audible alerts Alert a little sooner 

14 Taxi clearance (no comment) 

15 

Large red warning symbol with audio alert 

Show ownship with larger symbol, use larger 

traffic symbols at threshold 

16 Give warning of incursions when traffic was 

difficult to see (no comment) 
61

2b. What do you consider to be the worst feature of the RIPS displays/technology you evaluated today? 

Can the feature be improved? If so, how? 

EP # Worst Feature Improvement 

1 Text warning didn't get my attention as well as 

audible and text 

Maybe use a short flashing box that highlights the 

warning 

2 Location of display in cockpit - too much eye 

movement for cross check Ability to declutter ID tags readily 

3 Top down map looked busy Less clutter 

4 (no comment) (no comment) 

5 Did not like to watch the rotation of the top 

down view Not really - leave it in as a choice 

6 Perspective view (no comment) 

7 No "worst" feature Minor tune ups 

8 N/A N/A 

9 None (no comment) 

10 (no comment) (no comment) 

11 Location (no comment) 

12 Caution chevron (yellow) hard to see; placement 

of display Flashing chevron - reposition panel 

13 Position of screen Move it 

14 

Clutter, color of symbology 

During warning, change display to show only 

conflict, traffic, speed, distance, time to conflict 

15 

Small symbol for ownship & traffic & delayed 

warning on 2 stage alert 

With 2 stage alert - make warning earlier - 

increase size of symbols of a/c moving on surface 

(traffic and ownship) 

16 Display a little cluttered in top down display (no comment) 
62

2c. What are the advantages and disadvantages of the RIPS displays/technology you evaluated 

compared to the baseline condition (baseline round dials only)? 

EP # Advantages Disadvantages 

1 Pin points your location in relation to the other 

a/c on airport and warnings (audible and text) No disadvantage 

2 Situation awareness Possible sunlight washout of screen 

3 Better situational awareness Training, more to look at 

4 Traffic advisories allow for a much higher sense 

of awareness 

The chance a pilot may focus on guidance line 

during approach 

5 

The pilot can see traffic and know their direction 

and location even in poor visibilities and 

distance locations obscured by airport facilities 

Pilots can be too dependent on the technology and 

less on looking out the window. What if some 

plane is not compatible with the technology and is 

not displayed? 

6 Situational awareness greatly enhanced none 

7 Great improvement in SA and for intuitive and 

information Essentially zero 

8 N/A N/A 

9 Audible + perspective + graphic alert None 

10 

Situation awareness 

Would take training to ensure attention 

distribution optimally and how to interpret the 

warnings 

11 More aware of traffic and surrounding Left placement of screen 

12 Audible alerts + SA Audible caution was confusing 

13 Gives information you need - my position 

relative to traffic & airport features (no comment) 

14 (no comment) (no comment) 

15 Audio alert + red (large) warning lettering & 

symbols on display None 

16 Gave excellent traffic warning especially when 

audio alert used None 

III. RIPS Surface Ops Evaluation 

3. Please indicate your level of agreement with each of the following statements for all display 

concepts. 1 = Low; 10 = High 

3a. Where am I? The display concept provides sufficient awareness of my ownship position with 

respect to runways, taxiways, and stationary objects. 

If unequal rating, please indicate reason. 

If not “completely agree” for Surface Map conditions, what can be done to improve the concept to 

increase awareness? 
63

EP # B BMO BMOT BA BAMO BAMOT BRIPS Reasons Improvement 

1 2 10 10 2 10 10 10 Baseline doesn't 

show my 

orientation on 

airport (no comment) 

2 1 10 10 1 10 10 10 (no comment) (no comment) 

3 1 6 6 1 6 6 7 (no comment) (no comment) 

4 2 8 8 2 8 8 8 

(no comment) 

No vertical 

indications 

5 1 10 10 1 10 10 10 (no comment) (no comment) 

6 1 8 8 1 8 10 8 Easier to 

interpret top 

down (no comment) 

7 1 10 10 2 10 10 9 Perspective 

slightly poor due 

to relative 

visibility of 

certain areas Same comment 

8 6 8 8 6 8 10 10 (no comment) (no comment) 

9 1 9 9 1 9 9 9 Audible provides 

no info for my 

position 

awareness Nothing 

10 4 4 7 6 8 10 10 Airport 

environment + 

traffic + alerting N/A 

11 1 8 8 2 8 8 10 Perspective view (no comment) 

12 1 10 10 1 10 10 10 Surface map 

provides all the 

information you 

need (no comment) 

13 1 8 9 1 9 9 8 Prefer top down 

view on airport 

surface; traffic 

alert without 

map position not 

as helpful 

Move display; 

adjust scale 

14 1 8 8 1 8 8 8 (no comment) (no comment) 

15 1 10 10 2 10 10 10 (no comment) (no comment) 

16 1 9 9 1 9 9 9 

(no comment) 

Draw picture 

of clearance 

Mean 1.63 8.5 8.75 1.94 8.81 9.19 9.125 

Std. 

Dev. 

1.41 1.67 1.24 1.65 1.17 1.17 1.02 

 

3b. Where am I relative to other moving objects? The display concept provides sufficient awareness of 

my ownship position with respect to moving traffic, such as vehicles and other aircraft. 

If unequal rating, please indicate reason. 
64

If not “completely agree” for surface map with traffic conditions, what can be done to improve the 

concept to increase awareness? 

EP # B BMO BMOT BA BAMO BAMOT BRIPS Reasons Improvement 

1 1 1 9 1 1 9 10 No traffic info 

without map 

(baseline rounds 

dials only) 

Perspective 

view gave a 

broader view of 

airport traffic 

2 1 1 8 5 5 10 10 

(no comment) 

Audible/ visual 

alerting 

enhances SA to 

moving traffic 

3 2 2 6 2 2 6 7 (no comment) (no comment) 

4 1 1 8 3 3 9 9 Alert brings any 

incursions to the 

users notice (no comment) 

5 1 1 10 2 2 10 10 Audible alerting 

provides info for 

one a/c not all (no comment) 

6 0 0 8 6 6 10 8 Like top down (no comment) 

7 1 2 8 2 2 10 9 Need to see traffic 

+ self on map (no comment) 

8 4 4 8 6 6 10 10 (no comment) (no comment) 

9 1 1 6 1 1 8 8 Msg & audible 

alerts adds value on 

other traffic 

position and need 

to respond (no comment) 

10 4 4 7 6 6 10 10 Elements are 

shown on map + 

relevant features, 

identifiable 

landmarks, etc. (no comment) 

11 5 5 8 7 7 10 10 (no comment) (no comment) 

12 1 1 10 5 5 10 10 (no comment) (no comment) 

13 1 3 8 6 6 9 8 

Prefer top down 

Position of 

display; change 

scale 

14 1 1 8 5 5 9 9 (no comment) (no comment) 

15 1 1 10 1 1 10 10 The 1's don't show 

traffic (no comment) 

16 1 1 9 7 9 9 9 

(no comment) 

If unit could 

show a potential 

conflict in a 

different color 

Mean 1.63 1.81 8.19 4.06 4.25 9.31 9.19 

Std. 

Dev. 

1.41 1.42 1.22 2.26 2.54 1.08 0.98 

 
65

IV. RIPS Display and Alerts 

4. Please indicate the effectiveness of the perspective map display compared to the top-down views for 

prevention of runway incursion. 1 = Low; 10 = High 

Reason for rating. 

EP # Rating Reason 

1 9 The perspective gave a broader view of airport traffic 

2 2 Preference 

3 4 Easier to pick out ownship in less time 

4 0 No difference 

5 5 Top down showed more, perspective was easier to look at 

6 2 Fast orientation to top down 

7 8 Focus on approach environment (for approaches only, perspective more effective) 

8 5 (no comment) 

9 1 Same information and equally as readable 

10 0 Same effectiveness; preference is for 2D top down 

11 1 (no comment) 

12 3 Top down was easier to interpret, perspective items get lost 

13 3 Top down gives broader view to sides & rear 

14 1 (no comment) 

15 0 No difference - both maps displayed the alert warnings satisfactorily 

16 2 Map less cluttered 

Mean 2.88 

Std. Dev. 2.73 

5. When traffic was presented on surface map, was incurison traffic easily discernable? 

Yes or No 

All 16 EPs responded Yes. 

If yes, please indicate how discernable. 1 = Low, 10 = High 

EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

 10 10 7.5 10 9 8 10 5 9 9 6 8 7.5 5 10 10 

Mean 8.38 

Std. Dev. 1.77 

6. Is traffic presentation necessary on surface map to prevent runway incursions? 

Yes or No 

EPs 8 and 15 responded No. The other 14 EPs responded Yes. 
66

If so, please indicate how much additional situation awareness is provided with the addition of traffic 

presentation on the surface map. 

Comparing surface map with ownship TO surface map with ownship and traffic, situation awareness 

enhancement is: 1 = Low; 10 = High 

EP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 

 10 9 8 7 10 8 9 4 10 8 8 10 10 8 9 9 

Mean 8.56 

Std. Dev. 1.56 

7. Is graphical presentation of alerts necessary on surface map to prevent runway incursions? 

Yes or No 

If so, please indicate how much additional situation awareness is provided with the addition of 

graphical alerting on the surface map. 1 = Low; 10 = High

EP # Yes No Rating Comment 

1  10 

2  Graphic not needed if audible alert available 

3  3 

4  But is helpful 

5  9 

6  8 

7  9 

8  3 

9  

10  8 

11  3 

12  4 

13  8 

14  4 

15  

16 

Total 10 6 6.27 Mean 

2.83 Std. Dev. 

8. If a runway incursion situation occurs, which of the following is most likely to bring the event to 

your attention? If multiple, please indicate the rank order. 

OTW = out-the-window 

Map = surface map 

Audible = audible alerting 

Note: For B and BMO, OTW is the only method of acquiring incursion traffic. 
67

EP # 

BMOT / 

OTW 

BMOT / 

map 

BA / OTW 

BA / 

audible 

BMO / 

OTW 

BMO / 

audible 

BAMOT/ 

OTW 

BAMOT/ 

audible 

BAMOT/ 

map 

BRIPS / 

OTW 

BRIPS / 

audible 

BRIPS / 

map 

1 2 1 1 2 1 2 2 3 1 2 3 1 

2 1 2 2 1 2 1 2 1 3 2 1 3 

3 2 1 2 1 2 1 3 1 2 3 1 2 

4 2 1 1 2 1 2 2 3 1 2 3 1 

5 2 1 1 2 1 2 2 3 1 2 3 1 

6 1 2 2 1 2 1 2 1 3 2 1 3 

7 2 1 2 1 2 1 3 1 2 3 1 2 

8 1 2 1 2 1 2 1 2 3 1 2 3 

9 2 1 2 1 2 1 3 1 2 3 1 2 

10 1 2 2 1 2 1 3 1 2 3 1 2 

11 1 2 1 2 1 2 3 2 1 3 2 1 

12 1 2 2 1 2 1 2 1 3 2 1 3 

13 2 1 2 1 2 1 3 1 2 3 1 2 

14 1 2 1 2 1 2 1 2 3 1 2 3 

15 1 2 2 1 2 1 3 1 2 3 1 2 

16 2 1 2 1 2 1 3 1 2 3 1 2 

Mean 1.5 1.5 1.63 1.38 1.63 1.38 2.38 1.56 2.06 2.38 1.56 2.06 

Std. 

Dev. 

0.52 0.52 0.5 0.5 0.5 0.5 0.72 0.81 0.77 0.72 0.81 0.77 

9. Upon receiving an incursion alert, was your immediate reaction to: 

Confirm the hazard, Take evasive action, or Other 

Please give a reason for your answer. 

EP # Confirm Evasive Action Other Reason 

1  I saw traffic first and the audible confirmed traffic incursion 

and I took action. 

2  Limited time to react 

3  (no comment) 

4  Evaluate what the evasive action should be 

5  The audio alerts were given after I saw the incursion 

6  Training that warning requires action 

7   Initiate immediately and confirm, then decide 

8  (no comment) 

9  Need to act quickly 

10  Wasn’t in an emergency situation so was safe to get to safe 

altitude 
68

11  Best course of action 

12  Trusted the alert 

13  I’m a skeptic 

14  (no comment) 

15  At slow speeds best to abort 

16  May miss an approach or create a dangerous situation if 

hazard is not confirmed first 

Total 4 13 0 

What do you feel is the appropriate procedure on receipt of a RI alert for the following? 

Landing, Takeoff, and Taxi 

EP # Landing Takeoff Taxi 

1 Go around Abort Stop 

2 Go around Abort Stop 

3 Go around Abort Stop 

4 Go around Abort Stop 

5 Go around Abort Stop 

6 Go around Abort Stop 

7 Go around Abort Stop 

8 Go around Abort Stop 

9 Go around Abort Stop 

10 Go around Abort Stop 

11 Go around Abort Stop 

12 Go around Abort Stop 

13 Go around Abort Stop 

14 Go around Contention Stop 

15 Go around Abort Stop 

16 Go around Abort Stop 

10. Which display combination would provide a minimal configuration to be effective for runway 

incursion prevention. 

EP # Audible Alert Surface Map with 

Ownship only 

Surface Map with 

Ownship & Traffic 

Baseline Round Dials 

1 

2 

3  

4 

5 

6  

7  

8  
69

9  

10 

11 

12  

13 

14  

15 

16 

Total 10 0 6 1 

For an optimal configuration? 

EP # Audible Alert Surface Map with 

Ownship only 

Surface Map with 

Ownship & Traffic 

Baseline Round Dials 

1  

2  

3  

4  

5  

6  

7  

8  

9  

10  

11  

12  

13  

14  

15  

16  

Total 16 0 16 0 

11. Please rate the effectiveness of the terms used for the incursion alerts. 1 = Low; 10 = High 

EP # “Warning, 

traffic on 

RWY” 

“Caution, 

traffic on 

RWY” 

“ Warning, 

traffic departing 

RWY” 

“Caution, traffic 

departing 

RWY” 

“Warning, 

traffic 

approaching 

RWY” 

“Caution, 

traffic 

approaching 

RWY” 

1 10 10 10 10 10 10 

2 10 10 10 10 10 10 

3 5 5 8 8 8 8 

4 10 10 10 10 10 10 
70

5 9 9 9 9 N/A N/A 

6 10 7 10 7 10 7 

7 9 9 9 9 9 9 

8 10 10 10 10 10 10 

9 9 9 9 9 9 9 

10 10 10 10 10 10 10 

11 10 10 10 10 10 10 

12 10 5 10 5 10 5 

13 9 9 9 9 9 9 

14 8 8 8 8 8 8 

15 8 4 4 8 3 3 

16 10 10 10 10 10 10 

Mean 9.25 8.44 9.13 8.88 9.07 8.53 

Std. Dev. 1.29 2.06 1.54 1.41 1.83 2.10 

If not “completely effective”, would another term be more effective or descriptive? 

How could the alert term be improved (i.e.wording)? 

The following EPs responded to this question: 

EP 3 More specific about location of incursion traffic 

EP 12 Delete cautions 

EP 13 Repeat warning – may not be understood first time 

EP 15 Caution traffic crossing left, to right, right to left 

12a. For single pilot operations, was the two stage alerting (“Caution” & “Warning”) more effective in 

preventing runway incidents than single stage alerting (“Warning” only)? Yes or No 

12b. Did the two stage alerting (“Caution” & “Warning”) enhance your situation awareness (SA) 

more than the single stage alerting (“Warning” only)? Yes or No 

12c. If yes, please indicate how much more the two stage alert enhanced your SA compared to the one 

stage alert. 1 = Minimal; 10 = Substantially 

What was the reason for your rating? 

What are the benefits, if any, of two stage alerting? 
71

12a 12b 12c 

EP # Yes No Yes No Reason Benefits 

1   Gives the pilot time to react 

to threat of incursion 

2   4 Increased comfort level with 

having more perceived time for 

decision making 

3   4 Felt like I had more time Reaction time, scene 

evaluation 

4   4.5 Allowed more time for response 

and reaction when giving a 

caution and warning 

Same comment 

5   Only needed if the first alert 

is not heard 

6   Better that none, single more 

effective 

7   8 Draws attention to hazard earlier 

& provides assessment + 

decision time advantage 

Same comment 

8   

9   2 Very slight improvement in 

timing of information 

Personal option to act sooner 

10   4 Gave opportunity to look at 

surface map to confirm 

More tim to analyze 

situation 

11   4.5 Provided more response time Same Comment 

12   Caution was confusing N/A 

13   1.5 Not a big difference More time to confirm / 

consider 

14   7 Single for takeoff; two stage for 

landing 

Early warning 

15   None 

16   

Total 6 10 9 7 4.39 Mean 

 2.07 Std. Dev. 
72

13. For each display concept, indicate which type of alerting provides the greatest amount of runway 

incursion prevention awareness and the amount of situation awareness enhancement (based on the 

choice you selected) for the following displays: 1 = Minimal; 10 = Substantially 

S = Single stage alert T = Two stage alert SA = Situation awareness enhancement EP # BA + S SA BA + T SA BAMO + S SA BAMO + T SA BAMOT + S SA BAMOT + T SA BRIPS + S 

SA BRIPS + T SA 

1 1 8 1 8 1 10 1 10 

2 1 6 1 7 1 9 1 9 

3 1 5 1 5 1 7 1 8 

4 1 5 1 5 1 9 1 9 

5 1 5 1 5 1 9 1 10 

6 1 8 1 8 1 10 1 8 

7 1 8 1 8 1 8 1 8 

8 1 4 1 4 1 4 1 4 

9 1 2 1 2 1 3 1 3 

10 1 3 1 3 1 3 1 3 

11 1 4 1 4 1 2 1 2 

12 1 10 1 10 1 10 1 10 

13 1 2 1 2 1 2 1 2 

14 1 2 1 2 1 2 1 2 

15 1 7 1 8 1 9 1 10 

16 1 2 1 2 1 2 1 2 

Total 6 10 6 10 7 9 7 9 

Mean 6.5 4.2 6.7 4.3 7.7 5.0 7.6 5.2 

Std. 

Dev. 

2.95 1.99 3.01 2.11 3.30 3.16 3.23 3.42 
73

14. Would you like to be given escape guidance on the maneuver to take to avoid the conflict in 

addition to the runway incursion alert? 

On Final 

Approach 

On Takeoff 

Roll 

When 

Taxiing 

Across a 

Runway? 

What type of advisory / guidance would you like to see 

during landing, during takeoff, or during taxi across a 

runway? 

EP # Yes No Yes No Yes No 

1    Safest maneuver to use to avoid collision 

2    Missed approach instructions 

3    Simple escape guidance (e.g. Climb right, descend left) 

4    Guidance by heading to go behind the conflicting aircraft 

5    Landing: direction to go around; takeoff: “abort takeoff”; 

taxi: “stop taxi” 

6    Go around or missed approach, climb & steering 

commands 

7    Whatever appears to provide relatively maximal 

probability of incident avoidance, resolution of imminent 

hazards, and aircraft/crew survival 

8    If advising, simple command: “hold short” 

9    Landing: course & climb decision; takeoff: direction 

change if needed; taxi: just stop 

10    Explicit aural annunciation and display 

11    Stop! 

12    

13    If at controlled airport, issue vector, clearance etc.; if at 

uncontrolled field, short statement of bese evasive action 

14    

15    

16    Direction – stop – turn – heading? Altitude? 

Total 12 4 8 8 9 7 

15. Please provide any additional comments that will help us in our evaluation and development of the 

synthetic vision system concepts. 

The following EPs responded to this question: 

EP 1 Increase volume on the aircraft noise in cockpit. 

EP 4 Velocity vector display on map 

EP 9 Value of surface map may have increased if in a different location, closer to primary displays 

 in multifunction display area 

EP 13 Why not put traffic lights at hold lines 

EP 15 Location of display needs to be within pilots heads-up field of view; land-and-hold-short 

 planning for all cross-runway landings 
74

Appendix J: Paired Comparisons Questionnaire 

Overview: This questionnaire is designed to allow statistical analysis of your subjective assessment of 

situation awareness and runway incursion prevention for each of the following display configurations you 

evaluated today. Please look at the pictures of the display concepts when making your comparisons. 

Paired Comparison Rating Instructions: Each paired comparison will be listed on the left side of the 

questionnaire. You will be asked to make separate comparisons based on what you experienced during 

the flying tasks in the simulator. 

Situation Awareness: If situation awareness is not equal, indicate the magnitude of the difference by 

marking the appropriate box on the scale to the right of the comparison. The following definition of 

situation awareness should be used for reference: 

The pilot’s awareness and understanding of all factors that will contribute to the safe flying of 

their aircraft under normal and non-normal conditions. 

Runway Incursion Prevention Awareness: If runway incursion awareness is not equal, indicate the 

magnitude of the difference by marking the appropriate box on the scale to the right of the comparison. 

The following definition should be used for reference: 

The pilot’s awareness and understanding of the dynamic environment and degree to which he or 

she is aware of, and can successfully avoid, potential incursions with other aircraft or vehicles 

in the airport environment. 

The following example shows how to make the comparisons. Do not take an excessive amount of time 

on each comparison; your first impression is usually best. However, please feel free to correct any 

comparisons. Also, the data will be checked for consistency; if the results are inconsistent, you may be 

asked to clarify your responses. 

Situation Awareness Comparisons If not equal, how much more or how much less? 

Barely Substantially 

Display Concept ‘X’ 

Provides (__ more)(__ equal)(__ less) SA than  

Display Concept ‘Y’ 
75

Situation Awareness Comparisons If not equal, how much more or how much less? 

Barely Substantially 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) SA than 

B + surface map with ownship only (BMO) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) SA than 

B + surface map with ownship + traffic (BMOT) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting only (BA) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting + map with ownship only 

(BAMO) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + map with 

ownship + traffic (BAMOT) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + perspective 

map (BRIPS) 

B + surface map with ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + surface map with ownship + traffic (BMOT) 

B + surface map with ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting only (BA) 

B + surface map with ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting + map with ownship only 

(BAMO) 

B + surface map with ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + map with 

ownship + traffic (BAMOT) 

B + surface map with ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting +perspective 

map (BRIPS) 
76

Situation Awareness Comparisons If not equal, how much more or how much less? 

Barely Substantially 

B + surface map with ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting only (BA) 

B + surface map with ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting + map with ownship only 

(BAMO) 

B + surface map with ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + map with 

ownship + traffic (BAMOT) 

B + surface map with ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + perspective 

map (BRIPS) 

B + audible alerting only (BA) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible alerting + map with ownship only 

(BAMO) 

B + audible alerting only (BA) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + map with 

ownship + traffic (BAMOT) 

B + audible alerting only (BA) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting +perspective 

map (BRIPS) 

B + audible alerting + map with ownship only 

(BAMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting + map with 

ownship + traffic (BAMOT) 

B + audible alerting + map with ownship only 

(BAMO) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting +perspective 

map (BRIPS) 

B + audible alerting + map with ownship + 

traffic (BAMOT) 

Provides (__ more)(__ equal)(__ less) SA than 

B + audible & graphical alerting +perspective 

map (BRIPS) 
77

Runway Incursion Prevention If not equal, how much more or how much less? 

Barely Substantially 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + surface map + ownship only (BMO) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + surface map + ownship + traffic (BMOT) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible alerting only (BA) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible alerting + map + ownship (BAMO) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + map 

+ ownship + traffic (BAMOT) 

Baseline round dials only (B) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + perspective 

map (BRIPS) 

B + surface map + ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + surface map + ownship + traffic (BMOT) 

B + surface map + ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible alerting only (BA) 

B + surface map + ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible alerting + map + ownship (BAMO) 

B + surface map + ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + map + 

ownship + traffic (BAMOT) 

B + surface map with ownship only (BMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 
78

B + audible & graphical alerting +perspective 

map (BRIPS) 

Runway Incursion Prevention If not equal, how much more or how much less? 

Barely Substantially 

B + surface map + ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible alerting only (BA) 

B + surface map + ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible alerting + map + ownship (BAMO) 

B + surface map + ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + map with 

ownship + traffic (BAMOT) 

B + surface map + ownship + traffic (BMOT) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + perspective 

map (BRIPS) 

B + audible alerting only (BA) 

Provides (__ more)(__ equal)(__ less) potential for RI 

prevention than 

 

B + audible alerting + map + ownship (BAMO) 

B + audible alerting only (BA) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + map + 

ownship + traffic (BAMOT) 

B + audible alerting only (BA) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting +perspective 

map (BRIPS) 

B + audible alerting + map + ownship (BAMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting + map + 

ownship + traffic (BAMOT) 

B + audible alerting + map + ownship only 

(BAMO) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting +perspective 

map (BRIPS) 
79

B + audible alerting + map + ownship + traffic 

(BAMOT) 

Provides (__ more)(__ equal)(__ less) potential for 

RI prevention than 

 

B + audible & graphical alerting +perspective 

map (BRIPS) 
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Publication

 4. TITLE AND SUBTITLE

Simulator Evaluation of Runway Incursion Prevention Technology for 

General Aviation Operations

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Jones, Denise R.; Prinzel, Lawrence J., III

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

L-19917

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 03

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

A Runway Incursion Prevention System (RIPS) has been designed under previous research to enhance airport surface 

operations situation awareness and provide cockpit alerts of potential runway conflict, during transport aircraft category 

operations, in order to prevent runway incidents while also improving operations capability. This study investigated an 

adaptation of RIPS for low-end general aviation operations using a fixed-based simulator at the National Aeronautics and 

Space Administration (NASA) Langley Research Center (LaRC). The purpose of the study was to evaluate modified RIPS 

aircraft-based incursion detection algorithms and associated alerting and airport surface display concepts for low-end general 

aviation operations. This paper gives an overview of the system, simulation study, and test results.

15. SUBJECT TERMS

Runway safety; Runway incursions; Incursion detection algorithms; General aviation

18. NUMBER

 OF 

 PAGES

86

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

609866.02.07.07.02

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/TP-2011-217045

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 01 - 2011