April 2009 

NASA/TP-2009-215707 

Evaluation of Fused Synthetic and Enhanced 

Vision Display Concepts for Low-Visibility 

Approach and Landing 

Randall E. Bailey, Lynda J. Kramer, Lawrence J. Prinzel III, and Susan J. Wilz 

Langley Research Center, Hampton, Virginia 
NASA STI Program . . . in Profile

 Since its founding, NASA has been dedicated to 

the advancement of aeronautics and space science. 

The NASA scientific and technical information (STI) 

program plays a key part in helping NASA maintain 

this important role. 

 The NASA STI program operates under the 

auspices of the Agency Chief Information Officer. It 

collects, organizes, provides for archiving, and 

disseminates NASA’s STI. The NASA STI program 

provides access to the NASA Aeronautics and Space 

Database and its public interface, the NASA Technical 

Report Server, thus providing one of the largest 

collections of aeronautical and space science STI in 

the world. Results are published in both non-NASA 

channels and by NASA in the NASA STI Report 

Series, which includes the following report types: 

• TECHNICAL PUBLICATION. Reports of 

completed research or a major significant phase 

of research that present the results of NASA 

programs and include extensive data or 

theoretical analysis. Includes compilations of 

significant scientific and technical data and 

information deemed to be of continuing 

reference value. NASA counterpart of peerreviewed formal professional papers, but having 

less stringent limitations on manuscript length 

and extent of graphic presentations. 

• TECHNICAL MEMORANDUM. Scientific 

and technical findings that are preliminary or of 

specialized interest, e.g., quick release reports, 

working papers, and bibliographies that contain 

minimal annotation. Does not contain extensive 

analysis. 

• CONTRACTOR REPORT. Scientific and 

technical findings by NASA-sponsored 

contractors and grantees. 

• CONFERENCE PUBLICATION. Collected 

papers from scientific and technical 

conferences, symposia, seminars, or other 

meetings sponsored or co-sponsored by NASA. 

• SPECIAL PUBLICATION. Scientific, 

technical, or historical information from NASA 

programs, projects, and missions, often 

concerned with subjects having substantial 

public interest. 

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign scientific and 

technical material pertinent to NASA’s mission. 

 Specialized services also include creating custom 

thesauri, building customized databases, and 

organizing and publishing research results. 

 For more information about the NASA STI 

program, see the following: 

• Access the NASA STI program home page at 

http://www.sti.nasa.gov

• E-mail your question via the Internet to 

help@sti.nasa.gov

• Fax your question to the NASA STI Help Desk 

at 443-757-5803 

• Phone the NASA STI Help Desk at 

443-757-5802 

• Write to: 

 NASA STI Help Desk 

 NASA Center for AeroSpace Information 

 7115 Standard Drive 

 Hanover, MD 21076-1320
National Aeronautics and 

Space Administration 

Langley Research Center 

Hampton, Virginia 23681-2199 

 

April 2009 

NASA/TP-2009-215707 

Evaluation of Fused Synthetic and Enhanced 

Vision Display Concepts for Low-Visibility 

Approach and Landing 

Randall E. Bailey, Lynda J. Kramer, Lawrence J. Prinzel III, and Susan J. Wilz 

Langley Research Center, Hampton, Virginia 
Available from: 

NASA Center for AeroSpace Information 

7115 Standard Drive 

Hanover, MD 21076-1320 

443-757-5802 
...............................3

...............................4

...............................5

6

2 ...............................7

...............................8

...............................9

.............................11

.............................12

.............................13

.............................13

.............................15

.............................15

.............................15

.............................16

.............................16

.............................16

.............................17

.............................17

.............................17

.............................17

.............................17

.............................18

.............................19

.............................19

.............................19

.............................21

.............................24

.............................24

.............................25

.............................26

.............................26

.............................27

5 .............................29

.............................29

.............................29

.............................30

.............................30

.............................30

.............................31

.............................32

5.2.1. AFFTC Workload Estimate – PF HUD.....................................................................................................33

5.2.2 AFFTC Workload Estimate – PNF-AD......................................................................................................33

5.2.3. SWORD.....................................................................................................................................................33

5.2.4 Workload Discussion..................................................................................................................................34

5.3 Situation Awareness..........................................................................................................................................35

5.3.1 SART – PF HUD........................................................................................................................................35

5.3.2. SART – PNF-AD.......................................................................................................................................35

5.3.3. SA-SWORD ..............................................................................................................................................35

5.3.4. Situation Awareness Discussion................................................................................................................36

5.4 Pilot Display Preferences...................................................................................................................................37

Table of Contents 

List of Figures.................................................................................................................................

List of Tables..................................................................................................................................

Symbols and Abbreviations............................................................................................................

1. Introduction ...............................................................................................................................................................

. Background.................................................................................................................................

2.1 Integrated and Fused Synthetic and Enhanced Vision Systems Concepts............................

2.2 Previous Research.................................................................................................................

2.2.1 Pilot-Flying Role ...........................................................................................................

2.2.2 Pilot-Monitoring Role....................................................................................................

2.3 Current Research ..................................................................................................................

2.4 Experiment Objectives..........................................................................................................

3. Methodology...............................................................................................................................

3.1 Experiment Method ..............................................................................................................

3.2 Simulator ..............................................................................................................................

3.3 Head-Up Display ..................................................................................................................

3.4 Auxiliary Display .................................................................................................................

3.5 Head-Down Displays............................................................................................................

3.6 Synthetic Vision Database....................................................................................................

3.7 Out-the-Window (OTW) Scene............................................................................................

3.8 Enhanced Vision System......................................................................................................

3.9 Symbology............................................................................................................................

3.9.1 HUD Symbology ...........................................................................................................

3.9.2 HUD Tunnel ..................................................................................................................

3.9.3 Auxiliary Display Symbology .......................................................................................

3.10 Display Concepts................................................................................................................

3.10.1 Head-Up Display Concepts .........................................................................................

3.10.2 Auxiliary Display Concepts.........................................................................................

4. Procedures ..................................................................................................................................

4.1 Evaluation Task ....................................................................................................................

4.2 EVS Crew Procedures ..........................................................................................................

4.3 Experiment Matrix................................................................................................................

4.4 Measures...............................................................................................................................

4.5 Procedure..............................................................................................................................

. Results........................................................................................................................................

5.1 Path Control Performance.....................................................................................................

5.1.1. Approach Lateral Path Error.........................................................................................

5.1.2. Approach Vertical Path Error .......................................................................................

5.1.3 Vertical and Lateral Path Error on Final........................................................................

5.1.4 FTE on Approach ..........................................................................................................

5.1.5. Path Control Results Discussion...................................................................................

5.2 Mental Workload..................................................................................................................
2

.............................37

.............................40

.............................40

.............................43

.............................43

.............................45

.............................46

.............................50

.............................51

53

7. References ...............................................................................................................................................................54

Appendix A: Head Tracker Data Analysis Details......................................................................................................58

Appendix B. Final Questionnaire Subject Ratings and Comments .............................................................................62

5.5 Head-Tracking Analysis.......................................................................................................

5.6 Subjective Assessments of Display Clutter ..........................................................................

5.6.1 HUD Concept Ratings...................................................................................................

5.6.2 PNF-AD concept Ratings ..............................................................................................

5.7 Fusion Controls.....................................................................................................................

5.8 Non-Normals........................................................................................................................

5.8.1 Runway Incursions ........................................................................................................

5.8.2 Navigation Error............................................................................................................

5.8.3. Illegal Landings............................................................................................................

6. Conclusions .............................................................................................................................................................
3

List of Figures 

...............................8

.............................16

.............................18

.............................18

.............................21

.............................23

.............................24

.............................27

.............................29

.............................30

.............................36

.............................39

.............................41

cept. ....................42

Figure 15. Fusion control settings by altitude.............................................................................................................44

Figure 16. SV/EV fusion and EV-Only setting by altitude.........................................................................................45

Figure 17. Runway incursions (baggage cart and firetruck) as seen at 60 ft AFL using various displays..................48

Figure 18. Lateral navigation offset examples............................................................................................................51

Figure 1. Synthetic vision and enhanced vision comparison. ........................................................

Figure 2. Integration Flight Deck simulation facility with HUD and AD. ....................................

Figure 3. HUD symbology with advanced pathway guidance.......................................................

Figure 4. HUD runway outline symbol and glideslope reference line...........................................

Figure 5. Head-Up Display (HUD) formats on approach at 700 feet AGL...................................

Figure 6. Four Auxiliary Display (AD) formats on final...............................................................

Figure 7. Sparks Visual arrival to RNO runway 16R. ...................................................................

Figure 8. Post-run display clutter questions...................................................................................

Figure 9. RMS lateral path error during approach segment for each HUD concept......................

Figure 10. RMS vertical path error during approach segment for each HUD concept..................

Figure 11. Means of SART ratings for HUD symbology and raster combinations. ......................

Figure 12. Percentage of time where PNFs directed gaze. ............................................................

Figure 13. Mean ratings of demand on visual attention clutter metric for each HUD concept......

Figure 14. Mean ratings of supply of visual attention resources clutter metric for each HUD con
4

List of Tables 

...............................8

.............................25

.............................31

.............................38

.............................40

.............................46

.............................52

itude Segment.....58

.............................59

.............................59

60

Table A-5. Student-Newman-Keuls Test for Altitude Segment: 300 < AFL < 400 ft................................................60

Table A-6. Student-Newman-Keuls Test for Altitude Segment: 200 < AFL < 300 ft................................................60

Table A-7. Student-Newman-Keuls Test for Altitude Segment: 100 < AFL < 200 ft................................................61

Table 1. EV and SV Comparison ..................................................................................................

Table 2. EVS Crew Procedures.....................................................................................................

Table 3. Flight Path Tracking Performance Summary ..................................................................

Table 4. Percentage of Time PNF Directing Gaze at Various Head Locations.............................

Table 5. Percentage of Time PNF Directing Gaze, By Display Configuration .............................

Table 6. Above Field Level (AFL) Altitudes for Theoretical Detection of Runway Incursions ...

Table 7. Illegal Landing Trials ......................................................................................................

Table A-1. ANOVAs on Percentage Dwell Time at the Auxiliary Display Concepts for each Alt

Table A-2. Student-Newman-Keuls Test for Altitude Segment: > 1000 ft AFL...........................

Table A-3. Student-Newman-Keuls Test for Altitude Segment: 500 < AFL < 1000 ft.................

Table A-4. Student-Newman-Keuls Test for Altitude Segment: 400 < AFL < 500 ft................................................
5

Symbols and Abbreviations 

nter 

nt 

S ision System 

splay 

D 

Technologies 

nditions 

e 

ASA nautics and Space Administration 

ortation System 

ion 

nternational Airport 

SA Situation Awareness 

SART Situation Awareness Rating Technique 

SA-SWORD Situation Awareness – Subjective Workload Dominance 

SV Synthetic Vision 

SVS Synthetic Vision System 

SWORD Subjective Workload Dominance 

VFR Visual Flight Rules 

VMC Visual Meteorological Conditions 

AD Auxiliary Display 

AFFTC Air Force Flight Technical Ce

AFL Above Field Level 

AGL Above Ground Level 

ANOVA Analysis of Variance

CRM Crew Resource Manageme

DH Decision Height 

EFVS Enhanced Flight Vision System 

EP Evaluation Pilot 

EV Enhanced Vision 

EVO Equivalent Visual Operations 

EV Enhanced V

FAR Federal Aviation Regulation 

FLIR Forward Looking InfraRed 

FPM Flight Path Marker 

FTE Flight Technical Error 

HDD Head-Down Di

HU Head-Up Display

IFD Integration Flight Deck 

IIFDT Integrated Intelligent Flight Deck 

IMC Instrument Meteorological Co

LaRC Langley Research Center 

MDA Minimum Descent Altitud

MMWR Millimeter Wave Radar 

N National Aero

ND Navigation Display 

NextGen Next Generation Air Transp

nm nautical mile 

OTW Out The Window 

PF Pilot Flying 

PFD Primary Flight Display 

PNF Pilot Not Flying 

RI Runway Incurs

RMS Root-mean square 

RNO FAA airport identifier for Reno/Tahoe I

RNP Required Navigation Performance 
6

ABSTRACT 

ce technologies 

iers that would 

generation air 

s conducted to 

hanced Vision 

echniques for 

Vision and its 

 approach and 

 showed that 

ut concomitant 

rovided by the 

on technologies 

 in lateral path 

isplay concepts 

 vision or fusion 

ing non-normal 

navigational 

errors and runway incursions were neither improved nor adversely 

impacted by the display concepts. The addition of Enhanced Vision may 

ion 

date a projected 3-

ing developed to 

neration Air 

Transportation System (NextGen). One of the key capabilities envisioned to achieve these goals is the 

perational tempos 

independent of the 

able is to create a 

 and 

visibility conditions, through application of Enhanced Vision (EV) and Synthetic Vision (SV) 

technologies. 

An experiment was conducted to evaluate the complementary use of SV and EV technologies, 

specifically evaluating the utility, acceptability, and usability of integrated/fused enhanced and synthetic 

vision technologies and its effect on two-crew operations. This work begins the development of an allweather commercial aviation operations capability, approaching that which might create an EVO 

capability. 

NASA is developing revolutionary crew-vehicle interfa

that strive to proactively overcome aircraft safety barr

otherwise constrain the full realization of the next 

transportation system. A piloted simulation experiment wa

evaluate the complementary use of Synthetic and En

technologies. Specific focus was placed on new t

integration and/or fusion of Enhanced and Synthetic 

impact within a two-crew flight deck during low-visibility

landing operations. Overall, the experimental data

significant improvements in situation awareness, witho

increases in workload and display clutter, could be p

integration and/or fusion of synthetic and enhanced visi

for the pilot-flying and the pilot-not-flying. Improvements

control performance were realized when the Head-Up D

included a tunnel, independent of the imagery (enhanced

of enhanced and synthetic vision) presented with it. Dur

operations, the ability of the crew to handle substantial 

not, of itself, provide an improvement in runway incursion detect

without being specifically tailored for this application. 

1. Introduction 

The United States air transportation system is undergoing a transformation to accommo

fold increase in air operations by 2025.1

 Technological and systemic changes are be

significantly increase the capacity, safety, efficiency, and security for this Next Ge

concept of Equivalent Visual Operations (EVO), whereby Visual Flight Rules (VFR) o

and also, perhaps, operating procedures (such as separation assurance) are maintained 

actual weather conditions. One methodology by which the EVO-goal might be attain

virtual visual flight environment for the flight crew, independent of the actual outside weather
7

2. Background 

autics and Space 

 research effort to 

t-induced hazards. 

 that reduce the 

vercoming aircraft 

at would otherwise constrain the full realization of the NextGen. Part of this research 

g technologies to 

d from aircraft attitude, 

her required flight 

ns in the potential 

ologies.2-5 

ng sensor, such as 

ion §91.175 of the 

ucting straight-in 

 may now operate 

) when using an 

Up Display (HUD). (EFVS 

 Enhanced Vision 

 considered to be 

are and software 

ding.) This FAR 

or SV. 

ational benefits of 

methodologies by 

ificantly different 

; they are, in fact, 

arly advantageous 

tation of which an 

e vehicle external 

 Under conditions 

of smoke, haze, and night, a FLIR/EV provides orders-of-magnitude improvement over the pilot’s natural 

vision; greatly enhancing the pilot’s situation awareness and reducing the pilot’s workload. The 

comparison of SV and EV in Figure 1 on a night visual meteorological conditions (VMC) approach into 

an airfield highlights the similarities and differences in these two technologies. In the SV image, color 

coding is available, the features of the SV image are only dependent upon the database content and the 

engineering provided by the SV designer. No atmospheric or weather obscuration is present. Conversely, 

in the EV image, the image is critically dependent upon the EV sensor and the external environment. In 

this case, the FLIR provides an outstanding view on a night approach, where roads and runways are 

The Integrated Intelligent Flight Deck Technologies project, under the National Aeron

Administration (NASA) Aviation Safety Program, is comprised of a multi-disciplinary

develop flight deck technologies that mitigate operator-, automation-, and environmen

Towards this objective, crew/vehicle interface technologies are being developed

propensity for, and minimize the risks associated with, pilot error while proactively o

safety barriers th

effort involves the use of SV and EV systems and other interface modalities as enablin

meet these challenges. 

SV is a computer-generated image of the external scene topography, generate

high-precision navigation, and data of the terrain, obstacles, cultural features, and ot

information. SV provides significant improvements in terrain awareness and reductio

for Controlled-Flight-Into-Terrain incidents/accidents compared to current cockpit techn

EV is an electronic means to provide a display of the external scene by use of an imagi

a Forward-Looking InfraRed (FLIR) or millimeter wave radar (MMWR). In 2004, Sect

US Federal Aviation Regulations (FAR) was amended such that operators cond

instrument approach procedures (in other than Category II or Category III operations)

below the published Decision Height (DH) or Minimum Descent Altitude (MDA

approved Enhanced Flight Vision System (EFVS) shown on the pilot’s Headis the terminology adopted by the FAA to indicate the operational application of an

System (EVS). In the context of this report, the use of the terms EFVS and EVS are

equivalent. An EFVS (and EVS) pertains to the sensor, HUD, and associated hardw

which enable an enhancement to the pilot’s natural vision during an approach and lan

change provides “operational credit” for EV equipage. No such credit currently exists f

The intended use of EV and SV technology mirror each other as they both attempt to eliminate lowvisibility conditions as a causal factor to civil aircraft accidents and replicate the oper

clear-day flight operations, regardless of the actual outside visibility condition. The 

which this capability is achieved, however, are significantly different, leading to sign

operational considerations. While some may consider the technologies to be competing

complementary6

 as illustrated by the system comparison in Table 1. 

SV, by virtue of being weather-independent and unlimited in field-of-regard, is particul

during flight phases, such as approach, which may be obscured by clouds and precipi

EV sensor may not penetrate. On the other hand, EV provides a direct view of th

environment; independent of the derived aircraft navigation solution or of a database. 
8

ure such as clouds 

; in this case, in the far horizon of the EV image. The EV image is detail-rich and 

remise that7 “the 

c system and that 

stem can compensate for the deficiencies in the enhanced vision 

system.” While these goals are obvious, optimal methods and capabilities are not necessarily well 

defined. 

clearly demarked from vegetation because of thermal differences. Atmospheric moist

are clearly visible

“locked” to the aircraft. 

2.1 Integrated and Fused Synthetic and Enhanced Vision Systems Concepts

The complementary capabilities of SV and EV have been well-recognized6 with the p

strengths of (the) enhanced system can compensate for the deficiencies in the syntheti

the strengths of (the) synthetic sy

Enhanced Vision Synthetic Vision

Figure 1. Enhanced Vision And Synthetic Vision Comparison. 

d S

V) Synthetic Vision (SV) 

Table 1. EV an

Enhanced Vision (E

V Comparison 

Benefits For Te

& Situa

Awaren

rr

tion 

ain Significant Significant 

ess? 

Affected by Outs

Weather and 

i

Visibility Conditi

ndent up

vironme

conditions. 

ity independent of external 

conditions. 

de 

on? 

Yes – 

> Image quality depe

characteristics and en

on sensor 

ntal 

No – 

> Image qual

Affected By Aircr

Navigation 

“Solution”? 

m

t 

itically depends on 

e information 

aft No – 

> System “bolted” to airfra

aligned to aircraft boresigh

e and 

Yes – 

> Image accuracy cr

position and attitud

Training Require

Image Interpretation 

and Understanding? 

> Sensor phenomenology must be 

understood as image content and 

presentation is not predetermined 

Minimal – 

> Image content and presentation 

method is predetermined by SV 

designer. 

d for Yes – 

Real-Time Obstacle 

Detection? 

Yes – 

> Real-time sensor view from aircraft; but 

image quality depends upon sensor and 

target characteristics and environmental 

conditions 

No – 

> Image drawn from stored database of 

the terrain, obstacles, cultural features; 

unless augmented by real-time detection 

sensors. 

Operational Credit? Yes, under FAR Part 91.175. No 
9

2.2 Previous Research 

provides the direct 

ind the glass” for 

 detection. Image 

ight crew. This 

ous to the actual 

igation errors are 

 almost “perfect” 

 the study conclusion from Parrish8

 – SV “concepts 

should not be implemented without incorporating image-processing decision aides for the pilot” – 

 database integrity 

While degrees of success in developing these decision aids have been met, technology for “perfect” object 

y always be gaps, 

 may still warrant 

deck procedures and human interventions for integrity and error checks. 

 formats depends upon the 

volving a balance 

ation, the 

ture.18 

ual scanning) and 

e effort necessary to generate understanding) workload for 

V information, but the “costs” 

associated with transitioning to and from this visual environment and their effect during non-normal flight 

the transition from 

t of display clutter 

al information, or 

. 

data presented to the flight crew. For the HUD, in particular, FAA certification policy16 formalizes this 

conundrum in that “clutter should be minimized” yet, “essential or critical (HUD) data must always be 

displayed” such as that necessary for EFVS operational credit (as per Chapter 14 of the Code of Federal 

Regulations, Part 91.175(l)). Thus, task-critical or essential EV/SV imagery must be displayed but the 

information must not excessively interfere with pilots’ ability to see the actual runway environment 

through the display or impede their perception, understanding, and use of the information. Declutter 

control in present EFVS implementations allow the pilots to selectively add or remove symbology or 

raster (i.e., SV or EV) imagery, but declutter controls also introduce pilot workload during task- or timeSeveral studies8-10 have shown that the optimal combination of SV and EV technology 

display of SV to the flight crew without direct display of EV, but instead, using EV “beh

navigation error detection, database integrity monitoring, and real-time obstacle/object

processing performs these functions automatically without intervention by the fl

arrangement provides a highly usable display presentation (i.e., SV) that is impervi

weather and visibility conditions, yet if un-charted obstacles, database errors or nav

detected by the EV running in the background, the situation is annunciated, and

decision-making by the pilot occurs.8-10 In fact,

launched a 5 year effort at NASA and elsewhere developing enabling technologies for

monitoring and object detection (i.e., SV Systems (SVS)).11-15 

detection and database/navigation error detection does not yet exist. Further, there ma

such as minimal radar cross-section objects or below-detection threshold errors, which

flight 

Research suggests that the optimal configuration of displays, controls, and

intended function of the system, the flight phase, and the role of the pilots16,17 in

between: 

• Understanding – how the design promotes the perception of the SV and EV inform

comprehension of its meaning, and the projection of their status into the near fu

• Workload – how the design minimizes the physical (e.g., use of controls, vis

mental (e.g., amount of cognitiv

effective use and understanding of the information. 

These factors must not only include the “steady-state” use of EV and S

situations. For example, consideration must include how the design eases or impedes 

instrument to visual flight references and vice versa. 

A key interconnection between the factors of understanding and workload is the concep

– i.e., an excessive number and/or variety of color and symbols, that obscures essenti

presents distracting, disorganized and unnecessary information delaying visual detection

Clutter warrants special consideration for EV/SV as the designer is intentionally increasing the volume of 
10

critical situations and raise the possibility that critical information might be inappropriately removed. 

ngle display with 

his case is defined 

iable in the image. 

n integrating the 

 take advantage of 

egard EV sensor. 

rces for improved 

de to prevent an 

les modulation of 

magery. This 

ple pixel-by-pixel 

EV). As a simple 

A common method of combining SV/EV information is by use of an integrated si

simultaneous presentation of unadulterated SV and EV content. Image integration in t

as the use of a single display with the separate information sources being readily identif

By visual proximity, this method minimizes the visual scan and cognitive effort i

disparate information. Typically, “inset” displays of EV information have been used to

the “unlimited” field-of-regard in SV information to complement a limited field-of-r

Integration in this manner allows the user to retain comparison of the separate sou

understanding. However, careful design of the integration process must be ma

unintended loss of understanding. Often, a control for the pilot is provided which enab

the amount of EV imagery shown in the inset image compared to the SV background i

functionality may be generated by many methods. One often-used method uses a sim

combination of the two information sources (a so-called “pixel-averaging” of SV and 

illustration, the gray scale level for a inset pixel could be determined as ij aInset ij EVaSV ij + − )1(

 value (Inset), 

=

where the ij subscript denotes the ith row and jth column of the display inset image gray scale

which is a function of the SV and EV image input gray scale values, and the paramet

value ranging from 0 to 1, based on the pilot-control input. This technique would ena

image to range between 0% SV and 100% EV to 0% SV and 100% EV). Researc

averaging method is problematic.10 Image degradation of the blended image was the pr

pure pixel-averaging technique may create a situation where a poor quality (low cont

obscures good synthetic data without adding any value to the image; converse

er, a, is a decimal 

ble the final inset 

h shows that this 

imary factor.17,19 A 

ent) sensor image 

ly, a good quality sensor 

ds are available to 

must always be 

considered. Further, pilot control of the process introduces the costs of workload for control and a high 

 one display also 

mation. 

 the same display 

ys. This process 

 integration. This 

under NASA flight test, wherein SV information was 

nted on the HUD. 

g runway. This 

provements being 

 Pilot Flying (PF) 

can create “a loss 

of situation awareness” and additional workload from visual scanning. Designs utilizing visual 

momentum may assist in the integration.19 

• Temporal separation – by displaying either SV or EV information on the same display, using 

automatic or manual selection for which source is displayed. An automatic transition has been 

tested and it felt “natural” to the pilot.20 Without retaining some elements of SV or EV display, 

however, visual momentum (i.e., the use of symbology or imagery to provide cognitive coupling 

which assists the user in navigating a transition from one format or context to another) and the 

complementary benefits of both EV and SV may be lost. 

image is obscured by uncorrelated synthetic data. More sophisticated blending metho

minimize this problem,17 but the potential to lose information in the combination process 

potential for miss-setting of the controls.10 Simultaneoulsy showing both images on

introduces clutter, degrading the effective conveyance and useage of the displayed infor

Display clutter can be mitigated by not displaying both EV and SV simultaneously on

using one of two methods: 

• Spatial separation – by locating SV and EV information on different displa

forces the pilot to look across displays and mentally perform the information

methodology has been demonstrated 

presented on a head-down primary flight display, and EV information was prese

The pilots transitioned to the HUD at a specific height above the landin

methodology was found to be acceptable although not necessarily without im

desired. Sufficient information must be on the HUD to restrict the need for the

to go “head-down” to acquire task critical information; otherwise, HUD usage 
11

 image processing 

ltant image. The 

 comparisons and 

ant display.17 The 

e more features in 

ntrast than that of 

image processing 

er image contrast 

individual source information. Unfortunately these fusion methods may also pass through some 

age sources is of 

g symbols, icons, 

and/or perspective shapes for conformal overlay with EV imagery. For instance, SV-derived symbology, 

ay outlines, with simulated EV imagery provided visual momentum yet didn’t create 

s of 1500 ft 

ewhere.21 

 declutter the 

formality but their 

g task is critical, 

on to be displayed 

tion to the piloting 

 but the information must not excessively interfere with pilots’ ability to see the actual environment. 

ypically penalizes 

lay of symbology 

romatic, variable 

ially on potential 

erent information 

ignificant factor in 

nce symbology6,34

V 

 More 

typically, terrain imagery has been regarded as supplemental information that complements guidance 

information or the EV/SV imagery provides required visual references allowing certain tasks or 

operations, such as EFVS.35 

In addition to influence on flight path control, the complementary use of EV/SV information for the PF to 

perform navigation integrity checks and obstacle/object detection has been evaluated in simulation and, in 

general, performance is relatively mediocre.8-10 In an evaluation of an integrated SV/EV Head-Down 

Display (HDD) concepts8

 (using an EV inset), four pilots (~10% of the non-normal trials) flew to 

Using a single display, image fusion may also reduce clutter. Fusion, in this context, is

where the separate SV and EV image sources are not readily identifiable in the resu

drawback to fusion is the potential loss of understanding by the user from source

contrasts. The specific fusion methodology significantly affects the utility of the result

use of pixel-averaging image integration of two sensors was shown to generally provid

the resultant image than either sensor alone but it also yielded lower overall image co

either individual input. In comparison, feature-level fusion methods – that is, 

techniques that are specifically designed to concentrate on areas which contain high

and/or greater spatial power - generally enabled higher contrast in the resultant image while retaining the 

undesirable noise content of the sensor inputs. In either case, registration of the two im

paramount importance for fusion. 

Another method of EV/SV fusion displays SV-derived information symbolically, usin

in the form of runw

any perceptual confusion even when flying with significant navigational (intentional) error

and 6000 ft inserted into the SV-derived guidance.22 Similar results have been found els

2.2.1 Pilot-Flying Role 

EV/SV integration on head-down displays permits the use of color to differentiate and

respective data attributes.23 In addition to color, head-down displays do not require con

limitations must be recognized.24 The head-down to head-up transition for the landin

however, and the literature shows a clear, consistent pilot preference for flight informati

head-up during approach and landing operations.6

 

EV imagery must be displayed on the HUD – if it is critical or essential flight informa

task –

Excessive information might also exacerbate cognitive tunneling25-27 and clutter t

performance.28 The HUD provides many other advantages due to its conformal disp

(i.e., scene-linked) and imagery.29-31 However, its display characteristics – monoch

lighting background, and limited luminance capabilities – introduce constraints. espec

options for clutter control.24,32-33 For instance, color cannot be used to differentiate diff

sources. 

The presence of EV/SV imagery (e.g., terrain) for the PF has been found to not be a s

flight path performance when compared to the overwhelming influence of flight guida

unless, however, the guidance information is obscured or hindered by the presence of SV and/or E

imagery.22 In this latter case, performance degrades in the presence of EV/SV information. 
12

 error in the EV 

ed the incursions 

ts would likely be 

d fusion methods employed but conclusive cause-and-effect has 

d beyond unaided 

ze this capability. 

 sensor resolution 

/object against the 

scure an object.16 

ily focused on the landing task, and the PF may not have 

resources for “unexpected” scenarios. Theunissen9

 found that if the PF had to 

perform navigation integrity checking using EV/SV information, the situation was rated as 

g (PNF) or pilotring and verifying 

tifying the visual 

ding area.37 Unfortunately, like the PF data, the use of EV/SV 

 runway incursion 

influenced by the 

ngs10 where most 

misalignment, runway incursions, and uncharted 

the runs with the 

ected. 

red this head-up 

e EV inset image, 

pt “missed” 100% 

% rate of missed 

showed a missed 

sert 

concept was attributed to the clutter of the image, the small EV image size, and confusion 

between the SV/EV images. 

• With EV-only shown on a HDD, all uncharted obstacles were successfully detected (but 20% of 

runs continued into hazardous conditions because the evaluation pilots (EPs) didn’t recognize the 

severity of the threat) yet 33% of the runs missed the database misalignment. A missed detection 

rate of 22% for runway incursions was found. This concept was well liked for its large image 

size and minimal display clutter, but disliked because of the workload in transitioning from headdown to head-up flight. 

touchdown on the synthetic runway image, essentially ignoring a lateral navigation

(actual) inset image. Similarly, when presented with runway incursions, 3 pilots miss

(~12.5% of the non-normal trials) when using this same display concept. These resul

influenced by the EV/SV integration an

not been identified. 

EV imagery has been touted as providing an element of traffic detection above an

natural vision. While this seems intuitive, several factors must be considered to reali

First, imaging sensors are not perfect. Detection and recognition is dependent upon the

and its sensitivity to the environment and “target”, the size and contrast of the obstacle

image background, and the display resolution.6,36-37 Second, if the PF display uses flight vector

information, the flight path marker symbol and guidance symbol, if provided, may ob

Finally, the pilot-flying’s attention is primar

sufficient attentional 

“unacceptable” and resulted in unacceptable workload. 

2.2.2 Pilot-Monitoring Role 

While the pilot-flying is primarily focused on flying the aircraft, the pilot-not-flyin

monitoring in a two-crew operation does have the primary responsibility for monito

flight path performance, cross-checking guidance and raw data indicators, and iden

runway environment and clearing the lan

information by the PNF to perform the navigation integrity, obstacle detection, and

detection results in relatively mediocre performance. The results, again, appear to be 

EV/SV integration and display methods employed, but during low-visibility auto-landi

included an “anomaly” consisting of SV database 

obstacle, pilot-monitoring evaluations showed that: 

• Using an EV-only HUD, 57% of the obstacles were undetected and 50% of 

database misalignment went undetected. 38% of the runway incursions went undet

(Despite this relatively poor performance, the pilots subjectively prefer

implementation overall, mirroring PF display preferences.) 

• A Primary Flight Display (PFD) concept, showing SV with a pilot-controllabl

exhibited the worst performance for anomaly detection. The PFD inset conce

of the uncharted obstacles - all leading to hazardous situations - and a 33

detections for the database misalignments. Detection of runway incursions 

detection rate of 33%. The poor performance and low-acceptance of the head-down EVS in
13

hown elsewhere9 but pilot 

cause of the additional pilot control for display management. 

ilot-monitoring roles 

upon crew 

inappropriate. 

dination, in a twoach operation was 

 because of issues 

 development of a 

e 

from flight demonstrations suggest that “a centrally-located display of EVS imagery” may facilitate CRM 

 extensive evaluation of PNF display requirements for SV/EV operations showed 

some promising findings, but the test was run with the full expectancy for failures/non-normals. Also, the 

xperiment Objectives 

e of SV and EV 

ility, and usability 

 business aircraft 

rations as defined 

tionary expansion 

in) whose primary 

t path control, recognition and identification of the EV “visual references” 

under §91.175(l), and recognition and identification of the “visual references” under §91.175(l)(4), 

control of the 

rol at a time when 

“monitored approach” where the Captain takes on the role of pilot-monitoring and the First Officer takes 

r commands a gocrew experiment, the interaction of these factors on crew-coordination and decision-making were also 

examined. The experiment was developed based on the past research cited above and as required by the 

assumed EFVS operation: 

1. Investigate the effect of SV information on EFVS operations. Unlike these previous works, this 

experiment examined two aspects of adding SV to EV. First, a Required Navigation Performance 

(RNP)-type approach was flown, using a curving, descending arrival into a terrain-challenged 

airport. Since weather and other atmospheric conditions might negate a useful EV image during 

A preference for control of the EV sensor image in an inset display has been s

workload was shown to increase be

2.3 Current Research 

Some trends for optimal presentation of SV/EV information in the pilot-flying or p

may be inferred from past research cited above. However, to conclusively infer effectiveness 

resource management (CRM) or optimal display configurations might be misleading or 

In particular, the effects of EV/SV usage on crew resource management and crew coor

place cockpit, has not been studied extensively. Crew coordination in an EFVS appro

the primary focus of one study,38 but the results of this work may lack generalizability

uniquely associated with their use of millimeter wave radar as an EV sensor, such as the

perspective image from radar data and the lack of vertical (height) EV information. Anecdotal evidenc

and training.39 The most

test was not a “crew” experiment.31 

2.4 E

The experiment reported herein was conducted to evaluate the complementary us

technologies. The overall objective of this experiment was to test the utility, acceptab

of integrated/fused EV and SV technology concepts in a two-crew commercial or

cockpit. 

The present study focused on the complementary use of SV and EV during EFVS ope

under 14 CFR §91.175 since this operational credit provides a logical path for evolu

toward EVO. Under these rules, EV is presented head-up to the pilot-flying (Capta

responsibilities include fligh

without reliance on the enhanced flight vision system. In this operation, the PF remains in 

aircraft through the approach and landing. This procedure precludes a transfer of cont

such a transfer could interfere with safe landing of the aircraft. (This operation contrasts with a

on the role as pilot flying. Prior to or at minimums, the Captain (left seat pilot) eithe

around or takes control and lands.) 

Five specific experimental objectives, presented in the following, were addressed. By conducting a two-
14

 works previously; 

quires that SV not 

 PF. SV imagery 

e before that (as 

d. This transition 

et “windows” was 

rs from clutter and 

l of the inset imagery.8,10 The automation methodology was 

l for human-error 

ions. Unless the 

mance is expected 

ment 

transitioned to a EVFS operation from a curving, descending arrival into a terrain-challenged 

y information was 

scussed in Section 

3. Investigate the influence of scene-linked SV information. Since HUD clutter is a significant 

 or imagery was 

 investigated the 

 its influence on 

or data during an 

 the PNF has the 

e, cross-checking 

 environment and clearing the 

y-over, with some 

 as a PNF, showed 

ance using an EV 

tigate the presence 

e part of the flight 

5. Investigate the influence of adding SV information to a PNF display of EV sensor data during and 

EFVS operation. In a single pilot study, with the pilot serving as a PNF, relatively poor detection 

of database alignment errors or runway incursions occurred with EV-only HDD or an integrated 

EV/SV display using a pilot-controllable EV inset.10 This work contrasts this study, investigating 

a feature-level fusion of SV/EV which allows the PNF to tailor the presentation. Further, this 

work is being run the expectancy on the part of the flight crews for failures and non-normals. 

Also, the test is a true “crewed” experiment. 

this flight phase, SV may provide potential benefit as has been shown in other

however, in this case, the operation transitions to a EVFS operation, which re

degrade the EV imagery. As such, EV imagery was shown on the HUD for the

was, therefore, not shown near decision height and for some period of tim

discussed in Section 3) to ensure that the required EV imagery was unaffecte

from SV to EV imagery was automated. Integration of SV with EV using ins

not used, although it has been found to be viable,5,7 because performance suffe

deficiencies related to manual contro

designed to minimize display clutter and reduce PF workload and the potentia

compared to a manually controlled operation. 

2. Investigate the influence of Pathway Guidance information on EFVS operat

guidance information is cluttered by SV and/or EV imagery,5

 superior perfor

from the addition of pathway guidance. However, unlike these previous works, this experi

airport. As such, EV imagery was shown on the HUD for the PF and pathwa

decluttered near decision height and for some period of time before that (as di

3). 

concern, particularly for an EFVS operation, minimal additional symbology

added to the HUD, especially near decision height and landing. This work

presence or absence of retaining some elements of SV for an EV display and

visual momentum. 

4. Investigate the influence of adding symbology to a PNF display of EV sens

EFVS operation. While the PF is primarily focused on flying the aircraft,

primary responsible for monitoring and verifying flight path performanc

guidance and raw data indicators, and identifying visual runway

landing area (FAA, 2003). Many of the design principles noted above carr

significant differences. In particular, a single pilot study, with the pilot serving

significantly better object detection and runway incursion detection perform

display that did not include symbology.10 This work did not, however, inves

and absence of symbology. Further, this work is being run the expectancy on th

crews for failures and non-normals. Also, the test is a true “crewed” experiment. 
15

3. Methodology 

g synthetic vision 

ring low-visibility 

f adding synthetic 

 to the PNF’s display of the EV sensor data. A two-crew experiment 

ation during lowated in the experiment. 

re currently flying. 

mght affect crew 

t affliation. 

vious experience flying with HUDs. The subjects had an average of 1787 hours of 

nd military flying 

ere familiar with 

bjects were currently 

flying EV-equipped aircraft. 

F equipment for a 

um exposure and 

experience to the pilots in all operations of the equipment, including PF- and PNF-unique operations. 

ne pilot started in 

and was trained as the PF. The other pilot, seated in the right seat, was 

sly trained as the PNF. At the completion of their training in these roles, they exchanged 

e simultaneously trained in the opposite roles. 

the first and third 

nd fourth session, 

riment was conducted in the Integration Flight Deck (IFD) simulation facility (see Fig. 2) at 

NASA Langley Research Center (LaRC). The IFD emulates a Boeing B-757-200 aircraft and provides 

researchers with a full-mission simulator capability. The collimated out-the-window (OTW) scene is 

produced by an Evans and Sutherland ESIG 4530 graphics system providing approximately 200 degrees 

horizontal by 40 degrees vertical field-of-view at 26 pixels per degree resolution. 

The participants occupied the left (as PF) and right (as PNF) seats. The left seat included an overhead 

HUD projection unit and the right seat included an auxiliary display (AD) under the right side window 

(see Fig. 2). 

3.1 Experiment Method 

A fixed-based simulation experiment was conducted to evaluate the effect of addin

information and advanced pathway guidance to an enhanced vision HUD for the PF du

approach and landing operations. In addition, the experiment evaluated the effect o

vision information and symbology

was conducted to assess the interaction of these display concepts on crew coordin

visibility approach and landing operations. 

Twenty-four pilots, representing seven airlines and a major cargo carrier, particip

An attempt was made to pair the pilots based on which airline or cargo carrier than we

This procedure minimized any standard operating procedure differences which 

coordination durng the experiment. All but one crew were successfully paired by curren

All participants had pre

HUD flying experience and an average of 13.8 years and 16.2 years of commercial a

experience, respectively. EV experience was not required although some pilots w

imaging sensor technology from prior military flight experience. None of the su

Because of the unique nature of the test and the dicotomous nature of the PF and PN

single HUD-equipped aircraft, the training and testing was designed to provide maxim

The crews were trained and flew together as subjects in both the PF and PNF roles. O

the left seat, as Captain, 

simultaneou

seats and wer

Data collection was broken into four equal sessions. The first pilot flew as PF in 

session while the second pilot flew as PNF. The second pilot flew as PF in the second a

while the first pilot flew as PNF. 

3.2 Simulator 

The xpe e
16

3.3 Head-Up Display 

HUD presentation 

onsisted of a video 

 in Section 

 the scene imagery 

lot had a declutter 

 of the PF yoke. The button cycled through 

three “declutter” states: 1) No declutter (full symbology and scene imagery); 2) “Raster” declutter (full 

symbology, no scene imagery); and 3) “Full declutter” (no HUD display). 

. 

The PNF-Auxiliary Display (PNF-AD) was located outboard of the PNF location. The display was 

y/instrument panel 

color with 1024 x 

gy and computern Section 3.10.2). 

3.5 Head-Down Displays 

Minimal changes were made to the PFD and Navigation Display (ND) for the experiment so they closely 

resembled current B-757 equipage. The PFD was only modified to include a Flight Path Marker (FPM) 

and guidance cue. The PFD FPM and guidance cue were driven by algorithms identical to the HUD. 

Standard B-757 ship’s flight director needles were disabled. No changes to the ND were made. The ND 

showed the Flight Management System approach but did not include Enhanced Ground Proximity 

Warning System “peak’s mode” nor Traffic Alert and Collision Avoidance System information. 

The HUD subtended approximately 32o

 horizontal by 24o vertical field of view. The 

was written strictly in a raster format from a video source (RS-343) input. The input c

mix of symbology and computer-generated scene imagery (either EV or SV as described

3.10.1). The symbology included “haloing” to ensure high-contrast symbology against

background. Brightness and contrast controls were provided to the pilot. Also, the pi

control, implemented as a push-button on the left hand horn

Figure 2. Integration Flight Deck Simulation Facility With HUD and AD

3.4 Auxiliary Display 

Pilot Flying 

Head-Up Display 

Pilot Not Flying 

Auxiliary Display 

positioned as a compromise between optimal PNF viewing position, minimal displa

obscuration, and moderate installation complexity. The 8.4” diagonal display was full768 pixel resolution. The display video source was a video mix of “haloed” symbolo

generated scene imagery (either EV or the output of a fused EV/SV signal as described i
17

3.6 Synthetic Vision Database 

y 53 nautical mile 

t identifer: RNO). 

 runway, taxiways, and terminal 

buildings. The DEM was draped with 1 meter/pixel satellite photographic imagery within a 16 x 21 nm 

airport and 4 meter/pixel imagery outside this inner region. 

the same source data as the SVS database but was rendered using different 

ted from the OTW 

he characteristics 

ite-hot” presentation. The time-of-day, time-of-year, 

rnal properties were held constant. Atmospheric properties (cloud layer, cloud height and 

thickness, fog, and visibility) were varied experimentally to modulate the visibility that the evaluation 

LIR and the OTW scene presentations (as described in Section 4.3). The EV imagery 

was provided in 640 horizontal by 480 vertical pixel resolution. 

always shown by 

nstrument landing 

r “raw data” indicators. When the dog-bone indicator was 

displaced one “dot” left or right of course, this corresponded to 87.5 ft vertical and 150 ft lateral linear 

LS indicators and 

ed path centerline, 

ad of ownship. Horizontal and vertical position of the ball corresponded to the track and 

flight path angles to fly to the center of the desired path. 

Depending upon the experimental condition, the following symbology elements were also used, as 

described in Section 3.10.1 A glideslope reference line was drawn (Fig. 4) using the RNO Runway 16R 

ILS descent angle of -3.1 degrees. Also, a runway outline symbol was drawn to conformally position the 

symbol based on the threshold coordinates of Runway 16R/34L based on the simulated aircraft navigation 

position. The symbol portrayed an 8000 ft long by 200 ft wide runway, consistent with certified Head-up 

Guidance Systems. 

An SV database was created from a 1 arc-sec Digital Elevation Model (DEM) of a 53 b

(nm) square area centered around the Reno-Tahoe International Airport (FAA airpor

The airport was represented by three-dimensional models of the

area centered around the 

3.7 Out-the-Window Scene 

The OTW imagery used 

graphics processes and computers. 

3.8 Enhanced Vision System 

A physics-based FLIR simulation (using Evans & Sutherland EPX SensorsTM) was crea

visual database by applying materials properties to each component of the database. T

of a short/mid-wave FLIR were simulated in a “wh

and other diu

pilots had in the F

3.9 Symbology 

3.9.1 HUD Symbology 

The HUD format is shown in Figure 3. Vertical and horizontal path deviation was 

linear path deviation indicators (i.e., “dog-bones”) simultaneously with the standard i

system (ILS) (angular) glideslope and localize

path error. The linear dog-bone indicator was not equivalent to the angular deviation I

the pilots were briefed and trained on the differences. 

The pitch-roll guidance cue (“ball”) used modified pursuit guidance40 along the desir

5.5 seconds ahe
18

3.9.2 HUD Tunnel 

nnel41 was flown 

h represented the 

g the desired path. 

 ft high (±87.5 ft vertical) path, 1 nm 

ahead of ownship position, along the desired path. One dot of vertical and lateral path error (“dogbone” 

deviation) corresponds to the vertical and lateral extent of the tunnel, respectively. 

As an experiment variable, advanced pathway guidance in the form of a “minimal” tu

(see Fig 3). The minimal tunnel concept consists of a series of “crow’s feet” whic

truncated corners of nominally-connected rectangles spaced at 0.2 nm increments alon

The tunnel portrayed a constant 300 ft wide (±150 ft lateral) by 175

Flight Path Marker

Crow’s Feet Tunnel

Corners

Path Deviation Indicators

“Ball” Guidance Cue

Figure 3. HUD Symbology With Advanced Pathway Guidance. 

Runway Outline Glideslope Reference

Line

Figure 4. HUD Runway Outline Symbol And Glideslope Reference Line. 

The minimal tunnel was used to minimize HUD clutter. Past studies2,28 have shown that sufficient path 

information is provided by the minimal tunnel concept – at a minimal cost of display clutter – when path 

deviation indicators, guidance symbology and the FPM are also provided. 
19

3.9.3 Auxiliary Display Symbology 

o aid the PNF in 

mbology included 

flight path marker, 

 scales; waterline; 

ot needed for the 

re included for experimental data recording.) Alternative symbology sets to 

e display were tried, but the pre-test usability results were not encouraging enough to 

move forward with the concepts. 

aluated by the evaluation crew (PF and PNF) while 

s were invariant. 

ested, differing from each other by: 1) two types of raster 

bac

wn: 

The auxiliary display symbology (when used) was a subset of the HUD symbology t

monitoring the approach without obscuring too much of the raster image. The sy

digital readout of indicated airspeed and altitude; zero pitch attitude line (horizon line); 

pitch/roll (ball) guidance cue; path deviation indicators; ILS deviation indicators and

radio altitude, and event marker enunciators. (The event marker enunciators were n

evaluation subjects, but we

clear the center of th

3.10 Display Concepts 

Four HUD concepts and four AD concepts were ev

flying approaches to Reno-Tahoe Airport, Runway 16R. The head-down display format

3.10.1 Head-Up Display Concepts 

Four PF-HUD display concepts were t

kground; and, 2) two types of symbology. 

Two HUD raster (background) formats were flo

1) EV-only (hereafter referred to as “FLIR”). 

The FLIR concept represented our “baseline” HUD. In this configuration, th

output was exclusively displayed whether useful imagery was being provided o

e simulated FLIR 

r not. 

2) Fused SV/EV (hereafter referred to as “Fusion”). 

The Fusion raster started out as unadulterated SV imagery, transitioning throu

presentation 

gh a fused SV/EV 

beginning at 600 feet above field level (AFL), and ending with an unadulterated 

FLIR imagery by 500 feet AFL. Between 600 feet and 500 feet AFL, the fusion levels changed 

100% EV, over a 

d of approximately 10 seconds. 

it now offered by 

e was chosen from a usability study prior to the test and flight 

experience2

 as the altitude: 

ated to the FLIR 

2) that provides SV imagery to assist in establishing stabilized approach parameters up to and 

beyond the minimum recommended Instrument Meterological Conditions (IMC) stabilized 

approach altitude of 1000 ft AFL; and, 

3) at or just after the recommended minimum VMC stabilized approach altitude to allow full 

utilization of EV. The 100 ft transition between SV and EV imparts visual momentum between 

the concepts for the PF HUD. 

in 10% step increments from 100% SV and 0%, EV ending at 0% SV and 

perio

Each raster concept showed FLIR alone below 500 ft to enable the operational cred

EFVS. The 500 ft transition altitud

1) after which FLIR would be required yet with sufficient time to become accilim

imagery, 
20

lity of SV and EV 

 maximize image 

the PF. In this 

tageous to the PF, 

ver-burdened with 

 concept and its 

 from SV to EV, was trained to the flight crew. The HUD included either 

 annunication in the lower left hand corner of the display for positive 

. 

Two

 HUD symbology (hereafter referred to as “Baseline”). 

The “fusion” concept provided the basis to evaluate the utility, acceptability, and usabi

on the HUD. This methodology was based on the literature review cited in Section 2 to

legability and limit pilot manual control requirements for an EFVS operation as 

methodology, SV and EV are shown in the flight regimes where they are most advan

and pilot-controllable fusion methods are not implemented as the PF is already o

controlling the HUD brightness, contrast, declutter, and FLIR control. The fusion

behavior, including its transition

an “SVS,” “EVS,” or “Fused”

indication of the Fusion mode to the PF

 HUD symbology sets were flown: 

1) Standard

xt specifying the raster format 

n underneath the 

t). 

The glideslope reference line was always drawn. In addition, te

being displayed on the HUD (“EVS”, “SVS”, or “FUSED”) was always draw

airspeed tape. 

2) Enhanced HUD symbology (hereafter referred to as the “Tunnel” symbology se

 outline. The 

“Tunnel” symbology set was tailored to transition at the same altitudes as the Fusion raster. The 

de, the last tunnel 

t was no longer visible below 500 ft), and, upon 

line was projected 

it is identified as a 

, Fusion-Baseline 

s, depending for instance, upon the 

atmospheric conditions. Conversely, the Fusion-Tunnel concept uses tunnel guidance for distinct path 

demarcation and SV for clear terrain and runway references, above 500 ft AFL. Below 500 ft with the 

Tunnel symbol set, the runway outline provides an element of SV visual momentum within the EV raster 

background image. The tunnel is decluttered at 500 ft AFL to minimize clutter and is replaced by the 

glideslope reference line to ensure approach path angle awareness. Below 500 ft, the only additional 

symbology in the Tunnel symbology set, over that of the Baseline set, is the runway outline symbol. 

Below 500 ft, the FLIR-Baseline and Fusion-Baseline configurations are identical. 

The baseline symbology set was enhanced with pathway guidance and a runway

tunnel was shown above the 500 ft above field level (AFL) transition altitu

segment was positioned at 500 ft AFL (thus, i

reaching 500 ft AFL, the glideslope reference line was drawn and a runway out

until reaching 50 ft AFL. 

The glideslope reference line is part of both the Baseline and Tunnel symbology sets as 

required symbology element under FAR §91.175 EFVS operations. 

In Figure 5, all four concepts are shown - the FLIR-Baseline HUD, FLIR-Tunnel HUD

HUD, and the Fusion-Tunnel HUD. In the FLIR-Baseline concept, a minimum of symbology is used and 

the FLIR does not necessarily provide terrain and runway cue
Figure 5. HUD Formats On Approach At 700 feet AFL 

Fou by: 1) two types of raster 

background presented; and, 2) two types of symbology presented. 

Two

3.10.2 Auxiliary Display Concepts 

r PNF-AD display concepts were tested, differing from each other

 AD raster (background) formats were flown: 

1. EV only (hereafter referred to as “FLIR”); 

The FLIR concept represented our “baseline” AD. In this configuration, th

output was exclusively displayed whether useful imagery was being provided o

2.

e simulated FLIR 

r not. 

Fused SV/EV (hereafter referred to as “Fused”). 

In this configuration, the AD raster imagery was pilot-controllable and could be tuned at any time 

to one of 10 states: EV-only, SV-only, or 8 fusion combinations of EV and SVS, using an 

Equinox EP-3000TM fusion board. The fusion board employs a feature-level extraction algorithm 

with two pilot control inputs. The first control biased the feature level fusion through 8 weighting 

values weighting EV or SV. (A value of 1 biased the extraction to 11% FLIR and 89% SV 

whereas a value of 8 weighted the extraction to 89% FLIR and 11% SV). The second control 

modulated the false-color coding of the fusion image through 1 of 8 values. A setting of 1 did not 

apply any color-coding (the display was a monochromatic fused image). A setting of 8 applied 

maximal green shading to the features which were assessed by the fusion algorithm to be 

21
22

atures between the two input videos and which had spatial frequency content above 

ined on the use of 

irect guidance as to how to use it. In this way, the experiment was 

designed to identify through pilot usage data, flight data, and pilot comments, a first-order cut at optimal 

r the PNF. 

o

bology 

“common” fe

a threshold value. 

As detailed later, the fusion controls were recorded and the evaluation pilots were tra

the fusion, but not given any d

fusion displays and controls fo

Tw AD symbology sets were flown: 

1) No Sym

us for clear raster 

 equip. 

The absence of symbology has previously been shown to be very advantageo

imagery viewing. The absence of symbology would also likely be less costly to

2) Symbology. 

The symbology was derived during usability testing prior to the experi

“decluttered” version of the PF HUD symbology (see Section 3.9). 

In Figure , the four PNF-AD concepts are 

ment based on a 

shown: FLIR-No Symbology AD (upper left), FLIRSymbology AD (lower left), Fused-No Symbology AD (upper right), and Fused-Symbology AD (lower 

right). (The terminology “Fused” was used when the pilot controlled the blending of SV and EV 

imagery, such as the case for the PNF-AD. Whereas, the term “Fusion” was used when the blending was 

automatically controlled, such as the case for the PF-HUD.) 

6
Without Symbology With Symbology

FLIR Raster Fusion (SVS) Raster

Figure 6. Four Auxiliary Display (AD) Formats at 200 ft AFL 

23
24

4. Procedures 

 NextGen concept 

val – reflecting an 

 VMC for the pilot 

 references. The 

on task tests 

the ability of SV and EV technology to support this type of operation by providing “equivalent visual” 

al for operational 

 Runway 16R (see 

as configured for 

s 30 degrees). The path converged into the ILS for 

Runway 16R. The aircraft was established on a stabilized striaight-in approach by 1000 ft AFL. The 

PNF monitored the approach from the right-hand side of the flight deck using standard instruments and 

the AD. Pilot participants were instructed that the run would end at main gear touchdown but that they 

should perform a go-around if either felt the landing was not safe. 

4.1 Evaluation Task 

The evaluation task was selected to approximate what may be typical of the emerging

called an “equivalent visual operation.” The task was based on a published visual arri

efficient and preferred routing for ATC and noise-abatement – which currently requires

to see-and-avoid terrain, traffic, and obstacles while navigating with respect to ground

approach path mimicked a RNP-type arrival, using a curved, descending path. The evaluati

information into the cockpit. Further, these technologies may offer the potenti

efficiency and minimums reduction above and beyond what can be provided by RNP. 

The PF hand-flew the base and final leg portions of the Sparks Visual Arrival to RNO

Fig. 7) with autothrottles engaged at an approach speed of 138 knots. The aircraft w

landing prior to each run (landing gear down and flap

Initial Condition

Navigation Display Format Approach Plate

Figure 7. Sparks Visual Arrival To RNO Runway 16R. 
25

4.2 EFVS Crew Procedures 

 operations, were 

n 

overview of these procedures is given in Table 2, including automatic call-outs. The altitude call-outs 

t where the PF 

 the approach. In 

trast were properly 

t). No EV controls, such as FLIR senstivity or 

 flight crew under 

ocedures dictate that the PF must have the required 

the descent. 

ch light system for 

es were visible, the PF was 

eference for RNO 

 no later than 100 ft AFL. 

ding back-up on all decision heights, and was instructed to call “go-

 ts called 00 ft DH or was not called by 100 ft 

DH. The PNF was allowed to assist the PF in picking up the required visual cues (normal or EFVS). 

of twee Cap as not p

Ta w Proced

Bas

ts 

 / 

Altitudes 

(ft) 

tom

Callou

PF 

ks / C

PNF 

ks / Callouts 

EFVS crew procedures, adapted from those currently used in business aircraft EFVS

established. Instructions in the use of the procedures (see Section 4.5) were given to each crew. A

were set up assuming a 200 ft DH for the published, non-EFVS approach. 

At 500 ft AFL, the “EVS Normal, System Normal” call by the PF corresponds to the poin

would nominally check that the FLIR was set-up properly and functioning properly on

our test, this call-out cued the PF to ensure that the HUD declutter, brightness and con

set (i.e., good raster image brightness and contras

black/white hot options, were available to the flight crew in the experiment. The EPs were instructed that 

the FLIR was always set to its optimal setting to provide the best image possible for the

the prevailing weather, environment, and atmospheric conditions. 

By the published minimums of 200 ft DH, the crew pr

EFVS references or the required landing visual references (using natural vision) to continue 

The landing references were those published in FAR §91.175. For this test, the approa

RNO 16R provided the prominent EFVS references. If these EFVS referenc

instructed to call “EVS Lights”. 

If the PF saw the lights or markings of the threshold (the predominant landing visual r

16R), the PF called “Landing.” The “landing” call was required

The PNF provided m

around” if

onitoring, inclu

“EVS Ligh ” was not at or before 2 if “Landing” 

Transfer control be n the tain and First-Officer w ermitted. 

ble 2. EFVS Cre ures 

AltitudeEvened AFL Baro- Au atic 

ts Tas allouts Tas

50 feet AFL 500 / 4912 “5 s

orm

0 00” Response: “Sy tems Normal, Call “500 feet” 

EVS N al” 

100 feet 

Minimu

Above 

ms 

300 / 4712 “Approach

Minimums” 

ing Response: “Check” Call “100 feet Above” 

Published 

Minimums 

(200 ft AFL) 

200 / 4612 “Minimums” With EVS Visual Cues, 

Call “EVS Lights” 

When Visual Cues Appear,

Call “Lights” or “Field in Sight” 

 Without EVS Visual Cues,

Call “Going Around” 

Without PF Call of ‘EVS Lights’, 

Call “Go Around” 

EFVS Decision 

Altitude 

(100 ft AFL) 

100 / 4512 When Actual Visual Cues, 

Call “Landing” 

When Visual Cues Appear, 

Call “Lights” or “Field in Sight” 

 Without Actual Visual Cues, 

Call “Going Around” 

Without PF Call of ‘Landing’, 

Call “Go Around” 
26

counter to, others 

er case, the crew 

ions and issues of 

SOPs were discussed. 

These results are captured in Appendix B. Flight crews from the same airline were paired to the greatest 

nimize SOP differences and to mitigate potential interference in crew interaction. 

Nominally, 40 experimental runs were completed by each evaluation crew with each pilot flying 20 

proaches, as PNF, 

TW varied from 1 

n to 1⁄2 mile. The required EFVS visual references became visible on the HUD between 450 ft 

erences were 

. These four runs, 

ted no lower than 

e were passengers 

on the PF’s ability 

ition, the PF was 

the centerline of the runway. 

 by measuring the 

normal runs were 

 each crew. The non-normals were runway incursion (RI) scenarios and database integrity 

s were designed to 

d an incursion with either a non-transponding baggage cart or fire truck. Both 

 cart and fire truck were stationary. Details of the runway incursion set-up are contained in 

n solution error (of 

 or 75 feet) with respect to the real runway. This error resulted in the synthetic vision terrain, 

pathway and guidance cue being misaligned from the FLIR and ILS data (which were defined in the flight 

crew briefing as being correct). 

4.4 Measures 

During each run, path error, pilot control inputs, PNF head-position and PNF-AD control inputs were 

recorded for analysis. In addition, an event marker was used by the EPs or the experimenter to log points 

of interest. 

The crew procedures were new to all of the flight crews. Some procedures were 

consistent with, their current airline Standard Operating Procedures (SOPs). In eith

procedures were trained and “enforced” for the test. During the post-test debrief, quest

how these procedures may or may not work within their airline operation and 

extent possible to mi

4.3 Experiment Matrix 

approaches, as PF, evaluating the HUD concepts and with each pilot monitoring 20 ap

while evaluating the AD concepts. 

The wind and weather varied on each run. The nominal visibility in the FLIR and O

mile dow

and 250 ft AFL. Four runs per flight crew were specifically designed so the EFVS ref

visible but the required runway (i.e., using normal un-aided vision) references were not

if properly flown using the EVS crew procedures, should conclude by a go-around initia

100 ft AFL. 

The PF was instructed to fly each approach as precisely as possible, albeit as if ther

aboard, using the display information available, as the effect of the display information 

to fly the approaches would be quantitatively and qualitatively evaluated. In add

instructed to land as close as possible to 

A significant component of the test, in addition to the nominal runs, was accomplished

ability of the flight crew to react and properly handle non-normal events. Four nonflown by

monitoring scenarios. The number and ordering of RI and database integrity scenario

avoid expectancy on the part of the flight crew.42 

The RI scenarios simulate

the baggage

Section 5.8. 

The database integrity monitoring scenarios purposefully introduced a lateral navigatio

either 50
27

t Technical Center 

echnique (SART),44 and 

 Fig. 8). 

tion Awareness – 

nce (SWORD)45

e, Fusion-Tunnel) 

gy, Fused-No 

i-structured interview to elicit 

sion concept, AD 

conducted for the 

ors, each with two 

 Tunnel). For the 

cepts, there were two main factors, each with two levels (as shown in Figure 6.): 1) Raster (FLIR, 

Fused) and 2) Symbology (On, Off). When the 2nd order interaction was significant, a simple main 

effects analysis was conducted using α=0.05. For the post-test paired comparisons, simple ANOVA and 

Student-Newman-Keuls (SNK) post-hoc multiple comparison tests with alpha (α) set at 0.05 were 

performed. 

After each run, pilots completed a run questionnaire consisting of the Air Force Fligh

(AFFTC) Revised Workload Estimation Scale,43 Situation Awareness Rat g T in

derstanding

four Likert-type (7-point) questions specific to different constructs of display clutter (see

After data collection was completed, pilots were administered two separate Situa

Subjective Workload Dominance (SA-SWORD)44 and Subjective Workload Domina

tests: one for HUD concept comparisons (FLIR-Baseline, FLIR-Tunnel, Fusion-Baselin

and another for AD concept comparisons (FLIR-No Symbology, FLIR-Symbolo

Symbology, Fused-Symbology). The pilots also participated in a sem

comments on the HUD/AD concepts, HUD SVS-to-EVS transition strategy for the fu

fusion strategy, and EVS crew procedures. These data are presented in Appendix B. 

For the post-run questions, separate analyses of variance (ANOVA) analyses were 

HUD concepts and the AD concepts. For the HUD concepts, there were two main fact

levels (as shown in Figure 5.): 1) Raster (FLIR, Fusion) and 2) Symbology (Baseline,

AD con

Rating #3 Un

What was your ability to quickly and accurately understand taskcritical display information?

Rating #4 

As I Performed the evaluation task, the level of display clutter on 

the HUD/Auxiliary Display was____.

Rating #2 Supply of Visual Attention Resources

How much spare visual attention and mental ability was 

available to accomplish secondary task(s)?

Rating #1 Demand on Visual Attention

How much visual search time and cognitive effort was required 

to scan and locate task-critical display information in the 

display?

7654321

Display Clutter Ratings Low....................................High

Rating #3 Understanding

What was your ability to quickly and accurately understand taskcritical display information?

Rating #2 Supply of Visual Attention Resources

Rating #4 

As I Performed the evaluation task, the level of display clutter on 

the HUD/Auxiliary Display was____.

How much spare visual attention and mental ability was 

available to accomplish secondary task(s)?

Rating #1 Demand on Visual Attention

How much visual search time and cognitive effort was required 

to scan and locate task-critical display information in the 

display?

7654321

Display Clutter Ratings Low....................................High

Figure 8. Post-Run Display Clutter Questions. 

4.5 Procedure 

The subjects were given a 1-hour briefing to explain the SV/EV concepts on the HUD and AD, EVS crew 

procedures, and the expected evaluation tasks. After the briefing, a 2-hour training session in the IFD was 

conducted to familiarize the subjects with the aircraft handling qualities, display symbologies, EVS crew 

procedures, and controls. The ‘rare-event’ scenarios were not discussed, although the pilot’s 
28

 collection lasted 

breaks were taken 

lso given a take-home final 

questionnaire. The entire session including lunch and breaks lasted approximately 9 hours. 

responsibility for maintaining safe operations at all times was stressed. Data

approximately 4.5 hours followed by a 30-minute semi structured interview. 10 minute 

approximately every hour during data collection. The pilots were a
29

5. Results 

 the measures for 

rs, each with two 

VA analyses were 

ch and final. The 

he final approach 

was significant, a 

hen appropriate, SNK post-hoc tests with α

excluded from these analyses due to simulation problems 

 display during those runs). (The non-normal runs with a lateral navigation 

nificant for RMS 

error when using a 

ft) compared to the baseline (mean=41 ft) symbology configuration. The other main 

factor, raster type (FLIR or Fusion), and the second order interaction between symbology and raster were 

not significant (p>0.05) for this measure. In Figure 9, post-hoc tests (SNK using α=0.05) showed two 

unique subsets for the approach lateral path error with the 4 HUD combinations: 1) FLIR-Baseline 

(mean=42 feet) and Fusion-Baseline (mean=40 feet) and 2) FLIR-Tunnel (mean=30 feet) and FusionTunnel (mean=27 feet). 

5.1 Path Control Performance 

Root-mean-square (RMS) calculations of lateral and vertical path error were used as

flight path control performance. For the HUD concepts, there were two main facto

levels: 1) Raster (FLIR, Fusion) and 2) Symbology (Baseline, Tunnel). Separate ANO

performed on RMS path error (lateral and vertical) for two segments of the run: approa

approach segment began at the task starting point and ended at 500 feet AFL. T

segment was between 500 feet and 100 feet AFL. When the 2nd order interaction 

simple main effects analysis was conducted using α=0.05. W

set at 0.05 were performed. Two runs were 

(e.g., lost path on navigation

error - 50 feet or 75 feet - were not included in the final segment analyses.) 

5.1.1. Approach Lateral Path Error 

An ANOVA revealed that HUD symbology (F(1,23)=12.60, p=0.002) was highly sig

lateral path error during the approach segment of the flight. Pilots had less lateral path 

tunnel (mean=28 

0

5

10

15

ach 20

25

30

35

40

45

FLIR-Baseline Fusion-Baseline FLIR-Tunnel Fusion-Tunnel

Appro RMS lateral path error, ft

Subset 1

Subset 2

Figure 9. RMS Lateral Path Error During Approach Segment For Each HUD Concept. 
30

5.1.2. Approach Vertical Path Error 

) and HUD raster 

ch segment of the 

ed to without one 

n=9 ft) compared 

and raster was not 

ee Fig. 10) revealed that the FLIR-Baseline 

(mean=12 feet) concept had significantly higher RMS vertical path error than the other three concepts: 

Fusion-Baseline=10 feet; FLIR-Tunnel=9 feet; & Fusion-Tunnel=9 feet. 

An ANOVA revealed that both main factors, HUD symbology (F(1,23)=5.67, p=0.026

(F(1,23)=4.70, p=0.041) were significant for RMS vertical path error during the approa

flight. Pilots had less vertical path error when using a tunnel (mean=9 ft) compar

(mean= 11 ft) and they had less vertical path error wh using the Fusion imagery (mea

to the FLIR imagery (mean=11 ft). The second order interaction between symbology 

significant (p>0.05) for this measure. Post-hoc tests (s

en

0

2

4

6

8

10

12

14

Approach RMS vertical path error, ft

Subset 1

Subset 2

FLIR-Baseline Fusion-Baseline FLIR-Tunnel Fusion-Tunnel

 Concept. 

RMS lateral path 

f the flight. Once 

sence or absence of the runway 

outline. This symbology element should be (and was found to be) inconsequential to flight performance. 

5.1.4 Flight Technical Error on Approach 

While the RMS measures for flight path control show statistically-significant differences on approach, the 

operational significance of these differences is questionable. To tease out the possible operational 

differences, the lateral and vertical tracking error data on approach was analyzed in terms of Flight 

Technical Error (FTE) as if it were a part of Required Navigation Performance (RNP). 

Figure 10. RMS Vertical Path Error During Approach Segment For Each HUD

5.1.3 Vertical and Lateral Path Error on Final 

As expected, there were no significant differences (p>0.05) for HUD concept for the 

error (mean=7 feet) or RMS vertical path error (mean=7 feet) during the final segment o

on final, the only difference between the configurations was the pre
31

ominal Sparks 16R 

nnel, and Fusedlculation46 include 

egligible, so FTE 

 components (path 

uld be equivalent 

ed that the other three components 

f the vertical RNP 

FTE analysis, all 

 of the time. No 

ally significant differences were found between display concepts. These results are consistent 

d HUD guidance 

ith lateral FTE of 

o that the required 

 was 300 feet.46 With these assumptions, all HUD concepts 

yielded a ver % of the time. In fact, the Fused - 

Tunnel, Fused-Baseline, and FLIR-Tunnel concepts yielded a vertical FTE navigational accuracy of 80 

 at least the tim IR- UD ield l FTE navigational 

accuracy of 120 feet at least 99.7% of the time. 

able 3. Fligh Trackin ormance ary for ch 

er Symbol Lateral 

mean 

Vert 

mean 

H

FTE 

Vertical FTE 

Lateral and vertical path error data were evaluated using a histogram analysis for the n

approach runs for the four HUD concepts (FLIR-Tunnel, FLIR-Baseline, Fused-Tu

Baseline). The horizontal and vertical path steering error components of the RNP ca

both FTE and display error. For this analysis, it was assumed that display error was n

was the only component of path steering error. It was also assumed that the other two

definition error and position estimation error) of the horizontal RNP calculation wo

across the display concepts evaluated. Similarly, it was also assum

(altimetry system error, vertical path definition error, and horizontal coupling error) o

performance calculation would be equivalent across the display concepts evaluated. 

A summary of the flight path tracking performance is shown in Table 3. Using the 

HUD concepts yielded a horizontal FTE navigational accuracy of 0.02 nmi at least 95%

statistic

with, and show even better performance than other NASA studies41,47 that showe

concepts, using flight path-centered symbology, can enable manual RNP operations w

0.05 nmi. 

For vertical FTE, it was assumed that the pilot was flying a specified vertical profile s

vertical navigation performance accuracy

tical FTE navigational accuracy of 300 feet at least 99.7

feet 99.7% of e. The FL Baseline H concept y ed a vertica

T t Path g Perf Summ Approa

Rast ogy rms ical rms orizontal 

Significance? 2 Unique Subsets 2 Unique Subsets None 2 Unique Subsets

FLIR

(N

 

o SV) 

aseline 

e Only) 42 ft* 1 120 ft* B(Guidanc 2 ft* 0.02 nm

Fusion Baseline 

(SV) (Guidance Only) 40 ft* 10 ft†

 0.02 nm 80 ft†

FLIR Tunnel 

(No SV) (Guid. + Tunnel) 30 ft†

 9 ft† 0.02 nm 80 ft†

Fusion 

(SV) 

Tunnel 

(Guid. + Tunnel) 27 ft†. 9 ft†

 0.02 nm 80 ft†

* Member of Subset 1 † Member of Subset 2 

5.1.5. Path Control Results Discussion 

As summarized in Table 3, the path control results show that the tunnel concepts have lower RMS lateral 

path error than the baseline, non-tunnel HUD concepts on the approach segment. The primary difference 

between the configurations was presence or absence of the tunnel and the turn anticipation cues that it 

provides. However, while statistically significant, the operational significance of the differences was not 
32

 an analysis of the 

 did not reveal any 

 nm FTE levels. 

t the very small 

ant vertical FTE 

s were found, where the baseline (FLIR-Baseline) configuration showed the worst performance. 

g little operational 

uidance concepts, 

 FTE of 0.05 nmi. 

 the same pursuit 

mbol, and 

path deviation indicators). However, it might be conjectured that the pilots with a Tunnel and/or SV 

al and lateral path 

 lateral tracking at 

 SVS background, 

improved flight path control performance because the database imagery in the 

nger roll reference visual cues. These features also limited a tendency to 

n the fixed-base B-757 simulator, particularly when compared to flying the baseline 

tly apparent in the 

ents, such as FTE 

for RNP, flying manually. 

er each experimental run, using the AFFTC workload estimate tool, and 

ool with significant operational crew acceptance. 

oint descriptors: 

4. Busy; Challenging But Manageable; Adequate Time Available 

5. Very Busy; Demanding To Manage; Barely Enough Time 

6. Extremely Busy; Very Difficult; Non-Essential Tasks Postponed 

7. Overloaded; System Unmanageable; Important Tasks Undone. 

The SWORD technique uses a paired-comparison of the experimental conditions to quantify their relative 

pilot workload. The SWORD is administered at the conclusion of the test; thus, reflecting the entire 

duration and experiences of the experiment and the experimental conditions. 

found. This conclusion is drawn based on the absence of significant differences using

FTE - a component of Required Navigation Performance. In this analysis, the data

statistically-significant differences. All HUD concepts could be effectively used to 0.02

Similarly, the approach vertical path error also showed statistical significance, bu

differences imply little operational significance. In this case, statistically-signific

difference

However, the FTE difference (see Table 3) was only 40 ft (80 ft vs. 120 ft), implyin

significance. 

The pilot performance results are consistent with past research41,47 that showed HUD g

using flight path-centered symbology, can enable manual RNP operations with lateral

Minimal performance differences were expected since each display concept utilized

guidance control laws and symbology (i.e., the flight path marker, integrated cue guidance sy

information in the raster background were better able to attend to the dual task of vertic

tracking. For the baseline-FLIR configurations, pilots may have been concentrating on

the expense of vertical path performance. 

Subjectively, the EPs felt that the tunnel provided good turn anticipation cues and the

when present, also 

background provided stro

overcontrol in roll i

symbology set (i.e., compensatory guidance symbol only). These differences are sub

data. In all cases tested, the pilots were able to meet operational performance requirem

5.2 Mental Workload 

Mental workload was assessed aft

post-test, using SWORD. 

The AFFTC workload estimate is a self-reported survey t

The scale assesses the immediate condition, using 7 p

1. Nothing To Do; No System Demands 

2. Light Activity; Minimum Demands 

3. Moderate Activity; Easily Managed; Considerable Spare Time 
33

5.2.1. AFFTC Workload Estimate – PF HUD 

 and symbology 

mbology 

interaction was not statistically significant (p>0.05). Post-hoc tests (SNK using α=0.05) showed two 

uniq

d FLIR-Baseline 

n the other 3 HUD 

e since the mean 

g boundary. On the AFFTC Workload Scale, a value of “3” indicates “Moderate 

Activity – Easily Managed; Considerable Spare Time” and a value of “4” indicates “Busy – Challenging 

ilable”. On average, all HUD configurations were rated as requiring 

bset tended more 

raster type (FLIR, 

g of 2.6 was 

given for the AD concepts by the pilots. On the AFFTC Workload Scale, a value of “2” indicates “Light 

inimum Demands” and a value of “3” indicates “Moderate Activity – Easily Managed; 

Considerable Spare Time.” 

absence of fusion 

F does not have a measurable effect on pilot monitoring workload. 

mental workload 

rm a task and the 

The post-test SWORD data indicate that there were no significant (p>0.05) differences among the PF 

HUD concepts for the SWORD ratings of mental workload. 

The post-test SWORD data show that AD concept was highly significant (F(3, 69)=15.02, p<0.001). 

Post-hoc tests (SNK using α=0.05) showed three unique subsets for the mental workload ratings with the 

4 PNF-AD concepts: Subset 1) FLIR-Symbology and Fused-Symbology (lowest workload); Subset 2) 

Fused-No Symbology; and, Subset 3) FLIR-No Symbology (highest workload). 

In the post-run data, the main factors of HUD raster (F(1,23)=5.46, p=0.028)

(F(1,23)=17.05, p<0.001) were significant for workload. The second-order raster by sy

ue subsets: 

Subset 1) Fusion-Tunnel (mean=3.1); and, 

Subset 2) FLIR-Tunnel (mean=3.4), Fusion-Baseline (mean=3.5), an

(mean=3.6). 

Pilots rated the Fusion-Tunnel HUD concept as having significantly less workload tha

concepts tested. While statistically-significant, the mean values show little differenc

values don’t cross a ratin

but Manageable; Adequate Time Ava

“Moderate Activity – Easily Managed; Considerable Spare Time” but the second su

toward being a “busy” activity. 

5.2.2 AFFTC Workload Estimate – PNF-AD 

For the PNF-AD concepts, there were no significant (p>0.05) differences between 

Fused), symbology (Off, On), or their interaction for post-run workload. A mean pilot ratin

Activity; M

This result indicates that the presence or absence of symbology and the presence or 

controls for the PN

5.2.3. SWORD 

Pilots were administered the pair-comparison SWORD test that enabled ratings of 

across the four display concepts (raster * symbology) for both the PF and PNF displays. The definition of 

mental workload was given as “the amount of cognitive resources available to perfo

difficulty of that task.” 
34

5.2.4 Workload Discussion 

pts. Pilots ranked 

 3 HUD concepts 

gnificant (p>0.05) 

Operationally, the 

nt that the average 

s tending toward 

e workload when 

and ND, easier to 

oad penalty in the 

tween the post-run 

lusion that the PF 

configuration differences. This result suggests that the addition of SV 

he addition of the 

n of both SV and 

d no statistically 

logy as requiring 

rkload in their SWORD ratings. (There were no appreciable differences 

between the FLIR-Symbology and Fused-Symbology AD concepts for mental workload ratings.) 

Pilo visually scan and 

cog ogy set contained 

two

ce cue superimposed on the outside world (either SV or EV); 

ding the PF to the 

F is following the 

rrain information, 

scan of the head-down instruments and the outerror in the form of “dogbones;” 

On the approach, no path error indication was provided on the head-down displays (neither PFD 

 the ND path and 

D. These data are 

The PNF workload data also suggest that the physical workload induced by adding fusion controls was 

either minimal or it was offset by the reduction in mental/cognitive workload provided by adding EV and 

SV information. 

The location of the PNF-AD was often noted as being “non-optimal.” It was too far away from the 

forward field of view / instrument panel and comments indicated that it added workload to the PNF 

because it required head and neck movements to see the display. The EPs would have preferred that the 

The workload data do not show substantial differences associated with PF-HUD conce

the Fusion-Tunnel HUD concept as having significantly less workload than the other

tested, post-run. However, the post-test SWORD data indicate that there were no si

differences among the HUD concepts for the SWORD ratings of mental workload. 

workload data suggest that the Fusion-Tunnel concept reduced PF workload to the exte

workload is “easily managed”, whereas the other concepts elicited workload rating

“challenging but manageable” workload levels. Pilot commentary suggested that th

flying the tunnel symbology concepts was easier (less scanning between the HUD 

anticipate the turns), but the differences were not of a magnitude to warrant a workl

opinion of the pilots. Since the definitions of workload and the types of tests differ be

(AFFTC Workload estimate) and the post-test (SWORD), it is difficult to draw a conc

workload was affected by the HUD 

(i.e., Fused vs. FLIR condition) did not increase PF workload on the approach nor did t

tunnel increase PF workload. If anything, the post-run results suggest that the additio

the tunnel symbology reduces PF workload. 

Similarly, the post-run AFFTC workload ratings for the PNF-AD concepts showe

significant differences, but post-test, pilots ranked the two AD concepts with symbo

significantly less mental wo

t commentary typically noted the advantage of symbology in reducing the need to

nitive task of integrating the different display information. The PNF-AD symbol

 key components of information: 

• Flight path marker and guidan

The FPM and guidance cue directly indicates to the PNF that the guidance is lea

correct location (guidance cue superimposed on a terrain background) and P

guidance (FPM is following the guidance cue). Without this symbolic and te

this information must be inferred from the PNF’s 

the-window view. 

• Path 

nor ND) although the presence of a lateral path error could be inferred from

ownship symbol. On final, path error was shown by the ILS raw data on the PF

not as precise nor as easily interpreted as the dogbones display. 
35

fortable visual field-of-regard of the primary instrument displays and the 

T, and after each 

both PF and PNF 

nd on attentional 

and, 3) understanding. From these components, the SART 

” (i.e., SART = 

ique that provides 

PNF displays. For 

mparisons, situation awareness (SA) was defined as “the pilot’s awareness and understanding of 

all factors that will contribute to the safe flying of their aircraft under normal and non-normal conditions.” 

measure differs from the post-run SART measure construct as it is a pair-comparison 

e SA ratings are 

both HUD raster type (F(1,23)=5.41, p=0.029) and symbology type 

0.001) and their interaction (F(1,320)=5.31, p=0.022) were significant for PF-HUD 

SART ratings. Pilots rated their SA significantly higher when the HUD symbology included 

in effects analysis 

n SA ratings was stronger within the Fusion raster type than 

pe (Fig. 11). 

sults for the main 

ion (p>0.05). 

3.61, p<0.001) for 

.05) showed three unique subsets for situation 

awareness ratings with the 4 HUD concepts: 1) Fusion-Tunnel (highest SA); 2) Fusion-Baseline and 

FLIR-Tunnel; and 3) FLIR-Baseline (lowest SA). 

The PNF-AD concepts were also highly significant (F(3, 69)=37.78, p<0.001) for the SA-SWORD 

ratings. Post-hoc tests (SNK using α=0.05) showed three overlapping subsets for situation awareness 

ratings with the 4 AD concepts: 1) Fused-Symbology; 2) FLIR-Symbology and Fused-No Symbology; 

and, 3) Fused-No Symbology and FLIR-No Symbology. Pilots ranked the Fused-Symbology AD concept 

as having significantly higher SA than the other 3 AD concepts tested. 

PNF-AD was within a com

forward view outside the aircraft . 

5.3 Situation Awareness 

Situation awareness was assessed after each experimental run using the post-run SAR

test, using the post-test SA-SWORD measures. Situation awareness was assessed in 

roles. SART is a multi-dimensional rating technique using the constructs of: 1) dema

resources; 2) supply of attentional resources; 

rating is “understanding” reduced by the difference of “demand” minus “supply

{(understanding) – (demand – supply)}). 

Similar to the SWORD described above, the SA-SWORD is a paired comparison techn

relative situation awareness ratings across the four display concepts for both the PF and 

these co

The SA-SWORD 

test, it was administered post-test, and the underlying definition and construct for th

different. 

5.3.1 SART – PF HUD 

An ANOVA revealed at th

(F(1,23)=18.85, p<

pathway/tunnel guidance and when the HUD used the Fusion imagery. A simple ma

revealed that the effects of symbology type o

within the FLIR raster ty

5.3.2. SART – PNF-AD 

An ANOVA on SART ratings across the PNF display conditions found no significant re

effects (raster, symbology) or the interact

5.3.3. SA-SWORD 

An ANOVA revealed that the PF HUD concepts were highly significant (F(3, 69)=4

the SA-SWORD ratings. Post-hoc tests (SNK using α=0
Figure 11. Means of SART Ratings With 95% Confidence Intervals for HUD Con

5.3.4. Situation A

figurations 

wareness Discussion 

tter understanding 

as reflected in the 

ey had to use the 

 unfortunately not 

ation unavailable 

rrain and pathway 

 Superior SA was 

 inferior for SA. 

e not significantly 

o SA was essentially equivalent 

to the tunnel contribution to SA. 

The benefits of Fused imagery and symbology on the PNF-AD only emerged within the post-test data. 

The post-test and post-run SA differences might be attributed to the fact that SA-SWORD asks for a 

general appraisal whereas the SART asks for ratings from what was experienced for that pilot on that run. 

SA can be high - it was high in all conditions, including the baseline, as they were all highly skilled pilots 

- and the task really wasn’t extremely demanding of the PNF. But, when asked to compare the PNF 

display concepts post-test to each other, SA differences emerged. 

Pilot commentary consistently noted that the presence of the tunnel gave the PF a be

and appreciation of the curving, descending visual arrival path. This commentary w

SART and SA-SWORD data. Without the tunnel guidance, pilots commented that th

head-down ND more frequently for path (turn) guidance. (Head-tracking data was

recorded for the PF.) 

Also, the SV component in the Fusion HUD concept provided significant terrain inform

in any other cockpit displays thereby providing an intuitive, conformal display of te

information. These SA features emerged in the SA data, both post-test and post-run. 

rated for the Fusion-Tunnel HUD concept whereas the FLIR-Baseline was rated

Interestingly, by the SA measures, the Fusion-Baseline and FLIR-Tunnel concepts wer

different. This result would imply that the pilots felt the SV contribution t

36
37

A 

 better monitor the 

t path with respect 

 and guidance cue 

he proper point on 

plays on the PNFere not shown on 

approach-tracking 

-AD attributes may not have been critical to the experiment on each run (i.e., 

n SART) but they can contribute significantly to SA - in general - for a PNF in 

this type of operation. 

the HUD and AD 

=73.17, p<0.001). 

display with the 4 

on-Tunnel; 2) Fusion-Baseline and FLIR-Tunnel; and 3) FLIR-Baseline. Pilots 

ranked the Fusion-Tunnel HUD concept as being preferred significantly more than the other 3 HUD 

ween the FusionD is equivalent in 

ed, but they didn’t 

s(F(3, 69)=23.74, 

the pilot-preferred 

ymbology; 

and 3) Fused-No Symbology and FLIR-No Symbology. Pilots ranked the Fused-Symbology AD concept 

cantly more than the other 3 AD concepts tested. In this case, the overlapping 

subsets in statistically-significant differences mildly suggests that symbology on PNF-AD was more 

rovided. Post-test 

 indicated that the 

symbology. (This 

5.5 Head-Tracking Analysis 

A head-tracker was used to quantify what display the PNF was using and when. An eye tracker would 

have been preferable for this task, but its installation was impractical. The head-tracker was unable to 

differentiate whether the PNF was looking OTW or at the instrument panel HDDs because very small 

head-movements, if any, were used by the PNFs to look between the HDDs and OTW. Fortunately, the 

head-tracker could very effectively distinguish when the PNF was looking at the AD or at the 

OTW/HDDs. The head-tracker statistical data analysis is detailed in Appendix A. 

Similar to the workload data, the pilots commented that they felt SA was impacted by several issues. S

was significantly improved with Fused imagery on the PNF-AD by providing a way to

EV and navigation system performance and improve their understanding of their fligh

to terrain. Symbology on the PNF-AD provided two key SA benefits. First, the FPM

(with FLIR and/or SVS imagery) provided visual evidence that the PF was flying to t

the ground (i.e., flying to the intended runway, touchdown point) and the raw data dis

AD symbology was a direct indication of path error for the PNF (i.e., the “dog-bones” w

the PNF PFD.) Without the dogbones, the PNF had to use the ND to monitor 

performance. These PNF

minimal impact on post-ru

5.4 Pilot Display Preferences 

Separate post-test paired comparisons for pilot display preferences were made on 

concepts after data collection was completed. 

The preferred HUD concept rankings showed highly significant differences (F(3, 69)

Post-hoc tests (SNK using α=0.05) showed three unique subsets for the pilot-preferred 

HUD concepts: 1) Fusi

concepts tested. Interestingly, the absence of statistically-significant differences bet

Baseline and FLIR-Tunnel HUD concepts indicates that the presence of SV on the HU

preference to the addition of tunnel/pathway information. Pilots prefer both to be add

prefer one more than the other. 

The preferred PNF AD concept rankings also showed highly significant difference

p<0.001). Post-hoc tests (SNK using α=0.05) showed three overlapping subsets for 

display with the 4 AD concepts: 1) Fused-Symbology; 2) FLIR-Symbology and Fused-No S

as being preferred signifi

preferred over Fusion, if a choice must be made and only one “feature” could be p

commentary, while strongly favoring the addition of symbology to the PNF-AD, also

PNFs wanted a symbology declutter control so they could selectively turn on or off the 

feature was not available to them.) 
38

D (also referred to 

e (in feet), Above 

age of time spent 

n phases of flight 

ctor - head location - was highly significant 

(p<0.001) within each altitude bins and post-hoc tests (SNK, α=0.05) showed 3 unique subsets for 

., HDD and OTW) 

 and then “Other” 

the PNF was using 

but increased their 

reached 400 ft AFL. Below 400 ft, AD usage quickly 

reduces to looking forward (HDD/OTW) 

below 200 ft (~90%) and pilot commentary suggested that the PNFs were looking OTW for the runway 

e

 4. P tage of Time PN cting at Va Head ions 

cation t > 1 500 <

< 1000 

40 L 

< 500 

3 FL 

< 400 

< AFL

< 300 

0 < AFL 

< 200 

AFL < 100 

In Table 4, the percentage of time that the PNF looked at either the AD, the OTW/HD

as the “Forward” location), or elsewhere is shown as a function of the aircraft altitud

Field Elevation (AFL). Simple ANOVA calculations were conducted on the percent

looking at the different head locations (Auxiliary Display, Forward, and Other) for seve

(or altitude segments). For each altitude segment, the fa

percentage of dwell across the head locations (Forward, AD, or Other). 

The mean percentages, also shown in Figure 12, indicates that the forward location (i.e

always had the greatest percentage for dwell time, followed by the Auxiliary Display

(i.e., neither looking Forward nor at the Auxiliary Display). The data also shows that 

the AD only about 14% of the time (collapsed across PNF-AD concept) above 1000 ft 

usage (up to 30%) as the aircraft descended until it

 about 10% of the time below 300 ft AFL. The PNF was mostly

visual refer nces.

Table ercen F Dire Gaze rious Locat

Head Lo Al 000 AFL 0 < AF 00 < A 200 10

Forward 

(HDD/OTW) 80.0% 64.7% 59.4% 72.1% 84.0% 88.7% 85.8% 

Auxiliary 

Display 14.1% 26.4% 29.9% 20.2% 11.2% 8.0% 9.5% 

Other than AD 

or Fwd 5.9% 8.9% 10.7% 7.7% 4.8% 3.3% 4.7% 
86%

89%

84%

72%

59%

65%

80%

10%

8%

11%

20%

30%

26%

14%

39

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

AFL < 100

100 < AFL < 200

200 < AFL < 300

300 < AFL < 400

400 < AFL < 500

500 < AFL < 1000

AFL > 1000

Altitude Range (ft, AFL)

Percentage of Time

Other

AD

HDD/OTW

ependent upon the 

ent looking at the 

hout symbology). 

rences for the AD 

The data shows that on approach, above 500 ft AFL, pilots focused on display concepts with symbology 

nditions, the FLIR 

FLIR-Symbology 

e part of the PNFs 

bology). This result is likely 

due to the PNF-AD symbology which included the path error – this information was not readily derived 

from the other PNF displays. 

Between 500 and 200 ft AFL, the usage of all the display concepts increased. The presence of symbology 

primarily and SV secondarily induced attention to the AD. While statistical significance is shown in 

Table 5 below 200 ft, the differences in display concept are operationally insignificant. (10% percentage 

represents approximately 1 second of time looking at the PNF between 100 and 200 ft AFL.) 

Figure 12. Percentage Of Time Where PNF Directed Gaze. 

In Table 5, the percentage of time the PNF spent looking at the AD is shown to be d

AD display concept. Simple ANOVAs were conducted on the percentage of time sp

different AD concepts (FLIR, with and without symbology and Fused, with and wit

Every altitude segment, except the AFL < 100 feet segment, showed significant diffe

concept. 

more than those with terrain information (SV or EV). (Given the simulated weather co

did not provide any terrain information above 1000 ft AFL.) For instance, the 

configuration induced more gaze, as estimated from the PNF head tracker data, on th

than the Fused-No Symbology configuration (terrain information but no sym
40

Table 5. Percentage of Time PNF Directing Gaze, By Display Configuration 

Altitude Range (ft, AFL) 

AD Concept FL

100

00

< 

400 < AFL < 100 < AFL < 

200 AFL < 100 A > 

0 

5 < AFL 

1000 < 500 

300 < AFL 

< 400 

200 < AFL

300 

Significance? 4 Uniq

Subsets 

4 Un

Subsets 

2 Unique 

Subsets 

2 Unique 

Subsets 

2 Overlapping 

Subsets 

verlapping 

Subsets None ue ique 2 O

FLIR - 

No Symbology 

5% 18% 24%* 15%* 9%* 6%* 9% 

Fused - 

ymbology 13% 22% 24%* 15%* 10%* 7%*† 10% No S

FLIRSymbology 16% 30% 36%† 25%† 12%*† 10%† 11% 

Fused - 

Symbology 23% 35% 36%† 27%† 14%† 9%† 9% 

* Member of Subset 1 † Member of Subset 2 

Based on the quantitative gaze data in Table 5 and from the pilot debriefs, below about 300 ft AFL, all the 

ing for visual landing references and preparing for landing. On 

occasion, a PNF would state that they tried to take a glance at the PNF-AD. They noted that if the PNFmight have had a 

more. 

on display clutter 

ure 8) mirror the SART rating paradigm and are not a validated 

estions were an attempt to quantify clutter effects as it affects a pilot’s SA. In 

essence, this creates a “Clutter-SA” rating – that is, it attempts to quantify the visual demand, supply, and 

 experimental trials. The last question was the pilot’s subjective 

eline for the “Clutter-SA” data. (These data are 

qua

5.6.

For the HUD concepts, the data were as follows: 

• PF-HUD Clutter (Rating #1 Demand on Visual Attention)

This rating corresponds to the pilot’s subjective assessment of their visual effort to scan and 

locate task-critical display information. Symbology type (F(1,23)=18.87, p<0.001) was 

significant for this rating with the Tunnel concept resulting in a lower rating than the Baseline 

symbology. The raster type main effect and the raster-by-symbology interaction were not 

significant (p>0.5) for this measure. Post-hoc tests (SNK using α=0.05) showed three unique 

PNFs were predominately “eyes-out” look

AD were installed on the forward instrument panel, in a convenient location, they 

tendency to use it more. But likely, not significantly much 

5.6 Subjective Assessments of Display Clutter 

After an experimental run, each pilot gave ratings for the 4 Likert-type questions 

(Figure 8) for the display (HUD or AD) concept they had just flown. 

The first three clutter questions (see Fig

methodology. The qu

understanding by the pilots during the

assessment of display clutter. This data serves as bas

being used primarily in another related study to assist in the development of validated quantitative and 

litative display clutter metrics.) 

1 HUD Concept Ratings 
41

combinations: 1) 

.3); 2) FLIR-Tunnel (mean=3.5); and, 3) Fusion-Baseline(mean=3.8); 

and FLIR-Baseline (mean=3.9). 

subsets (Fig. 13) for demand on visual attention ratings with the 4 HUD 

Fusion-Tunnel (mean=3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

n-Tunnel

Mean clutter rating

Subset 1

Subset 2

FLIR-Baseline Fusion-Baseline FLIR-Tunnel Fusio

Subset 3

UD Concept.

of visual attention 

condary tasks. The data shows that raster type (F(1,23)=4.90, p=0.037) 

and symbology type (F(1,23)=12.26, p=0.002) were statistically significant for the HUD. Pilots 

rated that they had a highest supply of visual attention resources when flying with the Fusion 

raster and the Tunnel symbology HUD concept compared to the other three concepts. Post-hoc 

tests (SNK using α=0.05) showed three overlapping subsets (Fig. 14) for supply of visual 

attention resources ratings with the 4 HUD combinations: 1) Fusion-Tunnel (mean=5.1); 2) 

FLIR-Tunnel (mean=4.8) and Fusion-Baseline (mean=4.6); and; 3) Fusion-Baseline (mean=4.6) 

and FLIR-Baseline (mean=4.5). 

Figure 13. Mean Ratings Of Demand On Visual Attention Clutter Metric For Each H

• PF-HUD Clutter (Rating #2 Supply of Visual Attention Resources )

This rating corresponds to the pilot’s subjective assessment of their supply 

available to attend to se
0.0

1.0

2.0

3.0

4.0

5.0

6.0

Subset 1

Subset 2

FLIR-Baseline Fusion-Baseline FLIR-Tunnel Fusion-Tunnel

Mean clutter rating

Subset 3

Fig ch HUD Concept.

• PF-HUD Clutter (Rating #3 Understanding)

 pilot’s subjective assessment of their understanding of task-critical 

ymbology 

ting for the HUD 

 the task-critical 

information on the HUD. 

splay clutter. The 

logy type, or their 

ond to a moderate 

Intuitively, one might infer that adding more information to a flight critical Primary Flight Display, like 

adding a pathway guidance or tunnel to the HUD, would negatively increase display clutter. On the 

contrary, these ratings suggest that the addition of the Tunnel to the HUD did not impact clutter. No 

difference in the overall clutter rating or in the pilot’s understanding of task critical information was 

shown. Further, the “Clutter-SA” construct ratings showed statistically-significant improvements in 

“Visual SA” by the addition of the tunnel. The ratings showed statistically-significant reductions in the 

demand on visual attention resources and increases in the supply of visual attention resources to attend to 

ure 14. Mean Ratings Of Supply Of Visual Attention Resources Clutter Metric For Ea

This rating corresponds to the

display information. Symbology type and raster type main effects and the raster by s

interaction were not significant (p>0.05) for this measure. The average ra

concepts was 6.1 which correspond to a moderately high understanding of

• PF-HUD Overall Clutter (Rating #4)

This rating corresponds to the pilot’s subjective assessment of the degree of di

rating data showed no statistically significant differences for raster type, symbo

interaction. The average rating for the HUD concepts was 3.3 which corresp

amount of display clutter. 

42
43

V information) also 

y of visual attention resources so that he/she could perform secondary tasks. 

 for raster type, symbology, or their interaction on 

the pilot’s subjective assessment of the demand on visual attention. The average rating for the 

ly low demand of visual attention to scan 

There were no significant (p>0.05) differences for raster type, symbology, or their interaction in 

 their supply of visual attention resources. The average rating 

f visual attention 

0.05) differences for raster type, symbology, or their interaction in 

-AD information. 

igh understanding 

• PNF-AD Overall Clutter (Rating #4)

 pilot’s subjective 

NF-AD concepts 

ncepts with symbology 

(mean=2.2). Both of these mean ratings for the PNF-AD concepts correspond to a moderately 

s is minimal. 

tributed to clutter 

(based on their “overall clutter” ratings). The three subjective elements of the “Clutter-SA” metric, 

t show statistically significant differences. The absence or presence of PNF-AD 

symbology did not affect the post-run PNF-AD ratings for visual understanding, demand or supply. 

and a completely 

clear FLIR or Fusion raster on the AD to promote better readability and understanding of the imagery. 

Their proposed solution was to include symbology on the PNF-AD and also, include a PNF-AD 

“declutter” button, analogous to the PF-HUD, so the symbology could be toggled on and off as needed. 

5.7 Fusion Controls 

By providing a plethora of controls to the PNF for the Fused AD concept, the experiment data provided 

first-order determination if: a) a “fusion” concept was viable in the commercial cockpit; b) allowing the 

secondary tasks with the addition of the tunnel. Fusion (i.e., the presence of S

increased the pilot’s suppl

5.6.2 PNF-AD concept Ratings 

For the PNF-AD concepts, the data were as follows: 

• PNF-AD Clutter (Rating #1 Demand on Visual Attention)

There were no significant differences (p>0.05)

AD concepts was 2.5 which correspond to a moderate

and locate task-critical display information on the PNF-AD. 

• PNF-AD Clutter (Rating #2 Supply of Visual Attention Resources )

the pilot’s subjective assessment of

for the AD concepts was 5.7 which corresponded to a moderately high supply o

resources to effectively attend to other secondary visual tasks. 

• PNF-AD Clutter (Rating #3 Understanding) 

There were no significant (p>

the pilot’s subjective assessment of their understanding of task-critical PNF

The average rating for the AD concepts was 6.4 which corresponded to a very h

of task-critical display information. 

Symbology type (F(1,23)=6.30, p=0.020) was statistically significant for the

assessment of the degree of PNF-AD clutter. The “No-symbology” P

(mean=1.8) were rated as having less overall clutter than the PNF-AD co

low level of display clutter; thus, the operational significance of these difference

The pilots noted that the symbology on the PNF-AD was beneficial to SA, but con

however, did no

The pilots, in their post-test debriefing, clearly advocated that they wanted symbology
44

 necessary for the 

tem operation was 

ng, the pilots were 

EPs quickly 

s were. 

 PNFs during the 

 segment (“APP”: 

). A fusion 

setting of 1 corresponds to SVS-only on the PNF-AD, a fusion setting of 10 corresponds to EVS-only , 

and a setting of 2 through 9 corresponds to the feature-level extraction algorithm bias as described in 

Section 3.10.2. 

PNF to control their presentation was viable or desirable; and, c) SVS and EVS was

PNF. No “guidance” was provided to the pilots for optimal usage. Instead, the sys

explained and they were given the training runs to “play” with the controls. After traini

allowed to use whatever control settings that they felt necessary and appropriate. The 

learned how the fusion worked and what the most effective means to employ the control

In Figure 15, the percentage of time that a fusion control setting was used by the

experiment is plotted by altitude range. The altitude ranges correspond to the approach

run start to 500 ft AFL), and 100 ft increments from 500 ft AFL to touchdown (or go-around

1 2 3 4 5 6 7 8 9 10

0

60

10

20

30

40

50

All Crew Average

Percentage

App

500−400

400−300

300−200

200−100

100−TD

PNF−AD Fusion Control Setting

Figure 15. Fusion Control Settings By Altitude. 

The data indicate a consistent trend. At altitude (on approach down to approximately 500 ft AFL), the 

PNFs tended to use the feature-level fusion of EV and SV. The most prevalent settings were heavily 

weighted toward EV but still contain SV content (i.e., settings of 8 and 9 in Figure 15). On the approach 

segment, the EVS did not have any information content because of simulated clouds on the approach. 

With feature-level extraction, the fusion image shows the SV database image without significant 

SV only EV only 
45

alue of 8 so when 

seful information. 

 data indicate that 

mation that 

assisted in their monitoring function. The EPs quickly learned how the fusion worked and what the most 

F used any fusion 

settings 1 through 

s flown. The data 

f the time. Below 

d Fusion (i.e., SV) 

0 distribution in EV-only and “SV-Fusion” settings, respectively, suggests that 

the PNF used both information sources cooperatively and effectively. The common PNF strategy was to 

cycle between EV-only and the highest level of SV/EV fusion. They could do this quickly because these 

were adjacent switch positions. 

alteration or contrast reduction. The PNFs often used an intensity (false color-coding) v

color appeared on the PNF-AD, this cued the pilot that the EV was starting to show u

The color signalled that they could effectively begin using an “EVS-Only” setting. The

fusion control was used - albeit not to its full-range - and the PNF gathered significant infor

effective means to employ the controls were. 

In Figure 16, these data are collapsed to highlight the percentage of time that the PN

settings (i.e., “SVS-Only” or feature-level fusion values, “SV-Fusion,” fusion control 

9) or “EV-Only” (i.e., fusion control setting 10) when the PNF-Fused AD concept wa

shows that, on the approach, fusion or SV information was displayed more than 85% o

500 ft, in the final approach segment to landing, EV-only was used 60% of the time an

reduced to 40%. The 60-4

0% 20% 40% 60% 80% 100%

AFL < 100

100 < AFL < 200

200 < AFL < 300

300 < AFL < 400

400 < AFL < 500

AFL > 1000

Altitude Range (ft, AFL) 

Percentage 

EV-Only

SV - Fusion

Figure 16. SV/EV Fusion And EV-Only Setting By Altitude AFL. 

5.8 Non-Normals 

Non-normals were injected into the test unbeknownst to the evaluation subjects. The non-normals were 

two runway incursions and four lateral offsets for each flight crew. 
46

5.8.1 Runway Incursions 

were positioned in 

d and just slightly 

., they were facing 

 The weather on the runway incursions was held constant at 2400 ft RVR 

n this condition – 

The baggage cart runway incursion was always performed before the fire truck incursion. The baggage 

r “just noticeable 

 concepts 

available to the PFs and PNFs are shown in Figure 17. Each of the pictures was taken at 60 ft AFL, 

7(a) and Fig. 17(d) 

tained in the other 

his figure. 

 key attributes that 

olor (presence and 

 in relation to the 

lated) visual scene 

e used to compute the theoretical altitude (AFL) that the RIs might be reasonably expected 

to be observed by the crew. While target detection is critically dependent upon visibility, lighting, target 

at 10 pixels (scan 

baggage cart 

 to a cart 6.5 ft tall 

nd 13 ft tall. An operating rotating beacon was depicted 

atop the Fire Truck. 

vehicles is shown. 

own on the HUD 

d PNF-AD FLIR 

 degree. The OTW resolution provides 26 pixels per degree. The vertical and 

horizontal resolutions (i.e., pixels per degree) were identical. 

The theoretical detection ranges show that the Fire Truck should be detectable at almost twice the 

distance as the Baggage Cart, particularly its vertical extent (i.e., height). The OTW provided better 

detection capability because of its higher resolution. Finally, none of the vehicles were theoretically 

“detectable” in this analysis above 200 ft AFL using the FLIR (HUD or PNF-AD). It should be noted, 

however, that the Fire Truck was detectable above 200 ft AFL on the EV, if the observer was cued to its 

existence and studied the display. 

The runway incursions were created by a baggage cart and a fire truck. Both vehicles 

the same location, approximately 850 ft from the RNO Runway 16R landing threshol

offset from the centerline. They were both positioned perpendicular to the runway (i.e

toward the runway edgelines.)

(OTW) with the lowest cloud layer at 500 ft AFL. The FLIR visibility was very good i

approximately 4 times the OTW RVR. 

cart was much more difficult to see due to its small size. This ordering tested fo

differences” for runway incursion detection. 

The view of the baggage cart and firetruck incursions using several of the displays and display

approximately on glideslope and localizer to landing. A yellow oval is added to Fig. 1

to highlight the location of the incurring vehicles. The incurring vehicles are also con

images in the approximately same location within the other display concepts shown in t

While innumerable issues factor into the data analysis for runway incursion, four of the

must be considered in analyzing these data are: display resolution, target size, display c

absence), and the presence or absence of symbology. These factors will be discussed

experimental data. 

To put the RI results into perspective of display resolution, the EVS and OTW (simu

resolutions wer

contrast, color, gray scales, etc., resolution only is used in this example. It is assumed th

lines) are required for a human observer to recognize a target/object in this example.29 The 

consisted of a tug and a cart. The tug was approximately 7.5 ft tall and 10 ft long, tied

and 10 ft long. The Fire Truck was 31.6 ft long a

In Table 6, the height of the aircraft (AFL) when 10 lines (pixels) draw the incurring 

The analysis assumes a 3 degree glideslope, while the limiting resolution of the EV sh

and PNF-AD was that of the simulated FLIR (640x480 resolution). The HUD an

resolution was 20 pixels per
47

Table 6. Computed AFL Altitudes for Theoretical Detection of Runway Incursion 

tion 

de for EVS

“Detection” 

de for OTW 

“Detection” Object Dimension / Scan Direc

AFL Altitu AFL Altitu

Baggage Cart Height 42 ft 58 ft 

Width 120 ft 156 ft 

Fire Truck Height 78 ft 101 ft 

Width 190 ft 246 ft 

The experimental results showed that, for the 12 flight crews, only one crew memb

baggage cart (they saw it in the OTW) and initiated a go-around. The other 11 crews h

er (PNF) saw the 

ad a runway 

ld be “detectable” 

very much time to 

y short of the intended touchdown zone. 

Eleven crews saw the Fire Truck OTW (7 by the PNF, 3 by the PF, and 1 simultaneously by the PF and 

PNF) and the one remaining crew saw it on the PNF-AD. Upon seeing the incursions, all crews initiated 

a go-around (all lower than 50 feet AFL). 

incursion with the baggage cart. From the analysis of Table 6, the baggage cart shou

between 50 and 150 ft AFL but this leaves only 5 to 15 seconds before landing – not 

spot a small object, parked slightl
b) Head-Up Display

c) Auxiliary Display

a) Out-The-Window

e) Head-Up Display

f) Auxiliary Display

d) Out-The-Window

Baggage Cart Fire Truck

Figure 17. Runway Incursions (Baggage Cart And Firetruck) As Seen At 60 Ft AFL Using Various Displays. 

These data highlight several important factors in RI detection by flight crews: 

• The vehicles were not moving so motion effects which might aid target detection were absent. 

48
49

and color contrast 

timistic detection 

ver color contrast 

was available between the incurring vehicles and the runway background was only available to 

d). 

ntal data suggests 

e HUD (FPM and 

tal HUD, like all 

ontrast is also a 

ing the aircraft and 

ing the proper visual references for landing. Therefore, successful runway incursion 

idered as a likely 

t of the 24 runway 

 the data does not 

 attuned to the RI 

proved gray scale 

 with Symbology). 

e detection on the 

hicles were much 

more obvious in the image, but the PNF noted that they were head-out the vast majority of the 

nt. The PNF was 

e time, ranging from 86% to 100% of the total time below 200 ft. 

ncursion detection 

scure the vehicles, 

 EV imagery) did 

 area. 9 out of 24 

elow, the PNF is 

y not be practical 

he tested PNF-AD 

of the display on final. Current flight crews are 

not familiar with using head-down displays on short final to check for incursions. This was not 

part of the pre-experiment flight crew instructional briefing. 

• The results suggest that EV on the HUD and PNF-AD were not useful

• The incurring vehicle scenarios were intentionally designed as low luminance 

targets. Acuity metrics, such as those touted in Table 6, will provide op

performance since the metrics typically assume high c st ontra targets. Whate

the pilots when using the OTW information for detection (see Figure 17a and 17

• The incurring vehicles were visible in the PNF-AD and HUD, yet the experime

that EVS imagery on the HUD and PNF-AD were not useful for RI detection. 

• In the HUD, the incurring vehicles were largely occluded by symbology on th

guidance cue) – see Figure 17b and 17e as examples. Further, the experimen

HUDs currently manufactured, have limited contrast ranges. The image c

function of pilot control. Additionally, the PF is consumed with the task of fly

identify

detection at this late stage on the approach by the PF should not be cons

outcome. The data confirmed this hypothesis in that the PF detected only 4 ou

incursions. 

• In the PNF-AD, the vehicles were much more apparent than in the HUD, but

indicate significantly greater detection success. The PNF-AD display is more

detection task because of the lack of symbology (in some cases) and its im

contrast performance (see Figure 17c and Figure 17f as examples of PNF-AD

But, as the theoretical data shows, the vehicle size and EVS resolution mak

PNF-AD moderately difficult above 200 ft AFL. Below 200 ft AFL, the ve

time. The head tracking measurements shown above quantify this pilot comme

head-out the vast majority of th

AFL. Based on these data and the pilot comments, the use of the PNF-AD for i

was not probable. The presence of symbology on the PNF-AD could also ob

particularly if the PNF only used cursory looks at the PNF-AD. 

• The use of PNF Fusion controls when available (i.e., toggling between SV and

not statistically help or hinder detection. 

• Unlike the PF, the PNF is tasked with monitoring the approach and the landing

incursions were detected by the PNF. But, starting around 500 ft AFL and b

going “eyes-out” so runway incursion detection using a head-down display ma

(only one PNF saw the incursion on the AD). The off-boresight design of t

was noted by the pilots as hampering their use 

 for RI detection. In the 

HUD, the incurring vehicles were largely occluded by symbology on the HUD (flight path 

marker and guidance cue) and the small size and relatively low resolution of HUD made vehicle 

detection extremely difficult for the PF. In the PNF-AD, the targets were not highly visible in the 

flight regime where the PNFs were engaged with the display. 
50

UD and PNF-AD 

cursion detection 

 of the problem, 

nd crew interface 

s on short final to 

 HUDs and more 

ions, including the 

M. The displays 

mage “zoom” - to 

n for object detection before the final phase of the approach – should be 

considered as it could increase the probability for object detection and recognition before the PNF goes 

apability to declutter symbology from the displays should be provided and used for a 

F-HUD with 

 The offsets could 

rs were noticeable from one of several principal ways 

iation symbology (HUD 

r deviation on the PFD when the PF is flying on the final approach path 

centerline. 

een the SV and the EV registration using the PNF-AD Fusion controls. 

• By differences between the runway outline and the EV imagery of the runway (HUD and PNFAD concepts with symbology). 

• By differences in the pitch/roll guidance symbol and the EV imagery (PF-HUD and PNF-AD 

concepts with symbology) 

The display concepts tested in this experiment – typical of current and future PF-H

displays – showed that requirements for display and sensor technology for runway in

should be developed. The requirements for this function should span the breadth

including human perception, sensor design and detection theory, crew procedures, a

issues. Current flight crews are not necessarily familiar with using head-down display

check for incursions. Flight crews should be trained on effective use of EV-capable

importantly, on the PNF’s use of EV-capable head-down displays to check for incurs

use of declutter controls, complementary SV information, and their implications on CR

are not necessarily optimized for this role. For instance, the capability to provide EV i

increase the EV resolutio

“eyes-out.” The c

runway “check” before landing. 

5.8.2 Navigation Error 

The navigation errors were either a 50 foot or 75 foot lateral offset (see Figure 18a for P

Tunnel Symbology and in Figure 18b for PNF-AD with FLIR-Symbology examples).

be detected by either the PF or the PNF. The erro

(depending upon the display configuration) using numerous display indications: 

• By a disagreement between the lateral path error and the localizer dev

and PNF-AD concepts with symbology). 

• By a non-zero localize

• By differences betw
Figure 18. Lateral Navigation Offset Examples. 

and 75 foot offset 

y. Each performed 

 

~85%) when they 

nway. The “real” 

non-zero localizer 

ke when confronted with a navigation 

ts had relatively little training and experience with the system. Despite this, the study 

lly acknowledged a significant percentage of time and, 

 and accurately on 

, and/or database 

ining to recognize 

es could further improve operations in the event of this anomaly. 

5.8.3. Illegal Landings 

Although not technically a “non-normal,” each flight crew was confronted with four trials where weather 

conditions obscured the required visual cues to complete the landing as defined by FAR §91.175. Of the 

48 “illegal landing” rare event trials, during only six of these trials did pilots continue and land the 

aircraft. (See Table 7.) 

The majority of flight crews verbally noted the presence of the 50 foot offset (15/24) 

(19/24) during the approach. None of the pilots executed a go-around with this anomal

a lateral correction and landed near the runway centerline.

Video analysis showed that navigation errors were predominately noted by the PF (

noticed that the pitch/roll guidance symbol was leading them to the left or right of the ru

runway was detectable in the EV image. One person (flying as the PNF) noted the 

deviation on the PFD presentation while tracking the path centerline. 

The flight crews were not instructed on the course of action to ta

error, and the pilo

showed that lateral navigation errors were verba

even when unrecognized (i.e., not explicitly verbalized), all flight crews landed safely

the runway. These results suggest that dissociations between raw data, sensor

presentation should be easily recognized and managed by experienced pilots. Pilot tra

these discrepa ci n

51
52

Table 7. Illegal Landing Trials 

Crew Observation HUD AD 

3 ered FLIR 

No 

Fused 

No Symbology Go-around consid Tunnel 

4 t 80 f F

No 

FLIR/ 

Symbology Threshold lights called a t AFL LIR Tunnel 

4 60 ft F

No Tu

Fused 

 Symbology Threshold lights called at AFL LIR nnel 

7 Threshold lights called at 100 Tunn

Fused 

 Symbology ft AFL FLIR el 

8 Threshold lights called at 120 ft AFL Fused 

ology 

FLIR 

No Tunnel No Symb

11 Threshold lights called at 90 ft AFL FLIR 

No Tunnel 

FLIR

No Symbology 

The results demonstrated a small but finite potential (12.5%) for flight crews to contin

landing during visibilit

ue approaches to a 

y conditions that instead require a go-around under the new §91.175 operational 

t visibility of the 

nces until passing 

S rule requires visual acquisition of the runway references 

bserved (nor was 

ected due to the low statistical power of the rare event) although 5 of the 6 

 runway with 

eclutter the HUD 

e used for judging 

L. An aural callmay have aided adherence to the DA. The aural call-outs were set to Cat. I decision heights. 

The few occurrences of “below minimums” landings suggest that the current regulations can be 

operationally viable. However, an aural call-out at 100 ft AFL is recommended. Nonetheless, there still 

exists an awkwardness in the transition from EV/HUD-to-visual runway references. The PFs typically 

commented that the EFVS provided suitable visual references to complete the flare and landing. Future 

research should examine the effects of eliminating the natural vision/visual segment requirement of FAR 

§91.175 in other than Category II or III operations while conducting low visibility approach and landing 

with an EFVS. 

rules for EFVS. On all of the illegal landing scenarios, the pilot flying had excellen

runway using the FLIR on the HUD, but didn’t see the runway visual landing refere

through 100 ft AFL. (The current §91.175 EFV

without use of the EFVS.) 

No statistically significant effects of symbology type on the HUD concept was o

statistical significance exp

events occurred when the runway outline was not present. 

The operational procedures necessary to follow the §91.175 regulation (i.e., for seeing the

“un-aided” vision) were generally found to be awkward for the PF, requiring the PF to d

or to look-around the HUD combiner. The radio altitude shown on the HUD could b

height above touchdown (HAT). 

The PNF was typically “eyes-out” and not closely monitoring the altitude at 100 ft AF

out of 100 ft 
53

6. Conclusions 

VS technologies, 

 on new techniques for integration and/or fusion of synthetic and enhanced vision 

technologies and crew coordination while operating under the newly adopted FAA rules which provide 

n and/or fusion of 

NF was not 

substantially different when flying the tested concepts. Increasing the “informational complexity” of the 

bology on a PNFand by 

adding fusion control and symbology on the PNF-AD. For the PF-HUD, the overall clutter was not rated 

– a new construct 

of the tunnel and 

 reduces the RMS 

epts. The tunnel 

. However, while 

 

y of the flight crew to handle a substantial navigational errors was not impacted by the display 

concepts. They were verbally acknowledged a significant percentage of time and for all these runs, the 

ror. These results 

thway information 

r induce pilots to 

n or other display 

acted nor significantly aided 

ormation (runway 

e was also not an 

 the FLIR. The display concepts and 

scenarios tested in this experiment – typical of current and future PF HUD and PNF-AD displays - did not 

show adequate incursion detection functionality. All but one of the runway incursion scenarios were 

detected without the use of the cockpit displays. Sensor and display design must be tailored to this 

function and corresponding crew procedures and interfaces developed to support RI detection. 

Symbology on the PNF-AD was strongly preferred and rated highly, but the presence of symbology 

degraded the readability of the raster, particularly of the runway and touchdown point. The PNFs 

strongly suggested that a declutter capability on the PNF-AD should be included. 

An experiment was conducted to evaluate the complementary use of SVS and E

specifically focusing

operating credit for EVS. 

These data show that significant improvements in SA can be provided by the integratio

synthetic and enhanced vision technologies for the PF and PNF. Workload for the PF and P

HUD by adding SVS and tunnel data, and increasing the number of controls and sym

AD did not affect PF or PNF workload. 

In contrast, SA for the PF and PNF was improved by the addition of tunnel and SVS on the HUD 

as being any worse by the addition of SV and tunnel information. Clutter-SA ratings 

introduced in the test - showed statistically-significant improvements by the addition 

SV. 

Analysis of the PF-HUD concept data showed that adding tunnel/pathway information

lateral path error on the approach compared to the baseline, no-tunnel HUD conc

provided turn anticipation cues for the turning, descending arrival flown in this test

statistically significant, the differences were not judged to be operationally significant. 

The abilit

pilots landed safely and accurately on the runway despite the large lateral navigation er

lend convincing evidence to the assertion that the potentially compelling display of pa

(in the form of a runway outline in this test) does not adversely capture attention no

follow erroneous display information in the presence of real-world visual informatio

cross-checks. 

The ability of the flight crew to handle a runway incursion was neither imp

by the display concepts tested. Although the increase in near-domain symbology inf

outline) did not degrade pilot response to the Fire Truck runway incursion event, ther

observed enhancement in incursion detection as hypothesized for
54

7. References 

, R.E., and Prinzel, L.J. III, Flight Testing An Integrated Synthetic 

ditor: Jacques G. 

Editor: Jacques G. 

In Proceedings of 

 5802, Paper 18, 

ial and Military 

n, Human Factors 

hods for Testing and Evaluation 

005. 

mparison Between 

nced Vision (FLIR) and Synthetic Vision Systems. In Proceedings of SPIE, Enhanced and 

5. 

ed, Enhanced and 

 Paper presented at the RTO SET Workshop on 

published in RTOnate Concepts for 

Simulated 

Display. The 23rd

2004. pp. 4.A.1 - 41-

uation of EVS for Approach Hazard 

 Synthetic Vision 

-107. 

[11] Harrah, S.D., et al. h To Realize A Sensor Enhanced-Synthetic Vision System 

(SE-SVS). Paper presented at 21st IEEE / AIAA Digital Avionics Conference, October 2002. 

[12] Uijt De Haag, M., et al. Flight Test Results of a Synthetic Vision Elevation Database Integrity 

Monitor. Paper presented at SPIE’s 15th Annual International Symposium on Aerospace / Defense 

Sensing, Simulation and Controls, AeroSense, Orlando, FL, April 16-20, 2001 

[13] Jones, D. R., Quach, C. C. and Young, S. D.: Runway Incursion Prevention System – Demonstration 

and Testing at the Dallas/Fort Worth International Airport. Proceedings of the AIAA/IEEE 20th 

Digital Avionics Systems Conference, 2001. 

[1] Joint Planning and Development Office. Next-Generation Air Transportation System Integrated 

Plan, Version 1. Dec 2004. 

[2] Kramer, L.J., Arthur, J.J., Bailey

Vision System. In Proceedings of SPIE, Enhanced and Synthetic Vision 2005, E

Verly, Volume 5802, Paper 1, April 2005. 

[3] Arthur, J.J., Prinzel, L.J., Kramer, L.J., Bailey, R.E., and Parrish, R.V. CFIT Prevention Using 

Synthetic Vision. In Proceedings of SPIE, Enhanced and Synthetic Vision 2003, 

Verly, Volume 5018, Paper 16, April 2003. 

[4] Schiefele, J. et al. Human Factors Flight Trial Analysis for 3D SVS: Part II, 

SPIE, Enhanced and Synthetic Vision 2005, Editor: Jacques G. Verly, Volume

April 2005. 

[5] Schnell, T., Theunissen, E., and Rademaker, R. Human Factors Test & Evaluation of an Integrated 

Synthetic Vision and Sensor-Based Flight Display System for Commerc

Applications. Paper presented at the NATO Research and Technology Organizatio

and Medicine panel workshop entitled “Toward Recommended Met

of EV and E/SV-Based Visionic Devices”, Williamsburg, VA, USA, 26-27 April 2

[6] Arthur, J.J., Kramer, L.J., and Bailey, R.E., and Prinzel, L.J. III. Flight Test Co

Enha

Synthetic Vision 2005, Editor: Jacques G. Verly, Volume 5802, Paper 3, April 200

[7] Craig, G., Jennings, S., Link, K., and Kruk, R.: Flight Test of a Helmet-Mount

Synthetic Vision System for Rotorcraft Operations.

"Enhanced and Synthetic Vision Systems" held in Ottawa, CA, Sept. 2002, and 

MP-107. 

[8] Parrish, R.V, Busquets, A.M., Williams, S.P., & Nold, D.E.: Evaluation of Alter

Synthetic Vision Flight Displays with Weather-Penetrating Sensor Image Inserts During 

Landing Approaches. NASA TP-2003-212643, October 2003. 

[9] Theunissen, E., et al: Integration of Imaging Sensor Data into a Synthetic Vision

Digital Avionics Systems Conference, 2004. DASC 04. Volume 1, 24-28 Oct. 

10. 

[10] McKay, D., Guirguis, M., Zhang, R. and Newman, R.L.: Eval

Detection. Paper presented at the RTO SET Workshop on “Enhanced and

System”, held in Ottawa, Canada, September 2002, published as NATO RTO-MP

The NASA Approac
55

g of Stored Geoing Forward-looking Remote Sensing Technology. PhD Dissertation, Ohio 

ilization of GPS Surface Reflected Signals To Provide Aircraft 

thetic Vision 2005 

S-ANM100-2001-

parison of Additive Image Fusion vs. 

t-Level and Realndu Sinha, Phillip 

e, Editors, Proceedings of SPIE, Vol. 4796 (2003), pp. 140-151. 

A. Wise, & V. D. 

, N.J.: Lawrence 

sed Synthetic and 

ceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, 

ues for Improving 

tion Awareness in the Aircraft Cockpit. International Journal of Aviation Psychology, Vol. 1, 

d 

t: 1997. pp, 32-1 

o and Symbology 

ehavioural 

tegrated Synthetic 

vionics Systems 

0 Oct.-3 Nov. 2005, pp: 4.E.4-1 - 4.E.4-18. 

y) III: Technical 

opment of a NASA Synthetic Vision System Concept. Paper presented at the 

Ottawa, Canada, 

[25] Crawford, J. & Neal, A.: A Review Of The Perceptual And Cognitive Issues Associated With The 

Use Of Head-Up Displays In Commercial Aviation. The International Journal Of Aviation 

Psychology, 16(1), 2006. pp. 1–19. 

[26] Prinzel III, L. J. & Risser, M.: Head-Up Displays And Attention Capture. NASA TM-2004-213000. 

Washington, DC: NASA, 2004. 

[27] Wickens, C. D., Ververs, P. M., & Fadden, S.: Head-Up Displays. In D. Harris (Ed.), “Human 

Factors For Civil Flight Deck Design.” Burlington, VT: Ashgate Publishing, Ltd. 2004, pp. 103-

140. 

[14] Young, S.D.: On the Development of In-flight Autonomous Integrity Monitorin

spatial Data Us

University, March 2005. 

[15] Ganoe, G., and Young, S.D.: Ut

Altitude Verification For SVS. SPIE. In Proceedings of SPIE, Enhanced and Syn

Editor: Jacques G. Verly, Volume 5802-12. 

[16] Federal Aviation Administration: HUD Certification Criteria - Working Paper. P

00085 (Draft), Dated January 13, 2005. 

[17] Bender, E.J., Reese, C.E., and van der Wal, G.S.: Com

Feature-Level Image Fusion Techniques for Enhanced Night Driving. Low-Ligh

Time Imaging Systems, Components, and Applications, C. Bruce Johnson, Divye

A. Laplant

[18] Endsley, M. R.: Situation Awareness in Aviation Systems. In D. J. Garland, J. 

Hopkin (Eds.), Human factors in Aviation Systems (pp. 257-276). Hillsdale

Erlbaum. 1999. 

[19] Simard, P., Link, N.K., and Kruk, R.V.: Feature Detection Performance with Fu

Sensor Images. Pro

pp. 1108-1112, 1999. 

[20] Andre, A.D., Wickens, C.D., and Moorman, L.: Display Formatting Techniq

Situa

No. 3, 1991, pp. 205-218. 

[21] Greig, I., Shaw, G., and Lumsden, B.: Augmented Vision Systems for the All Weather Autonomous

Landing Guidance of Fixed Wing Aircraft: An Investigation of the Key Parameters by Pilote

Flight Simulation. From Proceedings of the 10th CEAS Conference on Free Fligh

to 32-15

[22] Mann, T.L.: Autonomous Landing Guidance Concept - The Effects of Vide

Dynamics on Pilot Performance. Paper presented at the 6th SAE Aerospace B

Engineering Technology Conference, Paper 872390, 1988, pp. 131-138.

[23] Schnell, T.; Ellis, K.; and Etherington, T.: Flight Simulator Evaluation Of An In

And Enhanced Vision System For Terrain Avoidance. The 24th Digital A

Conference, DASC 2005. Volume 1, 3

[24] Bailey, R.E., Parrish, R.V., Kramer, L.J., Harrah, S.D., and Arthur, J.J. (Tre

Challenges In the Devel

RTO SET Workshop on “Enhanced and Synthetic Vision System”, held in 

September 2002, published as NATO RTO-MP-107. 
56

Head Down: The 

nd Visual Clutter On Cue Effectiveness For Display 

, & French, G.A.: Human Factors In Head-Up Synthetic Vision Displays. In proceedings 

ety of Automotive 

s, P. M., & Wickens, C. D.: Head-Up Displays: Effects Of Clutter, Display Intensity, And 

ology, 8(4), 377-

sing A Head-Up 

nd Display Type. Air Force Research Laboratory, Human 

Displays. IEEE Engineering in 

ckens, C. D., & Hardy, T. J.: Synthetic vision systems: The effects of guidance 

hway Design Effects On Synthetic Vision 

3, Editor: Jacques 

ystem for Seeing 

dium: Human Perception and 

dures for Flight 

ceedings of SPIE, 

s G. Verly, Vol. 5424, pp. 89-97. 

 Operations with 

. In 

rly, Vol. 5081, pp. 

s and HUD Scaling for V/STOL Terminal Area 

[41] Kramer, L.J., Prinzel III, L.J., Arthur III, J.J., & Bailey, R.E. (2005). Advanced Pathway Guidance 

Evaluations On A Synthetic Vision Head-Up Display. NASA Langley Research Center: NASA 

Technical Paper TP-2005-213782. 

[42] Foyle, D.C and Hooey, B.L., Improving Evaluation and System Design Through The Use of OffNominal Testing: A Methodology for Scenario Development. Proceedings of the 12th International 

Symposium on Aviation Psychology, pp. 397-402. Dayton, OH. 

[43] Ames, Lawrence L. & George, Edward J.: Revision and Verification of a Seven-Point Workload 

Estimation Scale. Air Force Flight Technical Center: AFFTC-TIM-93-01, 1993. 

[28] Yeh, M., Merlo J. L., Wickens, C. D. & Brandenburg, D. L.: Head Up Versus 

Costs Of Imprecision, Unreliability, A

Signaling. Human Factors, 45(3), 2003, pp. 390-407. 

[29] Snow, M.P.

of the 2001 Aerospace Congress (ACE2001CD-01-2652). Warrendale, PA: Soci

Engineers (2001) 

[30] Verver

Display Location On Pilot Performance. International Journal of Aviation Psych

403, 1998. 

[31] Snow, M.P., Reising, J.M., & Liggett, K.K.: Flying Complex Approaches U

Display: Effects Of Visibility A

Effectiveness Directorate Report No. AFRL-HE-WP-TR-2000-0011. 2000. 

[32] Ercoline, W.: The Good, The Bad, and The Ugly in Head-Up 

Medicine and Biology, 19, 2000. pp. 66-70. 

[33] Alexander, A. L., Wi

symbology, display size, and field-of-view. Human Factors, 47, 2005. pp. 693-707.

[34] Kramer, L.J., Prinzel, L.J., Arthur, J.J., & Bailey, R.E. Pat

Head-Up Displays. In Proceedings of SPIE, Enhanced and Synthetic Vision 200

G. Verly,, Volume 5424-08. 

[35] Larimer, J., Pavel, M., Ahumada, A. and Sweet, B.: Engineering a Visual S

Through Fog. AeroTech '92, SAE Paper No. 921130, pp. 428-432. 1992.

[36] Boff, K.R. and Lincoln, J.E., (Editors). Engineering Data Compen

Performance. H.G. Anderson Medical Research Laboratory, Ohio. 

[37] Federal Aviation Administration Advisory Circular: Standard Operating Proce

Deck Crewmembers, AC 120-71A, Dated February 27, 2003. 

[38] Lorenz, B. and Korn, B.: Crew Coordination Issues Of EVS-Approaches. In Pro

Enhanced and Synthetic Vision 2004, Editor: Jacque

[39] Tuttle, T.T., Imrich, T., Etherington, T.J., and Theunissen, E.: Terminal Area

Enhanced and Synthetic Vision: Experience in the Boeing Technology Demonstrator

Proceedings of SPIE, Enhanced and Synthetic Vision 2003, Editor: Jacques G. Ve

136-145. 

[40] Merrick, V.K. and Jeske, J.A., Flightpath Synthesi

Operations. NASA Technical Memorandum 110348, April 1995. 
57

o Human Performance Measurement. 

. Proceedings 

 Performance for 

. 

uation 

Awareness And RNP Capabilities For Terrain-Challenged Approaches. Proceedings of the 

American Institute of Aeronautics and Astronautics Third Aviation Technology, Integration, and 

Operations Technical Forum, AIAA 2003-6814, 1-11, 2003. 

[44] American Institute of Aeronautics and Astronautics: Guide t

Washington D.C.: American Institute of Aeronautics and Astronautics. 1993. 

[45] Vidulich, M.A. & Hughes, E.R.: Testing A Subjective Metric Of Situation Awareness

of the Human Factors & Ergonomics Society, 35, 1307-1311, 1991. 

[46] RTCA: Minimum Aviation System Performance Standards: Required Navigation

Area Navigation. RTCA DO-236A. Washington, D.C.: RTCA, Incorporated, 2000

[47] Kramer, L.J., Prinzel, L.J., Bailey, R.E., & Arthur, J.J.: Synthetic Vision Enhances Sit
58

Appendix A: Head Tracker Data Analysis Details 

F) was using and 

 for this. For this 

e reliably used for significant spatial location differences; that 

is, whether the pilot was looking forward (i.e., looking out the window or the head-down displays), 

le ANOVAs of the percentage of time that the PNF’s head was directed at either 

the AD, out t data are shown as 

a function of mean percentages are 

le 4 in the tex

Table age l Time at the AD Concepts for each Altitude Segment 

Altitude Segment es df Mean Square F Significance 

A head-tracker was used in an attempt to quantify what display the pilot not flying (PN

when. Because an eye-tracker was not used, the head-tracking data are only an estimate

reason, the head-tracker data could only b

looking at the AD (outboard of the PNF), or neither. 

In this appendix, the head-tracker statistical data analysis is detailed. 

In Table A-1, the simp

he window/head-down display (OTW/HDD), or elsewhere is shown. The 

the aircraft altitude (i.e., Above Field Elevation (AFL), in feet). (The 

shown in Tab t.) 

A-1. ANOVAs on Percent Dwel

Sum of Squar

AFL > 1000 Between G 9848 roups 5 .188 3 19949.396 101.631 0.000 

Within Groups 264799.155 1349 196.293 

Total 32464 44 7.3 1352 

500 < AFL < 1000 Between Groups 63680.988 3 21226.996 29.231 0.000 

Within Groups 979616.097 1349 726.179 

Total 10432 97.085 1352 

400 < AFL < 500 Between G 4004 roups 4 .709 3 14668.236 14.692 0.000 

Within Gro 1346777.572 ups 1349 998.353 

Total 13907 82.281 1352 

300 < AFL < 400 Between Groups 39969.357 3 13323.119 18.876 0.000 

Within Groups 952177.825 1349 705.840 

Total 992147.183 1352 

200 < AFL < 300 Between G 6085.649 roups 3 2028.550 4.808 0.002 

Within Groups 569158.532 1349 421.911 

Total 575244.181 1352 

100 < AFL < 200 Between Groups 2895.730 3 965.243 2.914 0.033 

Within Groups 443897.097 1340 331.266 

Total 446792.827 1343 

AFL < 100 Between Groups 813.379 3 271.126 .664 .574 

Within Groups 520396.461 1274 408.474 

Total 521209.840 1277 
59

Table A-2, when 

oncept. The four 

ibited on formats 

on (“fused”). The 

logy moreso than 

 the FLIR did not 

mation above 1000 ft AFL.) For instance, the FLIR-Tunnel configuration 

induced m bology configuration (terrain 

information but no symbology). 

 Student-Newman-Keuls st for A tude Se : > 1000 ft AFL 

Subset for α=0.05 

Post-hoc multiple comparison tests (SNK, α=0.05) revealed 4 subsets, shown in 

ownship was above 1000 ft AFL, based on percentage of dwell time for each AD c

unique subsets for percentage of dwell time indicate that more dwell time was exh

having symbology, and those formats having synthetic and enhanced vision informati

most interesting comparison is that the pilots focused on display concepts with symbo

those with terrain information (SV or EV). (Given the simulated weather conditions,

provide any terrain infor

ore gaze on the part of the PNFs than the Fused-No Sym

Table A-2. Te lti gment

Aux. Display N 

1 2 3 4 

FLIR – No Symb y olog 339 4.757 

Fused – No Symbology 354 12.962 

FLIR - Symbology 309 15.556 

Fused - Symbology 351 23.213 

Post-hoc tests (SNK, α=0.05) revealed 4 subsets, shown in Table A-3, when ownship was above 500 ft 

but less than cept in this altitude band, 

the same four unique sub ercentage of dwell times 

indicate again dwell e was exhib ts having symbology, and those formats 

having synthetic and enhanced vision information (“fused”). The pilots focused on display concepts with 

symbology m ation (SV or EV). 

uls Test for Altitu gment: 500 < AFL < 1000 ft 

Subse

1000 ft AFL. Based on percentage of dwell time for each AD con

sets - and order - were found as previous. The p

, that more tim ited on forma

oreso than those with terrain inform

Table A-3. Student-Newman-Ke de Se

Aux. Display N t for α=0.05 

1 2 3 4

FLIR - No Symbology 339 17.796 

Fused - No Symbology 354 22.340 

FLIR - Symbology 309 30.324 

Fused - Symbology 351 35.330 

As shown in Table A-4, post-hoc tests (SNK, α=0.05) revealed just two unique subsets for percentage of 

dwell time in the altitude band of 400 ft AFL to 500 ft AFL. More dwell time was exhibited on formats 

having symbology than those without. 
60

Table A-4. Student-Newman-Keuls Test for Altitude Segment: 400 < AFL < 500 ft 

Aux. Display N Subset for α=0.05 

1 2 

Fused - No Symb ology 354 24.292 

FLIR - No Symb y olog 339 24.330 

FLIR - Symbology 309 35.519 

Fused - Symbology 351 35.891 

Post-hoc tests (SNK, α=0.05) revealed just two unique subsets for percentage of dwell time in the altitude 

band of more dwell 

time was exhibited on ving sy ology hout. 

Table uls Test for Altitu ment: 300 < AFL < 400 ft 

Subset for α=0.05 

 300 ft AFL to 400 ft AFL, in Table A-5. Identically to the previous altitude band, 

 formats ha mb than those wit

A-5. Student-Newman-Ke de Seg

Aux. Display N 

1 2 

FLIR - No Symbology 339 14.934 

Fused - No Symbology 354 15.003 

FLIR - Symbology 309 24.564 

Fused - Symbology 351 26.764 

For the Table A-6, (SNK, α=0.05) 

revealed two overlapping dwell time. The fused/symbology AD format had 

statistically significan aps not o rationa ) more dwell time than the 2 AD concepts 

without symbology. Again, more dwell time nded toward formats having symbology than those 

without. 

Table ewma euls Test for Alt egment: 200 < AFL < 300 ft 

Subset for α=0.05 

altitude band from 200 to 300 ft AFL, post-hoc tests, shown in

subsets for percentage of

t (but perh pe lly significant

te

A-6. Student-N n-K itude S

Aux. Display N 

1 2 

FLIR - No Symbology 339 8.466 

Fused - No Symbology 354 10.358 

FLIR - Symbology 309 11.531 11.531 

Fused - Symbology 351 14.252 

For the altitude band from 100 to 200 ft AFL, post-hoc tests (SNK, α=0.05), shown in Table A-7, 

revealed two overlapping subsets for percentage of dwell time. The FLIR/No symbology AD format had 

statistically significant (but perhaps not operationally significant) less dwell time than the 2 AD concepts 

with symbology. 
61

Table A-7. Student-Newman-Keuls Test for Altitude Segment: 100 < AFL < 200 ft 

Aux. Display N Subset for α=0.05 

1 2 

FLIR –No Symbology 336 5.938 

Fused - No Symb y olog 351 7.367 7.367 

Fused – Symbology 351 9.246 

FLIR – Symbology 306 9.599 

Below 100 ft AFL, post-hoc tests (SNK, α=0.05) showed no statistically-significant differences between 

display configurations. 
62

Appendix B. Final Questionnaire Subject Ratings and Comments 

es 

ht crews are presented. The questions are written in italics. 

In the following, the responses and subsequent statistical analyses to the post-test questionnair

administered to the flig

Display Concept Questions: 

1. On a scale of 1 to 7, please rate the effectiveness of each concept in terms of conducting an 

approach in IMC as the Pilot Flying (from the left seat with the HUD). 

Borderline

1234567

An ANOV

Completely

Ineffective

Completely 

Effective

A revealed that the HUD display concepts were significantly different (F(3, 63) = 15.96, 

p<0.0001) for their effec veness in conducting an a the PF. Post-hoc tests (SNK, 

α=0.05) revealed ts. Pilots we conduct an approach in IMC with all of 

the HU rated the Fused HUD-Tunnel concept as being the most effective of the 4 

tested

Mean ject Ratings 

ti pproach in IMC for

 3 unique subse re able to effectively

D concepts but they

. 

 Sub

Display Concept Subset 1 Subset 2 Subset 3 

HUD EVS without tunnel 5.0 

HUD EVS with tunnel 5.7 

HUD Fused EVS&SVS without tunnel 5.7 

HUD Fused EVS&SVS with tunnel 6.6 

Seven pilots provided comments. With SVS, the horizon is inertially stabilized

improvement in terms of conducting an approach in IMC. The HUD fused im

 which is a major 

age (SVS/EVS) provides 

the best situation awareness and the tunnel provided the most precise path control during the approach. 

However once stabilized on approach, the tunnel seems to add clutter and detract from EVS and threshold 

light acquisition capability. Four out of the 7 pilots commented that fused HUD with tunnel provided the 

greatest SA. One pilot commented that he found the tunnel only useful for turn anticipation cues. 

2. On a scale of 1 to 7, please rate the effectiveness of each concept in terms of conducting an 

approach in IMC as the Pilot Not Flying (from the right seat with the Auxiliary Display). 

Borderline

1234567

Completely

Ineffective

Completely 

Effective
63

p<0.0001) in 

NK, α=0.05) 

aled 3 unique subsets. The PNFs were able to effectively conduct an approach in IMC with all the 

AD concepts but they rated the Fused AD – With Symbology concept as being the most effective of the 4 

tested. 

Mean S ect Ratings 

An ANOVA revealed that the AD display concepts were significantly different ((F(3, 60) =, 

their effectiveness when conducting an approach in IMC as the PNF. Post-hoc tests (S

reve

ubj

Display Concept Subset 1 ubset 2 Subset 3 S

AD with EVS only 4.5 

AD with EVS and symbology 5.0 

AD with Fused EVS/SVS only 5.3 

AD with Fused EVS/SVS and symbology 6.0 

Eight pilots added comments. Would like to see wind vector and bank angle indicato

symbology and symbology should be the same as that found on HUD for crew resource

(CRM) correlation. Symbology helped with situation awareness and CRM. PNF’s hig

back u

r included in AD 

 management 

hest priority is to 

p the PF by cross-referencing position, confirming EVS acquisition, and confirming threshold 

acquisition; all while communicating in a concise standardized format that builds situation awareness. 

on glideslope, EVS 

logy enhanced CRM 

symbology blocked the image he was most interested 

in seeing. 

3. On a scale of 1 to 7, please rate the overall effectiveness of each concept for flight path awareness 

and performance as you performed the evaluations as the Pilot Flying (from the left seat with the 

HUD). 

SVS on the AD is nice for overall SA but not essential on the AD. Once established 

and symbology are the most effective elements on the AD. Fused image with symbo

and SA for the PNF. One pilot commented that the 

Borderline

Completely

Ineffective

Completely 

Effective

1234567

An ANOVA revealed that the HUD display concepts were significantly different (

p<0.0001) in their effectiveness for the PF’s flight path awareness and performanc

(SNK, α=0.05) revealed 3 unique subsets. All display concepts were effective fo

F(3,63) =19.50, 

e. Post-hoc tests 

r flight path awareness 

and performance but they rated the Fused HUD-Tunnel concept as being the most effective of the 4 

tested. 

Four pilots provided comments. One pilot felt there was considerable excess information on all HUD 

concepts and that a minimum symbology set of AD on the HUD would be a vast improvement. The 

tunnel can become “cluttersome” at times (e.g., once wings-level on final), but it definitely assists in 

anticipating what is approaching in the flight path course (e.g., turns). Flight director and flight path 

marker are the best symbolic tools; while, the SVS/EVS gives a perception of reduced workload and 

easier correction and detection of deviations. 
64

Mean Ratings 

Display Concept Subse 1 Subset 2 Subset 3 t

HUD EVS without tunnel 4.7 

HUD EVS with tunnel 5.8 

HUD Fused EVS&SVS without tunnel 5.5 

HUD Fused EVS&SVS with tunnel 6.5 

4. On a scale of 1 to 7, please rate the overall effectiveness of each concept for terrain and obstacle 

awareness as you performed the evaluations as the Pilot Flying (from the left seat with the HUD). 

Borderline

1234567

Ineffective Effectiv

Completely ompletely 

e

An ANOVA revealed that the HUD display concepts were significantly different (F(3,63) =24.06, 

p<0.0001) in their effectiven s terrain ess. Post-hoc tests (SNK, 

α=0.05) revealed 2 un errain and obs aren very effectively provided by the 

Fused HUD ) but the FLIR-only HUD concept mean=4.0) were rated on average as 

being only e. 

Mean Subject Ratings 

C

ess for the PF’ and obstacle awaren

ique subsets. T tacle aw ess was

 concepts (mean=6.2 s (

 borderline effectiv

Display Concept Subset 1 Subset 2 

HUD EVS without tunnel 3.8 

HUD EVS with tunnel 4.1 

HUD Fused EVS&SVS without tunnel 6.0

HUD Fused EVS&SVS with tunnel 6.4 

Six pilots commented. Inertial stabilized horizon and pitch scale improve pilot output in high gain tasks. 

ed last minute awareness of terrain and obstacles on very short final approach but by this time 

as focused on approach light and runway environment acquisition. Tunnel did not 

ess and SVS is most effective for terrain and obstacle awareness. With or without 

tunnel, on short final to landing, the pilot is so focused on landing that it is difficult to see any obstacles 

with EVS and SVS. 

5. What features did you like the best on the HUD display concept? Which features did you like the 

least on the HUD display concept? 

Best features:

EVS allow

t p he ilot’s attention w

affect terrain awaren

• Declutter option. 

• Tunnel (10 pilots). 
65

y safer approaches 

and guidance cue (2 pilots). 

g out SVS at 500 feet; ability to see 

unway and landing environment in EVS; SVS/EVS workload reduction turning to final; HUD 

tor good cross check for positional error. 

• Fused EVS/SVS (3 pilots) makes it easier to fl

• Tunnel with flight path marker 

• Flight data, synthetic terrain, and tunnel. 

• Fusion with no tunnel. 

• Flight director and flight path marker on the HUD; phasin

r

runway outline and flight direc

• SVS

• Flight path marker with pitch reference. 

Worst features: 

• Inability to adjust HUD symbology brightness independently (3 pilots). 

ermine if you 

h. (3 pilots) 

d indicated at DA +100 feet. 

lified (2 pilots). 

s level and < 1000 ft Above 

n from SVS to EVS later in approach (~350 ft AGL) 

• Large size of display – information was too spread out for precision instrument task. 

6. On a scale of 1 to 7, please rate the overall effectiveness of each concept for flight path awareness 

and performance as you performed the evaluations as the Pilot Not Flying (from the right seat with 

the Auxiliary Display). 

• Didn’t like having to declutter HUD or look around HUD at minimums to det

should continue approach or go-around as this is a critical time in the approac

• Need provision for barometric decision altitude to be set an

• Too much symbology on the HUD, needs to be simp

• Remove tunnel once localizer and glideslope are captured or at wing

Ground Level (AGL) (2 pilots). 

• HUD symbology is too bright – obstructs threshold acquisition (3 pilots). 

• Have transitio

Borderline

1234567

Completely

Ineffective

Completely 

Effective

An ANOVA revealed that the AD display concepts w ferent (F(3, 63) = 24.3, 

p<0.0001) for the ath awareness and performance. Post-hoc tests (SNK, α=0.05) revealed 

3 un e AD concept with Fused VS bolog e most 

effective of the 4 tested. Si ilarly, the AD with EV ym and the AD concept with Fused 

EVS/SVS (but without ) were somewhat effective for the monitoring pilot’s flight path 

awar EVS was rated as being borderline to marginally ineffective for the 

PNF’ rmance.. 

Mean Subject Ratings 

ere significantly dif

 PNF’s flight p

ique subsets. Th EVS/S

S and s

and msy y was rated as being th

m bology

 symbology

eness. An AD concept with only

s flight path awareness and perfo

Display Concept Subset 1 Subset 2 Subset 3 

AD with EVS only 3.4 

AD with EVS and symbology 4.9 

AD with Fused EVS/SVS only 4.7 

AD with Fused EVS/SVS and symbology 6.0 
66

feet AGL, EVS and 

e. PNF mission is to back-up PF performance with AD symbology and to 

1 to 7, please rate the overall effectiveness of each concept for terrain and obstacle 

awareness as you performed the evaluations as the Pilot Not Flying (from the right seat with the 

Auxiliary Display). 

Three pilots commented. Fused is fine but once wings-level on final less than 500 

symbology is most effectiv

clear runway of obstacles. 

7. On a scale of 

Borderline

1234567

An ANOVA revealed that the

Completely

Ineffective

Completely 

Effective

 AD display concepts were significantly different (F(3,63) =24.06, 

p<0.0001) for their effectiveness for the non-flying pilot’s terrain and obstacle awareness. Post-hoc tests 

(SNK, α=0.05) revealed 2 unique su y d obstacle awareness was very 

effective (mean=5.9) when using the Fused AD conc wer rderline effective with the 

FLIR-onl mean=3.9). 

n Subject Ratings 

bsets. A non-fl ing pilot’s terrain an

epts but e only bo

y AD concepts (

Mea

Display Concept Subset 1 Subset 2 

AD with EVS only 3.6 

AD with EVS and symbology 4.1 

AD with Fused EVS/SVS only 5.6 

AD with Fused EVS/SVS and symbology 6.2 

ts commented. Having a reliable 3D view, with or without symbology, is hard to beat. Fused 

st EVS and 

on the PNF Auxiliary Display concept? Which features did you 

uxiliary Display concept? 

Three pilo

EVS and SVS for terrain and obstacle detection. Once wings level on glide path, ju

symbology. 

8. What features did you like the best 

like the least on the PNF A

Best features:

• Uncluttered presentation of symbology (4 pilots). Helpful in effective mon

flying. 

itoring of the pilot 

• Symbology, fused (2 pilots). 

• SVS display for situation awareness. 

• Used as VMC cross-check of instruments 

• Fused SVS and EVS (4 pilots) 

• Fused above 500 ft AGL, EVS only on landing for situation awareness and safety, 

• Knowledge of what pilot is seeing. 

• Uncluttered display of EVS and symbology in acquiring EVS lights and threshold lights and 

instant flightpath awareness on final approach (2 pilots). 
67

r-coding (“greening”) of imagery when EVS and SVS agree. 

• Pilot’s ability at will to select fused or EVS only. 

• Colo

Worst features: 

• Location – should be in pilot’s normal scan as moving head back and forth from side to front 

k angle to symbology. 

ss-check. 

9. For each display concept, please indicate the workload as you performed the evaluations as the 

Pilot Flying (from the left seat with the HUD). 

could cause spatial disorientation (9 pilots). 

• Add wind vector and ban

• No symbology condition (3 pilots) as had to scan flight instruments for cro

• Inability to declutter symbology. 

Moderate

1234567

Very

Low

Very

High

An ANOVA revealed that the HUD display concept different (F(3,63) =10.19, 

p<0.0001) as they affect PF workload. Post-hoc tests (SNK, α=0.05) revealed 2 unique subsets. Pilots 

workload w mean=3.6) when using the tunnel HUD concepts but was between 

moderate a h with the non-tunnel HUD concepts (mean=4.6). 

Mean Subject Ratings 

s were significantly

as moderately low (

nd moderately hig

Display Concept Subset 1 Subset 2 

HUD EVS without tunnel 4.8 

HUD EVS with tunnel 3.7 

HUD Fused EVS&SVS without tunnel 4.3 

HUD Fused EVS&SVS with tunnel 3.5 

Six pilots commented. Stabilized horizon from SVS helps keep workload from increasing during highgain tasks. Tunnel added workload due to its high precision but definitely raised SA. 

10. For each display concept, please indicate the workload as you performed the evaluations as the 

Pilot Not Flying (from the right seat with the Auxiliary Display). 

Moderate

1234567

Very

Low

Very

High

An ANOVA revealed that the AD display concepts were significantly different (F(3,63) =5.91, p=0.001) 

for PNF workload. Post-hoc tests (SNK, α=0.05) revealed 2 overlapping subsets. The AD FLIR with no 
68

d the highest workload of the 4 concepts tested but it was still rated moderately low by the 

non-flying pilots. 

 Subject Ratings 

symbology ha

Mean

Display Concept Subset 1 Subset 2 

AD with EVS only 3.6 

AD with EVS and symbology 2.8 

AD with Fused EVS/SVS only 3.2 3.2 

AD with Fused EVS/SVS and symbology 2.7 

Five pilots commented. Workload was lower with EVS only as there wasn’t much to lo

final. Symbology was difficult to look through on final. Workload was increased with FLIR 

ok at until short 

only as pilot 

had to continuously check when FLIR could be seen. Increased workload due to choosing alpha and beta 

 lights call by 200 feet. SVS reduced discrimination and added clutter 

to the AD. AD presentation provided comfort of visual approach. 

11. Please rate the essentialness of having a the tunnel concept depicted on the HUD in the presence of 

raster information (either SVS or EVS) 

values but that choice is a good thing. Decluttered HUD symbology on AD was very effective for 

helping the pilot flying get the EVS

Borderline

Completely

Inessential

Completely 

Essential

1234567

, pilots felt that it 

. 

nnel 

provements. 

, especially using it with the guidance 

cue and flight path marker, was a very useful concept. With the tunnel, pilots were able to anticipate 

nderstand the future track of the aircraft and it increased their ability to 

el was useful but not an essential element of 

y to turn the tunnel off; to increase the 

tunnel spacing to 0.2 or 0.3 nmi from the current 0.1 nmi separation; and making the tunnel symbology 

dimmer than the other symbology. Two of the pilots felt that if turn anticipation cues could be provide by 

other HUD symbology than the banking tunnel elements that the tunnel would not be as necessary. 3 

pilots felt that at times the tunnel added too much clutter to the HUD. 

Auxiliary Display Symbology Questions

The mean “essentialness” rating: was 5.0 with a standard deviation of 1.6. On average

was essential to include a tunnel on a HUD in the presence of EV and/or SV imagery

Please provide any comments with regard to how the tunnel concept was displayed. Is this tu

concept a useful one? If so, how? If not, please provide a reason and any suggested im

Eighteen pilots commented. Twelve of the pilots felt that the tunnel

upcoming course changes and u

scan other instruments. Two of the pilots felts that the tunn

the HUD symbology set. Pilot suggestions included the abilit

12. Please rate the effectiveness of the auxiliary display symbology for: 
Borderline

1234567

Completely

Ineffective

Completely 

Effective

e fligh eviation of 1.9. 

on of 1.7. 

dard Deviation of 1.9. 

areness of future 

f flight path errors and recognition of obstacles. 

13. Please rate the essentialness of having symbology depicted on the auxiliary display in the presence 

of raster information 

a) Awareness of futur t path: Mean Rating of 4.23, Standard D

b) Awareness of flight path errors: Mean Rating of 4.95, Standard Deviati

c) Recognition of obstacles: Mean Rating of 4.77, Stan

On average, pilots rated the symbology on the AD as being somewhat effective for aw

flight path and effective for awareness o

Borderline

1234567

Completely

Inessential

Completely 

Essential

viation 1.7. On 

 highly essential. 

s symbology set a 

ents. 

n the AD and that a 

 that the minimum 

anding any perception problems the PF might be 

he PNF’s job is to monitor the approach both from an aircrew performance 

standpoint and a systems performance standpoint. In order to have effective crew resource management 

symbology used on the 

ded just the right 

the capability of the 

elt that the AD should be mounted more in the forward view. Only one pilot commented 

that there was no usefulness in having symbology on the AD. 

Runway Incursion Questions

The mean rating for the “essentialness” of AD symbology was 5.45 with a standard de

average, pilots rated having symbology on the AD with imagery as being essential to

Please provide any comments with regard to how the symbology was displayed. Is thi

useful one? If so, how? If not, please provide a reason and any suggested improvem

18 pilots commented. In general, pilots felt that it was essential to have symbology o

pilot-selectable declutter capability should be available. Pilot commentary indicated

symbology set was absolutely necessary in underst

encountering and that t

and data correlation, pilots felt that the AD symbology should be a subset of the 

PF HUD. Pilots commented that the current AD symbology set was uncluttered, provi

amount of information, and was easy to reference positional error while optimizing 

EVS. Pilots f

The FAA (2006) defines runway incursion as, "any occurrence in the airport runway environment 

involving an aircraft, vehicle, person, or object on the ground that creates a collision hazard or results in 

a loss of required separation with an aircraft taking off, intending to take off, landing, or intending to 

land." 

69
70

ion awareness of 

es on the runway as you performed the evaluations as the Pilot Flying (from the left 

seat with the HUD). 

14. On a scale of 1 to 7, please rate the effectiveness of the display concept for situat

aircraft/obstacl

Borderline

1234567

Pilot comments indicated that in general they did not see either obstacle on the HUD

They commented that since they were required to use their natural vision to see the run

that they were not using the EVS imagery to see the fire truck or baggage care late in 

was suggested that part of the Standard Operating Procedures (SOPs) should be for th

AD to check for obstacles on the runway after the PF has been assured of the EVS

Completely

Ineffective

Completely 

Effective

 in the EVS imagery. 

way environment 

the approach. It 

e PNF to use the 

 lights and raw visual 

he PF is in 

hdown scan. Pilots also 

arker and guidance cue) might have been occluding 

the obstacles but that the tunnel did not as it was decluttered from the symbology by 500 feet AGL. 

15. On a scale of 1 to 7, please rate the potential effectiveness of the HUD display concept for 

prevention of runway incursions in real-world operations. 

of the runway environment. In essence, this would become a major function of PNF because t

a difficult visual transition from short sight HUD scan to far sight runway-touc

commented that the HUD symbology (flight path m

Borderline

12345

Completely

Ineffective

Completely 

Effective

6 7

ould be dimmer and that the symbology might 

also 

other pilot 

t these elements 

nd/or 

Pilots suggested that having the symbology dimmer and the EVS returns brighter would help detect a 

runway incursion. They also commented the flight path marker and guidance cue tend to obscure the 

runway and that HUD tends to reduce their focal scan especially under 100 ft AGL. They commented 

that concentrating on the symbology and raster image might distract them from detecting runway 

incursions. 

17. On a scale of 1 to 7, please rate the effectiveness of the display concept for situation awareness of 

aircraft/obstacles on the runway as you performed the evaluations as the Pilot Not Flying (from the 

right seat with the Auxiliary Display). 

Pilots commented that maybe the HUD symbology sh

require more of their attention so that they have less attention to detect obstacles. Other pilots 

commented that seeing obstacles is probably going to be a function of the PNF. An

commented that with training and proper fusion algorithms for the EVS and SVS tha

could be effective tools in preventing mishaps caused by runway incursions. 

16. While making the approaches as the Pilot Flying, please comment on the effect of symbology a

raster on the Head-Up Display during a runway incursion? 
71

Borderline

1234567

Pilots commented that late in the approach the PNF is concentrating out the front of t

at the AD is not “natural”. In general, the pilots did not look at the AD below 500 fee

Two

Completely

Ineffective

Completely 

Effective

he plane and looking 

t on the approach. 

 different training/SOP techniques were offered by the pilots: 1) have the PNF assure “EVS lights” 

a "clear 

ts" is announced 

18. On a scale of 1 to 7, please rate the potential effectiveness of the PNF Auxiliary Display concept for 

prevention of runway incursions in real-world operations. 

and "landing" calls by PF and then have the PNF scan the FLIR down the runway and require 

deck" call if RVR<4000; or, 2) require PNF to be head-up and forward after "EVS ligh

by PF. 

Borderline

1234567

Completely ompletely 

per procedures for the PNF that the AD could 

be a great advancement for runway incursion protection. Pilots also suggested that the AD be mounted in 

clutter the 

pproaches as the Pilot Not Flying, please comment on the effect of symbology 

and/or raster on the Auxiliary Display during a runway incursion? 

PNF needs to be 

mbology could 

prevent detecting an incurring vehicle but was necessary to monitor the PF’s flying of the approach. One 

pilot thought this was the greatest benefit in the whole experiment as the PNF can use the tool to scan the 

touchdown zone on short final. 

Head-Up Display Transition Questions

Ineffective Effective

C

Pilots commented that with training and development of pro

a forward location and that there be a way to adjust the symbology brightness and to de

symbology when needed. 

19. While making the a

Pilots commented that the AD needs to be mounted in a forward location and that the 

able to declutter the symbology when necessary. Pilots commented that the presence of sy

20. Please rate the effectiveness of the HUD SVS/EVS transition that lasted 6 seconds and was finished 

at 500 feet. Please comment on this transition strategy and point. 

Borderline

1234567

Completely

Ineffective

Completely 

Ineffective
72

ots rated the 

e to highly effective. 

sition altitude: 1) 

uggested that 1000 

inted out that 

ainting a picture 

ested that the 

ltitude should finish at a lower altitude than 500 feet (350 feet was suggested); and, 3) Three of 

the pilots liked the transition altitude strategy employed in the experiment. One pilot suggested removing 

ay environment with 

If you like the gradual transition as a concept. What duration would you use (in seconds) and at 

) ______ Other ______ 

t for transitioning from SVS to EVS on the HUD? 

tion and altitude: 

1000 ft AFL 

 500 ft AFL 

 1000 ft AFL 

e 500 ft AFL 

arting at 500 ft and concluding at 400 ft AFL 

f 500 ft AFL 

 500 ft AFL 

 500 ft AFL 

400 or 300 ft AFL 

• duration of 6 seconds with transition point of 500 ft AFL 

• duration of 6 seconds with transition point of 500 ft AFL 

• duration of 10 seconds with transition point of 500 ft AFL 

The majority of pilots (11/20) wanted the duration time to be similar to that tested in the experiment (4-6 

seconds); six of pilots wanted a shorter duration time (2-4 seconds) for the transition; two pilots wanted a 

greater duration time of greater than 6 seconds; and 1 pilot thought above 1000 ft there should be no 

transition as long as there was good correlation between the EVS and SVS. With regard to the transition 

altitude, again the majority of pilots (12/20) thought the tested transition altitude of 500 ft was 

The meand “effectiveness” was 5.38 with a standard deviation of 1.4. On average, pil

SVS/EVS transition that lasted 6 seconds and was finishes at 500 feet to be effectiv

Ten pilots commented. The pilot opinions varied with regard to the SVS/EVS tran

Three of the pilots thought the transition altitude should be higher than 500 ft. They s

ft AGL, the typical stabilized approach altitude, might be appropriate. Another pilot po

removing the SVS image at 1000 ft might not be a good idea as the FLIR might not be p

of the runway environment yet due to atmospheric conditions; 2) Two of the pilots sugg

transition a

the FLIR image between 200 ft and 100 ft AGL to force the pilot to acquire the runw

natural vision. 

21.

what altitude (in ft AFL) would you initiate (or conclude) the transition? 

Duration (sec) ______ Transition point (feet AFL

22. Is the height above touchdown an appropriate poin

Here are the pilot suggested SVS/EVS transition dura

• duration of 3 seconds with transition point of 

• duration of 2 seconds with transition point of 500 ft AFL

• duration of > 6 seconds with transition point of <200 ft AFL 

• duration of 6 seconds with transition point of 350 ft AFL 

• duration of 6 seconds with transition point of

• duration of 2 seconds with transition point of

• duration of 6 seconds with transition point finishing befor

• duration of 5 seconds with transition point of 700 ft AFL 

• duration of 5 seconds with transition point st

• duration of 6 seconds with transition point o

• duration of 3 seconds with transition point of

• duration of 6 seconds with transition point of

• duration of 4 seconds with transition point of 

• Fuse any time there is good correlation with transition point at 1000 feet. 

• duration of 3 seconds with transition point of 500 ft AFL 

• duration of 6 seconds with transition point of 500 ft AFL 

• duration of 3 seconds with transition point of 500 ft AFL 
73

 from 700-1000 ft 

than 500 ft (values ranged from <200 ft to 400 ft). 

appropriate; three pilots wanted a transition altitude greater than 500 ft (values ranged

AFL; and three pilots wanted a transition altitude less 

Auxiliary Display Fusion Control Questions

23. Describe your strategy for using the Alpha (spatial compatibility) and Beta (percent of EVS/SVS 

ave a fused image 

fter this altitude to 

ts want max alpha 

nted alpha fixed at 5 

 segments and then 

 500 ft AFL 

rotection. 4 pilots 

x alpha and max beta and one of the four commented that it helped with differentiating 

VS fusion on the Auxiliary 

commented. Four of them wanted the AD moved to the forward panel. In 

e pilot recommended a dual HUD 

still liked the AD set-up and 

functionality as tested in this experiment. 

Database Integrity Questions

fusion) controls on the Auxiliary Display 

Seventeen out of 22 pilots commented. The majority of pilots (9 out of 17) wanted to h

until 1000-500 ft AFL (range of values given by pilots) and then a pure FLIR image a

help obtain the approach lights and to make sure the runway was clear. 8 of the 9 pilo

above the FLIR only transition point. The other pilot (of the 9 that wanted fusion) wa

(mid range) always and beta to be 5 (mid range) for initial and intermediate approach

on 8 (highest value) from 1000 ft AFL to 500 ft AFL after which he wanted pure FLIR from

to landing. One pilot felt that FLIR under 100 ft AFL was key for runway incursion p

wanted ma

between objects and with depth perception. 3 of the pilots wanted mid-range values of alpha and beta. 

24. Can you suggest any changes or improvements for controlling the EVS/S

Display? 

Eleven out of 22 pilots 

general, the pilots like the fusion control implementation. On

configuration in the cockpit instead of an AD for the right seat but 

25. Please indicate your level of agreement with the following statements: 

Neutral

1 2 3 4 5 67

Disagree

Strongly

Disagree Agree Agree

Slightly

Disagree

Slightly

Agree

a) The e HUD to d and cor for the gross database offset errors 

of 50 y? 

Mean Subject Ratings 

Strongly

re is enough information on th etect rect

 feet and 75 feet, respectivel

Display Concept 50 feet 75 feet 

HUD EVS without tunnel 5.0 5.6 

HUD EVS with tunnel 5.0 5.6 

HUD Fused EVS&SVS without tunnel 5.2 5.8 

HUD Fused EVS&SVS with tunnel 5.3 5.9 
74

 to detect and 

at they could detect 

ean=5.1) and agreed that they could detect and correct the 75 ft offset 

b) There is enough information on the PNF Auxiliary Display to detect and correct for the gross 

y to detect and correct 

either a 50 ft or 75 ft database error. On average, pilots slightly disagreed that they could detect and 

correct the 50 ft offset (m were b reement/disagreement that they 

could detect and correct the 75 ft offset (mean=4

Me ubject Ratings 

AN ANOVA revealed no significant HUD concept effects (p>0.05) for the ability

correct either a 50 ft or 75 ft database error. On average, pilots slightly agreed th

and correct the 50 ft offset (m

(mean=5.7) with the HUD concepts. 

database offset errors of 50 feet and 75 feet, respectively? 

AN ANOVA revealed no significant AD concept effects (p>0.05) for the abilit

ean=3.8) and orderline in their ag

.1) with the AD concepts. 

an S

Display Concept 5 et 75 feet 0 fe

AD with EVS only 3.6 3.8 

AD with EVS and symbology 3.7 4.2 

AD with Fused EVS/SVS only 3.8 4.1 

AD with Fused EVS/SVS and symbology 4.0 4.3 

26. If there was a difference in ratings for 50 and 75 feet, please comment on the rea

Thir

son(s) why. 

teen out of 22 pilots commented. In general, pilots stated that it was harder to detect the 50 foot 

offset as compared to the 75 foot offset. Reason for this difference was that the 50 foot error was “subtle” 

foot error was an “attention grabber”. One pilot commented that he really didn’t see either 

the error. Another 

indow monitoring 

AD was said to affect the pilot’s ability to detect and correct offset 

while the 75 

error as his attention was transferred to the actual runway and he mentally queued out 

pilot commented that as the PNF his attention was not focused on the AD but out-the-w

the approach. The location of the 

errors. 

Crew Procedures

The crew procedures you used in the simulator are based on established EVS proc

operating environment. 

27. Are the aural callouts clear and understandable as you perform the approach?

edures used in today’s 

 

louts were clear and 

understandable but some noted that these callouts were different than their airline’s standard operating 

procedures. Two pilots commented that the procedures were reversed from the challenge/response 

philosophy used in their current airline operations and another commented that he’d prefer Cat II/III 

monitored approach procedures. Others commented that the “minimums” call was counterintuitive 

because in today’s operations a “minimums” call requires the pilot to make a decision to land or goaround and not continuing with the runway environment in site as was the case in this test. Pilots 

suggested that an additional call be made to indicate EVS minimums (or 100 ft HAT) at been reached. 

28. Were the aural callouts acceptable or do have any recommendations for improvements? 

Twenty-one out of 22 pilots commented. Twelve of the pilots thought the cal
75

d not be stepping on 

allouts); 2) 

ctions. Pilots suggested 

 of 100 ft HAT since 

F. Pilots 

ed that it was confusing to have 2 decision heights when the DA was 200 ft. They suggested 

” or “continuing to 100 ft” PF call if lights 

Two alternatives for the crew procedures were scripted by evaluation pilots. 

pt 1:

Pilots commented that the callouts were unacceptable because 1) crew callouts shoul

each other and the computer callouts (e.g., have both 500 ft AGL computer and PNF c

repetitive callouts should be removed; and, 3) calls should only be for critical a

that either a computer-callout or a PNF-callout should be added for EVS minimums

this is where the critical decision to continue the landing or go-around is made by the P

comment

replacing the “minimums” call at 200 ft AGL with a “continue

are in sight. 

Scri

PNF Challenge PF Response

"500 ft" "Final Flaps 30, EVS Normal" 

"Approaching Minimums" (No Response) 

"Minimums" "Roger" 

"100 ft" "Landing (EVS lig oing Around" hts)" or "G

Script 2:

Auto Call PF Call PNF Call

“Approaching Minimums” (@300 ft) (No Response) “300 ft” 

”Minimums” (auto minimums call) (No Response) “200 ft” 

“100 ft” (EVS lights) “Lights/Land” “Lights” 

 “Missed Approach” 

29. Are the crew procedures c able rm the approach? 

hirteen out of 20 pilots found e clear but three others thought revision was necessary. 

hese pilots thought that the cr lle ge/Response” procedures and callouts as is 

one in normal instrument proc ons. One suggested procedures script was: 

l

lear and understand as you perfo

T the procedures to b

T ew should use the “Cha n

d edure operati

Auto Cal PNF Call PF Call

“Approaching Minimums” (+100 ft) [Gives Situation R port] (no response) e

 oach “Appr Lights” 

 “No Approach Lights” 

“Lights or Threshold Lights” 

 “Field in Sight” 

“Runway in Sight” 

”Minimums” [Gives Situation Report] (no response) 

“No Lights – Go Around” 

 “Lights Continue” 
76

“100 ft” [Gives Situation Report] “100 ft” 

“Lights” or “Land” or 

“Land” “Go Around” 

based on PNF’s situation report 

30. Were the crew procedures acceptable or do you have any recommendations for improvements? 

Eigh lowing 

reco

ther since it’s a 

say either callout.]. 

UD image to comply 

needs to be a 

n see the lights with 

ked eye. 

all “Alert” at 200 

rument procedure 

rations. 

r cued a response for PF. 

ntrol at 

inimums call at 200 feet and have the PNF make a call at 

8) Crew should not be “stepping” on each other’s callouts and the computer callouts. 

 callouts should be eliminated. 

itude (if he called “EVS 

12) Have PNF call “lights” and PF call “land” or “missed approach”. 

Simulator Fidelity Questions

t out of 20 pilots found the crew procedures to be acceptable as they were. The fol

mmendations to the crew procedures were offered by the pilots: 

1) Change the “Runway in Sight” or “Landing” callouts to either one or the o

required call-out and not a pilot choice. [Current procedures allow pilot to 

2) Either the PNF call real lights in sight so that the PF can switch off the H

with the FAR requirement to actually see the lights using natural vision or there 

better implementation of the control/contrast of the display so that you ca

the na

3) If PF calls “EVS Lights” prior to decision altitude then the PNF should c

feet to remind the PF that he can legally go down to 100 feet HAT. 

4) Use Challenge/Response procedures and callouts as is done in normal inst

ope

5) Amend the callouts so that they are linked to a decision altitude, o

6) Use a “monitored procedure” where the PNF would make the landing (take co

decision height) or call the go-around. 

7) Need to adjust the wording of the m

100 feet. 

9) Repetitive

10) PF should say “continuing to 100 feet” upon reaching the decision alt

lights” earlier in the approach). 

11) Have an automatic callout at 100 feet. 

31. On a scale of 1 to 7, please rate the fidelity of the simulation to real-world operations. Please 

consider whether the OTW realistically depicted actual traffic on xiways and runways during CAT ta

IIIa approaches. 

Borderline

Realistic

1234567

Completely

Unrealistic

Completely 

Realistic

The mean simulation fidelity rating was 5.23 with a standard deviation of 1.1. On average, pilots rated 

the simulator fidelity to be realistic compared to real-world operations. 
77

u think the aircraft/obstacles 

see or approximated the difficulty in real-world operations? 

lighting from the 

t. You would need to view 

lity conditions to 

 as symbology floats. Yoke has significant amount of free play 

rast visually (day/and on FLIR) 

or 

me of the low RVRs the visibility seemed better than 

actual. 

32. If you have FLIR operational experience, on a scale of 1 to 7, please rate the fidelity of the FLIR 

simulation to actual FLIR imagery. Please consider whether the FLIR realistically depicted terrain, 

runways, traffic, obstacles, etc. during instrument approaches. 

You gave a rating of ____. Please explain the reason for the rating. Do yo

were unrealistically difficult to 

Pilot comments included: 

1) Have been in sophisticated simulations before and this is close to those. The 

aircraft to the ground would be a little brighter at the breakout point. 

2) There is not enough sample population to make a realistic assessmen

the same obstacle/traffic several times in the same place with varying visibi

better assess viability of EVS for acquisition response behavior. 

3) Obstacles were reasonably accurate. 

4) Too much computational time

indicating input to control time constants are too high. 

5) Obstacles would have more cont

6) Not too far off from training simulat

7) Weather seemed realistic though at so

8) Very clear, almost lifelike – great fidelity. 

9) Obstacles approximated real-world operations. 

10) Close to real-world for low visibility operations. 

Borderline

Realistic

Completely

Unrealistic

Completely 

Realistic

1234 567

age, pilots 

 imagery. 

y of lights drives the landing decision. 

eems completely realistic 

3) Realistic for only having hot on cold display. Having hot/cold selectable would have improved 

fidelity. 

4) All but obstacles on the runway which need to have some thermal contrast. 

5) Appeared very realistic except there was not an IR signature for the baggage cart or fire truck, 

which it would have. 

6) FLIR display was realistic particularly in the final approach of flight. 

General Comments

The mean FLIR simulation fidelity rating was 6.13 with a standard deviation of 0.86. On aver

rated the FLIR fidelity in the simulator to be very realistic as compared to actual FLIR

You gave a rating of ____. Please explain the reason for the rating. 

Pilots provided the following comments: 

1) Artificial – fidelit

2) FLIR s
78

 comments or suggestions, which may help to improve these synthetic 

ce cue, SVS and 

e very same scope 

 out of the picture as the PNF 

for the go-around 

h have 

ymbology/Tunnel – with ability for PF and PNF to deselect any one feature. 

 for PNF. PNF primary job is to 

 progress (altitudes) 

ent and eyeball 

EVS to visual runway 

symbology. Often it was so 

day of training. 

rations, he estimates 

t landings. 

 roll guidance (similar to 

 than the fly-to command for the flight path marker. 

departure will 

and obstacles are 

rected path was 

ation along with terrain information on one display is a very useful tool as 

orkload of viewing and processing information but overload of 

sents problems for the 

18) SVS is redundant for the PNF if PF has it and could trap the crew into complacency and breaking 

FARs for raw visual requirements. 

19) Symbology on AD is a must – include vertical speed information if possible. 

20) Two HUDs are better than one. 

21) Tunnel and guidance cue a must. 

22) Provision for callouts using programmable radar altitude and barometric altitudes will be needed 

for actual operations. 

23) No SVS for PF approaching 200 feet and no SVS or EVS for PF approaching 100 feet, PNF 

clears runway with EVS less than 100 feet, and PF has full EVS/SVS for flare/landing/rollout. 

Please provide any further

vision/enhanced vision concepts. 

The following comments were offered by the pilots:

1) Put PNF display in front of the PNF above the HSI. 

2) Give the pilots the capability to select levels of declutter. 

3) PF and PNF should be looking at same display depiction (e.g., tunnels, guidan

EVS pictures) so if there is a question between the two they are referencing th

objects. 

4) At around 500 feet, the auxiliary display is almost taken completely

is searching out the front of the airplane for the actual runway and preparing 

procedure if necessary. 

5) Overall good concept and integration for PD. Ideally, PF and PNF would eac

SVS/EVS/S

6) Auxiliary display is only good for snapshot situation awareness

monitor instrument displays for function/malfunction and monitor approach

for callouts. Going head-down to look at auxiliary display requires head movem

focal length shift. 

7) Crew procedures and callouts have significant impact and need to be changed. 

8) Most critical area in need of improvement is the transition from the full 

acquisition by 100 feet HAT. 

9) Provide the capability to dim the intensity and brightness of the HUD 

bright if would mask the EVS lights and/or the runway threshold lights. 

10) Like the tunnel but it is not needed after glideslope/localizer capture. 

11) Felt like could use this system in real-world operations with less than one 

12) In terms of actual weather that the pilot flies in during transport approach ope

that would use the system 5% for weather related landings and 50% for nigh

13) We need full symbology, SVS and EVS HUDs for Part 121 operations. 

14) Effectiveness would have been much higher with a null error pitch and

flight director system on Boeing 777) rather

15) Generally, pilots flying a path-directed approach, missed approach, arrival, or 

presume that if they remain within required parameters of the path, that terrain 

deconflicted. Therefore, the SVS was of value only as confirmation that the di

accurate. 

16) Important flight inform

it greatly reduces pilot w

information such as with a tunnel display is distracting. 

17) Having to look around or through the HUD (after declutter) at 100 feet pre

PF – better to have the PNF call landing or go around. 
79

ncept of the tunnel 

 reduced workload and increased 

25) Minimums call at 200 feet and no call at 100 ft by PNF is not the best approach. 

24) Even though had no previous FLIR or tunnel experience, once I grasped the co

I was a true convert - having SVS/FLIR and tunnel significantly

scan level and situational awareness. 
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

2. REPORT TYPE 

Technical Publication

 4. TITLE AND SUBTITLE

Evaluation of Fused Synthetic and Enhanced Vision Display Concepts for 

Low-Visibility Approach and Landing 

5a. CONTRACT NUMBER

 6. AUTHOR(S)

Bailey, Randall. E.; Kramer, L:ynda J.; Prinzel, Lawrence J., III; Wilz, 

Susan J.

 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

 8. PERFORMING ORGANIZATION

 REPORT NUMBER

L-19504

10. SPONSOR/MONITOR'S ACRONYM(S)

NASA

13. SUPPLEMENTARY NOTES

12. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified - Unlimited

Subject Category 03

Availability: NASA CASI (443) 757-5802

19a. NAME OF RESPONSIBLE PERSON

STI Help Desk (email: help@sti.nasa.gov)

14. ABSTRACT

NASA is developing revolutionary crew-vehicle interface technologies that strive to proactively overcome aircraft safety barriers that would otherwise 

constrain the full realization of the next generation air transportation system. A piloted simulation experiment was conducted to evaluate the complementary 

use of Synthetic and Enhanced Vision technologies. Specific focus was placed on new techniques for integration and/or fusion of Enhanced and Synthetic 

Vision and its impact within a two-crew flight deck during low-visibility approach and landing operations. Overall, the experimental data showed that 

significant improvements in situation awareness, without concomitant increases in workload and display clutter, could be provided by the integration and/or 

fusion of synthetic and enhanced vision technologies for the pilot-flying and the pilot-not-flying. Improvements in lateral path control performance were 

realized when the Head-Up Display concepts included a tunnel, independent of the imagery (enhanced vision or fusion of enhanced and synthetic vision) 

presented with it. During non-normal operations, the ability of the crew to handle substantial navigational errors and runway incursions were neither improved 

nor adversely impacted by the display concepts. The addition of Enhanced Vision may not, of itself, provide an improvement in runway incursion detection 

without being specifically tailored for this application. 

15. SUBJECT TERMS

Aviation Safety; Crew Resource Management; Enhanced Vision System; FLIR; Head-Up Display; Image Fusion; Runway 

Incursions; Synthetic Vision System

18. NUMBER

 OF 

 PAGES

84

19b. TELEPHONE NUMBER (Include area code)

(443) 757-5802

a. REPORT

U

c. THIS PAGE

U

b. ABSTRACT

U

17. LIMITATION OF 

 ABSTRACT

UU

Prescribed by ANSI Std. Z39.18

Standard Form 298 (Rev. 8-98)

3. DATES COVERED (From - To)

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

609866.02.07.07.02

11. SPONSOR/MONITOR'S REPORT

 NUMBER(S)

NASA/TP-2009-215707

16. SECURITY CLASSIFICATION OF:

The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, 

gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and 

Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person 

shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.

1. REPORT DATE (DD-MM-YYYY)

01- 04 - 2009