DOT/FAA/AM-00/26

Roger Schvaneveldt

Department of Psychology, New Mexico State University

Las Cruces, NM 88003

Dennis B. Beringer

FAA Civil Aeromedical Institute, P.O. Box 25082

Oklahoma City, OK 73125

John Lamonica

Lamonica Aviation, 4355 East La Osa Dr.

Tucson, AZ 85718-6757

Richard Tucker

U.S. Air Force Research Laboratory

Albuquerque, N.M. 87107

Christopher Nance

Department of Psychology, New Mexico State University

Las Cruces, NM 88003

July 2000

Final Report

This document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161.

Priorities, Organization, and

Sources of Information

Accessed by Pilots in Various

Phases of Flight

Office of Aviation Medicine

Washington, D.C. 20591

U.S. Department

of Transportation

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents thereof.
i

Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-00/26

5. Report Date

July 2000

4. Title and Subtitle

Priorities, Organization, and Sources of Information Accessed by Pilots in

Various Phases of Flight 6. Performing Organization Code

7. Author(s)

Schvaneveldt, R.1, Beringer, D.B.2, Lamonica, J.3, Tucker, R.4,

and Nance, C.1

8. Performing Organization Report No.

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

11. Contract or Grant No.

1

Department of Psychology, New Mexico State University, Las Cruces, NM 88003

2

FAA Civil Aeromedical Institute, P.O. Box 25082, Oklahoma City, OK 73125

3

Lamonica Aviation, 4355 East La Osa Dr., Tucson, AZ 85718-6757

4

U.S. Air Force Research Laboratory, Albuquerque, NM 87107 Grant # B98-G-019

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered

Office of Aviation Medicine

Federal Aviation Administration

800 Independence Avenue, S.W.

Washington, D.C. 20591 14. Sponsoring Agency Code

15. Supplemental Notes

This work was accomplished under the approved task AM-A-98-HRR-510.

16. Abstract

In the first project of the study, 27 pilots rated the priority of information required for flight. These pilots were

divided by flight experience into novices (65 to 820 hours’ flight time) and experienced pilots (1600 to 17,000

hours’ flight time). Participants rated 29 information elements across seven phases of flight. These data show the

shifting priorities of information across phases of flight, and some clear differences in priority assignments

appeared between the novices and the experienced pilots. In the second project, 34 pilots, some from Project 1,

participated in the collection of relatedness data for 231 pairs of information elements. A Pathfinder analysis and

hierarchical clustering were conducted showing connections among these elements and grouping of the

elements. Pilot experience had little influence on the form of the network of associations. The discussion

explores the potential of these data for instrumentation layout and integration of cockpit information systems,

datalink design, and development of flight instruction curricula.

17. Key Words 18. Distribution Statement

Flight Information, Pathfinder Analysis, Clustering of

Flight Information, Priorities of Flight Information

Document is available to the public through the

National Technical Information Service, Springfield,

Virginia, 22161

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 43

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

iii

ACKNOWLEDGMENTS

The authors would like to thank the following individuals for their assistance during the study and the

preparation of the manuscript: Stacie Bonnell, Harry Pedersen, and Robert Scarbrough for assistance in

preparing materials and collecting data; Chuck McClain, Wayne Stevenson, and Col. Frank Borman for their

expertise and for providing data collection facilities; the many pilots who participated in the study.

v

TABLE OF CONTENTS

Introduction ------------------------------------------------------------------------------------------------------ 1

Motivation ------------------------------------------------------------------------------------------------ 1

Display Design -------------------------------------------------------------------------------------- 1

Comparison with Task Analysis------------------------------------------------------------------ 2

Evaluating Changes in the National Airspace System --------------------------------------- 2

General Plan of the Investigation ---------------------------------------------------------------- 2

Project 1: Phases of Flight, Information Elements, and Information Sources -------------------- 3

Sources of Information in the Current Airspace System ------------------------------------ 3

Project 2: The Priorities of the Information Elements in Various Phases of Flight ------------ 5

Method ---------------------------------------------------------------------------------------------------- 5

Participants ------------------------------------------------------------------------------------------- 5

Materials ---------------------------------------------------------------------------------------------- 7

Design and Procedure------------------------------------------------------------------------------ 7

Results and Discussion --------------------------------------------------------------------------------- 7

Overall Priorities for Phases of Flight and Information Elements ------------------------ 7

Priority and Pilot Experience Level ------------------------------------------------------------10

Information Priorities in Phases of Flight----------------------------------------------------- 10

Information Priorities and Workload ----------------------------------------------------------21

Project 3: The Mental Organization of Information Accessed in Flight -------------------------22

Network Analyses of Relatedness Data --------------------------------------------------------22

Graph Theory and Relations among Information Elements ------------------------------ 22

Method ---------------------------------------------------------------------------------------------------23

Participants ------------------------------------------------------------------------------------------ 23

Materials ---------------------------------------------------------------------------------------------23

Design and Procedure----------------------------------------------------------------------------- 23

Results and Discussion --------------------------------------------------------------------------------23

Clusters of Information Elements --------------------------------------------------------------26

Experience and Networks ------------------------------------------------------------------------26

Conclusions, Applications, Recommendations, and Caveats ---------------------------------------- 27

Instrumentation Layout and Integrated Cockpit Information Systems ---------------------28

Flight-Management Systems -------------------------------------------------------------------------28

Controller-Pilot Communications and Datalink -------------------------------------------------28

Flight Instruction Curricula --------------------------------------------------------------------------29

An Example: Applying the Data to Display Allocation/Design -------------------------------29

Prospectus ------------------------------------------------------------------------------------------------30

References--------------------------------------------------------------------------------------------------------30

Appendices

Appendix A. Demographic Questionnaire ------------------------------------------------------- A1

Appendix B. Instructions for Priority Ratings and the Priority Rating Form------------- B1

Appendix C. Procedure for Collecting Relatedness Ratings ---------------------------------- C1
List of Figures

Figure 1. Mean priority for flying phases ----------------------------------------------------------------- 7

Figure 2. Distribution of ratings across phases of flight ----------------------------------------------- 8

Figure 3. Priorities, phase of flight, and experience ---------------------------------------------------10

Figure 4. Takeoff priorities ---------------------------------------------------------------------------------11

Figure 5. Climb priorities -----------------------------------------------------------------------------------12

Figure 6. Transition to cruise priorities ------------------------------------------------------------------ 13

Figure 7. Cruise priorities -----------------------------------------------------------------------------------14

Figure 8. Descent priorities ---------------------------------------------------------------------------------15

Figure 9. Approach priorities -------------------------------------------------------------------------------16

Figure 10. Landing priorities---------------------------------------------------------------------------------17

Figure 11. Preflight planning priorities --------------------------------------------------------------------18

Figure 12. In-flight planning priorities --------------------------------------------------------------------19

Figure 13. The pathfinder network for average data (q=2, r=8) --------------------------------------24

Figure 14. Hierarchical clusters in the pathfinder network --------------------------------------------26

Figure 15. Differences in novice and experienced pilots’ networks ----------------------------------27

Figure 16. The relatedness rating screen ------------------------------------------------------------------ C1

List of Tables

Table 1. Phases of flight -------------------------------------------------------------------------------------- 3

Table 2. Taxonomy of 29 information elements -------------------------------------------------------- 4

Table 3. Taxonomy of sources of information needed in flight-------------------------------------- 5

Table 4. Sources of the information elements in the National Airspace System------------------ 6

Table 5. Definition of the priority ratings---------------------------------------------------------------- 7

Table 6. Average priorities of information elements --------------------------------------------------- 9

Table 7. Major novice-experienced priority differences ----------------------------------------------20

Table 8. Highest priority information by phase of flight---------------------------------------------20

Table 9. Workload measures and correlations ----------------------------------------------------------22

Table 10. The 22 information elements used for relatedness ratings --------------------------------23

Table 11. Most highly connected pairs of information elements-------------------------------------25

vi
vii

EXECUTIVE SUMMARY

The Aviation Research Group (ARG) in the Department of Psychology at New Mexico State University (NMSU) has research interests relating to

several aspects of the organization and display of

information in the cockpit. In this report, we present

a systematic analysis of the information required by

pilots in various phases of flight. Our aim is to specify

what information is needed, when it is needed, and

how pilots conceive of the organization of the information. This project was devoted to an analysis of the

information required in normal flight operations as

contrasted with unusual or emergency situations. In

future work, we plan to conduct a similar study of

information requirements in various emergencies.

The nature of the present study is largely descriptive

in that it provides numerous analyses of the priorities

pilots assign to different information elements. The

study required the development of various taxonomies

including: (a) Information used in flight and planning,

(b) Phases of flight, and (c) Sources of information used

in flight. The analyses show overall priorities of the

information elements in various phases of flight and

how these priorities compare for more- versus lessexperienced pilots. Not surprisingly, priorities change

over the course of a flight, and we proposed that the

design of new displays would benefit from considering

these changing priorities. We also present data on how

pilots conceive of the organization of the various elements of information. These data, too, can assist display

designers in layout of information.

The differences in information priorities as a function of flying experience identify some issues that

flight instructors should consider in flight training.

Overall, more experienced pilots tend to give higher

priorities to several information elements as though

they are considering more factors with higher priority. In particular phases of flight, less experienced

pilots give lower priority to communications both

with ATC and with traffic than do more experienced

pilots. The less experienced pilots assign higher priority to vertical velocity than do more experienced

pilots. A discussion of these differences and others

can be found in the report.

Our analysis of the sources of information required in flight shows the dramatic impact of GPS

technology on aviation. The analysis of sources also

allows a determination of the degree to which redundant information is available. Clearly, safety may be

compromised when information is only available

from a single source.

Modern aviation is undergoing several dramatic

changes that affect nearly every aspect of aircraft

operating in the national airspace system. Among

these changes are: (a) the move to “Free Flight,” (b)

the development of new control systems for aircraft,

(c) development of new information displays and

methods for selecting displays, and (d) changes in

processes and policies of aviation-related organizations. As these changes come about and as an aid in

planning, it is valuable to have models that can help

analyze the effect of the changes on critical issues of

safety, automation, workload, and situation awareness. We have presented a pilot-centered model of

information requirements in flight tasks emphasizing the source, priority, and mental organization of

the information elements. This model should allow

one to make informed decisions about the design of

new technology including information displays. The

model should also be useful for evaluating the effect

of changes in the airspace system by providing baseline

information about where pilots can obtain information in the current system and by reminding policy

makers about what information pilots need and when

they need it.

1

PRIORITIES, ORGANIZATION, AND SOURCES OF INFORMATION

ACCESSED BY PILOTS IN VARIOUS PHASES OF FLIGHT

INTRODUCTION

This report summarizes the research accomplishments for a one-year project investigating the priorities pilots assign to different information in various

phases of flight under normal operating conditions.

We use the term, information, to emphasize our focus

on interpreted data, in contrast to raw data. For

example, when we refer to information about Traffic,

we mean the knowledge that aircraft traffic is present

in some particular position rather than the particular

data that leads to the knowledge. Thus, our analysis

is directed at a cognitive level in contrast to a concern

with perceptual processes. The term, information

elements, is used to refer to the knowledge pilots come

to possess (or should possess) during the course of

flight. An understanding of how pilots prioritize and

organize flight-related information can be of value in

the design of display systems for pilots. The data

presented in this report can also help characterize

what information pilots need as we attempt to characterize the nature of the National Airspace System.

This introductory section will provide the rationale

for the research effort, as well as background for the

research.

Motivation

There were several motivations for us to undertake

the research reported here. First, a characterization of

information priorities related to various phases of

flight together with an analysis of the mental organization associated with the information would be

useful in the design of new displays for aircraft. Those

displays could then be organized to present only the

most salient information for any given phase of flight

and in a well-integrated fashion, reduce the cognitive

requirements, and thus the workload, imposed on the

pilot. The end result should be that the pilot can

devote more effort to other tasks not directly associated with extracting information from cockpit displays. Second, this approach may provide an alternative

to traditional task analysis by examining operator

perceptions and cognitive organizations of data.

Third, identifying the priorities of information should

allow us to characterize how pilots view information

needs in the current National Airspace System, and to

derive from that how changes in the use of that

airspace or the allocation of tasks between actors in

the system may change perceived priorities. This

should then have an influence on how future displays

are designed to best support any new tasks or tasks

performed in an altered airspace environment.

Display Design

As modern aviation adopts new technology for the

display of information and the control of aircraft

systems, there is a clear shift away from individual

dedicated displays (e.g., altimeter, airspeed indicator, turn-and-bank indicator) toward integrated displays in which most information comes from a

common source such as panel-mounted cathode-ray

tubes (CRT’s), head-up displays (HUDs), or helmetmounted displays (HMDs). Some critical issues in

the design and engineering of such displays concern

the decisions about what information to display at

any given time, as well as how to display it.

Design decisions about what to display at what

times would benefit considerably from a thorough

analysis of the various required elements of information (aircraft position, weather, engine status, communications, etc.) and their properties (priority of

the information, source of the information, quality

of the information, etc.) in various phases of flight.

Together with data revealing how pilots organize

these information elements, the analysis can assist

efforts toward integration in displays and prioritization and layering of information elements. For

example, displays designed so that critical information is available when needed in a practical format

should be of great assistance to the pilot who is

actively controlling the aircraft. Andre and Wickens

(1991) demonstrated the ability of an integrated

three-dimensional cockpit display system to reduce

mental workload while increasing situation awareness. Well-designed displays should also help establish and maintain the pilot’s situational awareness
2

when automated systems are in control. Research on

mode errors and mode awareness in supervisory control (Sarter & Woods, 1992) indicates that the increased capabilities and autonomy of new automated

systems have made mode awareness problems more

prevalent in the modern cockpit.

Comparison With Task Analysis

A long-standing goal in the analysis of aircraft

systems is to produce thorough analyses of the tasks

involved in flying aircraft and the information required to perform the tasks (Alexander C. Williams,

Jr. in Roscoe, 1980). Although the analysis we propose to perform is related to task analysis, it is also

distinct in some important ways. In the usual approaches to task analysis, goal-directed activities are

analyzed in terms of goals to be accomplished and the

actions required to accomplish those goals (cf.

Sutcliffe, 1997). A limitation of this approach in the

context of aviation is that the actions required of a

pilot to accomplish goals are largely determined by

the nature of the technology in the aircraft so, in

essence, every aircraft calls for a distinct task analysis.

Of course, by abstracting the actions somewhat, one

could reduce the number of distinct task analyses

required, but abstracting would often move the analysis from the task per se toward the goals of the tasks.

For example, we could use “put in 10 degrees of flaps”

instead of “move the flap control to the 10 degrees

mark,” but in doing so, we have technically moved

from specifying an action to specifying a goal (or,

better, a subgoal). It seems clear that goals do remain

more constant across technological differences than

actions do. Even with extreme degrees of automation

in aircraft, most goals of flight remain the same as

they were with pilots in control throughout the

flight1

. Still, we should be able to provide a more

thorough analysis of the various phases of flight than

simply specifying the goals (and subgoals) of each phase

and still have some generality across technologies.

Our approach assumes that goals remain relatively

constant across technological changes, and the critical information pertaining to flight also remains

relatively constant. For the pilot actively controlling

the aircraft, it is obvious that particular information

elements are critically important at different times in

a flight, but it is also true that even when the aircraft

is being flown by automated systems, the same

information elements are critical. Moreover, for pilots to maintain (or to quickly attain) good situational awareness, they must have these critical

information elements whether they are actively in

control or not. We also propose that pilots’ mental

models of flight depend upon having the appropriate

information elements. When the pilot’s model contains

incorrect information, “pilot error” is not far behind.

Evaluating Changes in the

National Airspace System

Another application of the data we report is found

in its potential for assisting in the analysis of the

consequences of changes in the National Airspace

System. Our analysis identifies the critical information elements for the various phases of flight. Given

these data, it is reasonable to ask about how access to

this critical information will be affected by any change

in the system. If a proposed change in the system

eliminates certain information, our analysis should

help to determine whether that information is usually

used and, if so, alternative sources of the information

could be sought. Studies could then determine

whether the alternatives are adequate or whether new

options should be considered. In summary, we see

our model as providing baseline information about

the current state of the National Airspace System that

will be useful in planning.

General Plan of the Investigation

The research team consisted of several pilots and

student assistants in the Aviation Research Group

(ARG) at New Mexico State University, including

two pilot consultants. The decisions about materials

and taxonomies were made by consulting with the

pilots on the research team. The research involved

three distinct projects:

Project 1. Selecting particular phases of flight,

developing a taxonomy of information elements accessed during these phases, developing a taxonomy of

information sources, and identifying the sources of

each information element.

Project 2. Collecting priority ratings from pilots

for each information element in each phase of flight.

Project 3. Collecting relatedness ratings from pilots for a subset of the information elements to allow

for an analysis of the organization of the information

elements.

1

 There are clear exceptions where certain goals cannot be preserved with automation as with the goal of realizing the pleasure of

flying well.
3

Project 1: Phases of Flight, Information

Elements, and Information Sources

The ARG team developed initial taxonomies of

phases of flight, the information elements accessed in

these phases, and the sources of these information

elements in the National Airspace System. The taxonomies were refined by discussions with pilots resulting in the final products.

The phases of flight are shown in Table 1. To make

the project manageable within the one-year period

for the work, we initially decided to restrict our work

to flying phases, as contrasted with ground phases.

The major concerns of safety lie in the flying phases.

However, it became clear that it was also necessary to

include some planning phases, as well as flying phases,

because preliminary testing suggested that pilots want

to include considerations of planning. If separate

planning phases are not specified, considerations of

planning will be included in flying phases. To keep

planning and flying distinct, it was decided to include two planning phases, Preflight and In-Flight

Planning, in addition to flying phases. In-Flight

Planning was included to keep needed planning information distinct from information used in flying,

particularly in the Cruise Phase.

The flying phases we included are those found in

most discussions of the phases of flight. The Transition to Cruise Phase was included to keep the information used there distinct from the Climb and the

Cruise Phases. In contrast, the Transition to Descent

was not included because our pilot consultants

thought that the transition was not that distinct from

the Descent itself. Finally, the analysis was restricted

to normal operations, as distinct from emergencies,

to keep the project manageable in the time frame. A

follow-on study will conduct a similar analysis of

various emergencies.

Table 2 was arrived at after performing a preliminary analysis of the information required in normal

flight operations. To keep the number of elements

manageable, some of the items represent large categories of elements (e.g., Aircraft Configuration, Engine

Health, Airport Configuration, General Weather).

Other elements refer to very specific items, such as

Pitch, Bank, Airspeed, and Altitude. The choice of

these elements is intended to be appropriate for

evaluating information priorities in the various phases

of flight.

Sources of Information in the Current

Airspace System

To help relate our analyses of information requirements to the state of the National Airspace System,

we also undertook an analysis of how pilots obtain

the information they need in flight. First the taxonomy of information sources shown in Table 3 was

developed. The sources distinguish major categories of

information sources (Communications, Documents,

2

 The statistics on percent of flight time and percent of accidents come from a

1985 report by Boeing reported in Nagel (1988). This report distinguished

between Initial Climb and Climb. Percentages shown combine Initial Climb

with Takeoff. The report also distinguished between Initial Approach and Final

Approach, whereas percentages shown combine these into Approach. The

statistics may be more representative of air carrier aviation than of general

aviation.

Table 1. Phases of flight.

Phases of Flight

Percent of

Flight Time

Percent of

Accidents2

Pre-flight Planning NA NA

Takeoff 2% 21.4%

Climb 13% 8.8%

Transition to Cruise (no data) (no data)

Cruise 60% 8.8%

In-flight Planning NA NA

Descent 10% 13.4%

Approach 14% 41.8%

Landing 1% 3.8%
4

Table 2. Taxonomy of 29 information elements.

Aircraft Expanded Description of each Element

Configuration

Aircraft configuration Flaps, landing gear, cowl flaps, speed brakes, spoilers

Engine

Engine health General health of engines

Fuel quantity Fuel remaining

Fuel selection Current selected source of fuel

RPM (power) Power setting or desired power

Position

Altitude - AGL Altitude above ground level

Altitude - MSL Altitude above Mean Sea Level

Distance Distance to waypoint or airport

Orientation

Bank Bank

Pitch (attitude) Pitch

Yaw Yaw

Direction

Course Desired track - Planned course

Heading Nose direction

Track Actual track over the ground

Waypoints Location and type of enroute waypoints

Speed

Airspeed Indicated airspeed

Ground speed Actual speed over the ground

Vertical velocity Climb rate or descent rate

Schedule

Time - ETA/ETE Estimated time of arrival / Estimated time enroute

Environment

Airport

Airport configuration Altitude, runways, approaches, NOTAMs, active runway

Runway aim point Desired point of touchdown

Runway remaining Length of remaining runway on takeoff or landing

Communications

ATC Comm ATC instructions, clearances, etc.

Traffic / other Comm Communication with traffic or other information source

Hazards

Obstructions Towers, trees, mountains, etc.

Traffic Other aircraft

Regulations

Airspace Type of airspace A B C D, restricted, MOA etc.

Weather

General General weather conditions

Wind Wind direction and intensity
5

Table 3. Taxonomy of sources of information

needed in flight.

Communications

ATC Air Traffic Control Communications

Comm Other (Flight Service, Traffic, ATIS, etc.)

Documents

Chart Charts

Hist History (Aircraft or Flight)

Plan Flight Plan

Instruments

ASI Airspeed Indicator

AI Attitude Indicator (Artificial Horizon)

ALT Altimeter

TC Turn Coordinator

HI Heading Indicator

VSI Vertical Speed Indicator

Eng I. Engine & Fuel (RPM, Oil Pressure,

Temperature, etc.)

Clock Clock

DME Distance Measuring Equipment

GPS Global Positioning System

VOR VHF Omnidirectional Radio Range

Direct Perception

Aud Auditory (Excluding Communications)

Vis Visual (Excluding Reading Instruments)

Instruments, and Direct Perception). For each of the

information elements (Table 2), we identified the

sources that can provide the information. These

resulting sources for each information element are

shown in Table 4. The sources of information are

distinguished according to whether the information

is directly available (D), available through inference

using multiple sources of information and/or assumptions (I), or is planning information producing

desired or expected values (P). Clearly, direct information is more readily available and immediately

usable than is information that must be inferred. Thus,

one view of Table 4 might emphasize the “D” entries.

Several observations can be made about the outcome of the analysis of information sources shown in

Table 4. First, the observation that much of the

information is available from direct visual perception

(not counting reading instruments) brings home the

point that, without visual conditions, flight is a

strikingly different activity. The potential differences

between instrument meteorological conditions (IMC)

and visual meteorological conditions (VMC) are also

dramatized by the number of different information

elements available from ATC. The table also reveals

redundancy in information sources. Safety is certainly enhanced by the availability of alternate sources

of information in case one source should fail. Finally,

the ubiquity of GPS as an information source suggests that this technology is destined to become a

major resource in the future if it hasn’t already. As

others in the human factors and pilot communities

have noted, there are problems of over reliance, over

confidence, and lack of familiarity with particular

GPS units that can lead to dangerous situations. At

the same time, the value of the information provided

by GPS systems can greatly simplify tasks of navigating leading to lower workload. Making the system

work well and making it reliable and usable are

challenges for pilots, manufacturers, and researchers.

Project 2: The Priorities of the Information

Elements in Various Phases of Flight

During development of the priority-rating task,

we considered distinguishing between visual and

instrument flight for some of the properties of information elements because the sources of information

often depend on this distinction. Our initial tests,

however, indicated that the differences were minor,

so we simply asked for general priorities.

Following the selection of the phases of flight and

the development of the taxonomy of information

elements, we began to collect the priority rating data

by distributing questionnaires at several sites around

the country. The details of this methodology are

provided in the following sections.

METHOD

Participants

Twenty-seven certified pilots of varying age and

experience were used as sources for the priority rating

data. To examine the effect of experience on judgments of priorities, we separated the pilots into two

groups based on their total flying time. The hours of

the 11 less experienced pilots (novices) ranged from

65 to 820 with a mean of 361 and a standard deviation of 225. The 16 more experienced pilots (experienced) had a range of flying time from 1,600 to

17,000 hours with a mean of 6,634 and a standard

deviation of 4,352. Fifty-four percent of the less experienced pilots held the Instrument Rating, while 88% of

the more experienced pilots were instrument rated.
6

Table 4. Sources of the information elements in the National Airspace System.

Information Element

ATCCommChartHistPlanASIAIALTTCHIVSIEng InstClockDMEGPSVORAudVis

Aircraft Config. D D D

Engine Health P D D D

Fuel Quantity P D

Fuel Selection

Power I D D

Altitude - AGL DP P P D

Altitude - MSL DP P P D D

Distance DP P DP I P I D D D

Bank D DI I D

Pitch I D I I D

Yaw D I

Course DP P DP I P I DP D D

Heading DIP P I D I I D

Track DP P P P I I D I D

Waypoints P IP P P I D DP D D

Airspeed I P D I I

Groundspeed DP I P I I D D I D

Vertical Velocity DP I P I I I D D

Time - ETA/ETE DP I P D D D

Airport Config. DP D DP D D D

Runway Aimpt. P D P D

Runway Remain. D DP D D D

Obstructions DP DP D DP D D D

Traffic D DP D

Airspace DP D DP D D D D

General Weather DP DP P D

Wind DP DP P I I I I

Note. D = Directly obtained (current) values

I = Inferred (current) values using multiple sources

P = Planned or expected values
7

Table 5. Definition of the priority ratings.

Priority Description

1 Critical and/or Frequently Accessed

2 Important and/or Usually Accessed

3 Relevant and/or Sometimes Accessed

blank Not Relevant or Rarely Accessed

Note. Blanks were coded as 4 for analysis.

Materials

The priority rating data were collected using a

questionnaire that included demographic information (Appendix A), a definition of the information

elements (Table 2), instructions for the priority ratings (Appendix B), and a form listing the information

elements on the rows, the phases of flight on the

columns and cells for assigning the priority ratings

for each information element in each phase of flight

(Appendix B). The definition of the priority ratings

is shown in Table 5. The cells left blank were coded

as 4 (lowest priority) for analysis of the priority

scores.

Design and Procedure

Collection of the priority data occurred over an

eight-month period. The participants were given the

priority-rating packet and a stamped, addressed envelope. Upon completion, the participants returned

the demographic information and the priority ratings

through the mail.

RESULTS AND DISCUSSION

Overall Priorities for Phases of Flight and

Information Elements

The priority rating data lends itself to several

different analyses. One way to examine these ratings

is to average them across information sources within

a phase of flight, the result reflecting how many

sources of information were rated as being critical

during that phase. The average would approach 1 as

more sources were rated “critical.” The average priority ratings for each phase of flight are shown in Figure 13

.

By way of comparison, the mean priority ratings

were 2.45 and 2.35 for Preflight Planning and Inflight Planning, respectively. For the flying phases,

1.0

2.0

3.0

4.0

Takeoff Climb

Tran to

Cruise Cruise Descent Approach Landing

Phase of Flight

Mean Priority (1 = High)

Figure 1. Mean priority for flying phases.

3

 This figure and several others have lines connecting the points. Strictly speaking, the lines are inappropriate because the abscissa

represents discrete points rather than a continuum. However, the addition of lines facilitates reading the figures. Think of the lines as

leading the eye from point to point, rather than as an interpolation between points.
8

the highest average priority ratings were for the Approach, followed by the Landing, Takeoff, Descent,

Climb, Cruise, and Transition to Cruise. Climb and

Descent had nearly identical means. This ordering of

priority ratings appears to roughly correspond to

what pilots often report as the relative difficulty of

the different phases of flight. In a later section, we

discuss the relationship between priorities and

workload.

It is also of interest to examine how the priority

ratings were distributed across the four categories (1

= top priority to 4 = irrelevant or rarely accessed). The

distributions are shown in Figure 2. The figure shows

the proportion of each of the 4 rating categories in

each phase of flight. The proportions sum to 1 for

each phase. These distributions show that the overall

higher average priorities in the more demanding

phases of flight are primarily due to an increase in the

proportion of information elements receiving the

highest priority rating. These data also suggest that

the 3rd category (Relevant and/or Sometimes Accessed) is not used very often. However, its frequency

does increase in the less demanding phases of flight

where, presumably, pilots are monitoring several

factors on an occasional schedule.

In addition to looking at priorities for phases of

flight, it is informative to see how priorities are

assigned to the information elements overall. The

overall averages are shown in Table 6 with separate

columns for the flying phases and the planning phases.

There are clear differences in the priorities associated

with planning and with flying. Of course, we need to

examine the information priorities separately for each

of the phases, but the overall priorities give a general

idea of which items of information are of interest

throughout a flight (those at the top of the list). The

planning priorities reflect the relative importance of

the different information elements. Proper planning

should include special consideration of the highpriority items. Items at the end of the list may be

important in particular phases but not overall. For

example, Runway Remaining has an average priority

near 1 in Landing, in contrast to near 4 in Cruise.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Takeoff Climb Tran to

Cruise

Cruise Descent Approach Landing

Phases of Flight

Proportion of Ratings

1st Priority

2nd Priority

3rd Priority

4th Priority

Figure 2. Distribution of ratings across phases of flight.
9

Table 6. Average priorities of information elements.

Flying Planning

Information Element Priority Information Element Priority

RPM (power) 1.44 Fuel quantity 1.35

Airspeed 1.49 General weather 1.39

Engine health 1.58 Time - ETA/ETE 1.57

Traffic 1.66 Wind 1.61

Pitch (attitude) 1.67 Airport configuration 1.78

General weather 1.79 Waypoints 1.85

ATC 1.79 Altitude - MSL 1.87

Aircraft configuration 1.80 Distance 1.93

Altitude - MSL 1.85 Airspace 1.96

Heading 1.89 Course 2.07

Altitude - AGL 1.92 Obstructions 2.07

Wind 1.98 Ground speed 2.09

Traffic / other Comm 1.99 Engine health 2.31

Course 2.13 ATC 2.35

Obstructions 2.13 Fuel selection 2.35

Track 2.13 Heading 2.39

Airspace 2.19 RPM (power) 2.44

Vertical Velocity 2.20 Altitude - AGL 2.44

Bank 2.24 Airspeed 2.50

Fuel selection 2.24 Traffic / other Comm 2.57

Yaw 2.25 Track 2.67

Fuel quantity 2.30 Aircraft configuration 2.67

Airport configuration 2.55 Traffic 2.74

Distance 2.65 Runway remaining 3.22

Ground speed 2.74 Bank 3.43

Waypoints 2.76 Pitch (attitude) 3.46

Runway aim point 2.97 Vertical Velocity 3.48

Time - ETA/ETE 2.98 Yaw 3.50

Runway remaining 2.99 Runway aim point 3.54
10

Priority and Pilot Experience Level

It is of interest to consider how pilot experience

relates to the judgment of priorities. The differences

as a function of experience were small, but there are

some differences worth considering. Average priority

ratings for the various phases of flight and pilot

experience level are shown in Figure 3. Values for

planning are shown at the right end of the graph for

comparison. The more experienced pilots assigned

higher overall priorities for the most demanding

phases of flight. The 0.12 overall difference in the

average priority is significant, t(6) = 2.62, p = 0.031.

Information Priorities in Phases of Flight

Next, we turn to an analysis of the details of the

priorities in each phase of flight. The goal of this

analysis is to define which information elements have

high priority in particular phases. Such data would be

useful in designing dynamic information displays

that provide the most relevant information at the

proper time. Figure 4 through Figure 12 show the

information priorities associated with each phase of

flight. The figures are organized so that the information elements are sorted for each phase of flight using

the average priority of the more-experienced pilots

(Exp). The average priority of the novice pilots (Nov)

is also shown for comparison. Points of substantial

differences between the two groups of pilots may

identify areas that could usefully be addressed in

training. Table 7 lists the significant differences using p = 0.05. All differences with chance probability

less than or equal to 0.05 are shown along with the

probability of a difference at least this large by chance

according to a two-tailed t-test with 25 df. Because

these differences are selected from a large set of tests

(9 phases of flight x 29 information elements = 261

tests), we would expect to find 13 (5% of 261) such

significant differences by chance alone. Thus, the 16

significant differences found should be viewed with

caution. The lower p values shown on the table show

which differences are most likely to be reliable, and

the repeated occurrence of significant differences

associated with the same information element leads

to greater confidence in those effects.

As the overall means suggested earlier, there is a

clear tendency for more experienced pilots to assign

higher priorities. Most of the differences in 7 have the

more experienced pilots assigning the higher priority.

Novice pilots may have less capacity available to deal

with all of the important information, so some of it

takes lower priority. It is important to distinguish

between what should be high priority and what can

be high priority given the pilot’s ability. Interestingly

the novice pilots appear to assign priority ratings in

1.0

2.0

3.0

4.0

Takeoff

Climb

Tran to Cruise

Cruise

Descent

Approach

Landing

Preflight Planning

In-Flight Planning

Phases of Flight

Mean Priority (1 = High)

Experienced

Novice

Figure 3. Priorities, phase of flight, and experience.
11

Figure 4. Takeoff priorities.

1234

RPM

(power)

Wind Engine health Aircraft configuration GAirspeed eneral Weather Traffic / Other Comm Pitch (attitude) Obstructions Traffic Heading ATCFueComm l selection YawAltitude - AGL Fuel quantity Track BankRAirspace unway remaining Altitude - MSL Airport configuration Course Vertical Velocity DWistance Taypoints ime - ETA/ETE Ground speed Runway aim point

Priority Mean

Takeoff Exp Takeoff Nov
12

Figure 5. Climb priorities.

1234

RPM

(power)

Engine health Airspeed PA ii tr cc hraft configuration (attitude) Traffic Heading Obstructions Track Altitude - MSL CTr oa uf rfic / Other Comm se ATC Comm Altitude - AGL Airspace Bank YawFuel selection VGertical Velocity Weneral Weather ind Fuel quantity Waypoints Distance AGi rr oport configuration und speed Time - ETA/ETE RRunway aim point unway remaining

Mean Priority

Climb Exp Climb Nov
13

Figure 6. Transition to cruise priorities.

1234

Engine

health

RPM

(power)

Altitude

-

MSL

Traffic Airspeed ATC Comm Course Heading Pitch (attitude) ATircraft configuration rack General Weather Altitude - AGL Bank Airspace Fuel selection Fuel quantity YTraffic / Other Comm awWaypoints Vertical Velocity Distance Wind Ground speed Time - ETA/ETE Obstructions RAirport configuration unway aim point Runway remaining

Mean Priority

Transition to Cruise Exp Transition to Cruise Nov
14

Figure 7. Cruise priorities.

1234

Altitude -

MSL

RPM

(power)

Course Engine health Traffic Track ATC Comm Heading Airspeed Ground speed WGeneral Weather aypoints Fuel quantity FA uir ec lraft configuration selection Airspace Distance BTr aa nf kfic / Other Comm PTime - ETA/ETE itch (attitude) Altitude - AGL YawWind Obstructions AVertical Velocity Ruir np wort configuration R ay remaining unway aim point

Mean Priority

Cruise Exp Cruise Nov
15

Figure 8. Descent priorities.

1234

Traffic ATr latif tf uic /

Other

Comm

de -

MSL

RGeneral

Weather

PM

(power)

Airspeed ATC Comm Course Altitude - AGL Pitch (attitude) AA i i rr sport configuration pace Engine health Track Heading Fuel selection Obstructions Distance Bank VFertical Velocity uel quantity WAi ir nc draft configuration Waypoints YawTGime - ETA/ETE round speed RRunway aim point unway remaining

Mean Priority

Descent Exp Descent Nov
16

Figure 9. Approach priorities.

1234

Airspeed Altitude -

AGL

Traffic AGeneral

Weather

TC

Comm

RPM

(power)

ATr iarpff oic /

Other

Comm

Altit

rt

configuration

ude -

MSL

RPunway

aim

point

itch

(attitude)

WAi ir nc draft

configuration

Bank VOertical

Velocity

bstructions Engine health Airspace Fuel selection YawTrack Heading Fuel quantity RGurnway remaining ound speed Course Distance Waypoints Time - ETA/ETE

Mean Priority

Approach Exp Approach Nov
17

Figure 10. Landing priorities.

1234

RRunway

aim

point

Aunway

remaining

irspeed Wind ATri arc fraft configuration fic AGi er nport configuration eral Weather PT i r ta cf hfic / Other Comm (attitude) RPM (power) YawATC Comm Bank Altitude - AGL Engine health VOertical Velocity bstructions Altitude - MSL Track Heading Distance Fuel selection Airspace Fuel quantity Ground speed Course WTime - ETA/ETE aypoints

Mean Priority

Landing Exp Landing Nov
18

Figure 11. Preflight planning priorities.

1234

Fuel

quantity

WGeneral

Weather

ind

ATime

-

ETA/ETE

Obir sport

configuration

tructions

Altitude -

MSL

Airspace Waypoints DA i i src traft configuration ance Course Engine health Fuel selection ATC Comm Ground speed ATr latif tf uic / Other Comm de - AGL RPM (power) ARunway remaining irspeed Heading Traffic Track Vertical Velocity RPunway aim point itch (attitude) YawBank

Mean Priority

Preflight Planning Exp Preflight Planning Nov
19

Figure 12. In-flight planning priorities.

1234

Time

-

ETA/ETE

Distance Fuel quantity Waypoints General Weather Course Altitude - MSL Heading WindAA Tirport configuration C Comm Ground speed Airspace Airspeed Traffic RPM (power) Track Engine health Altitude - AGL TOr ba sffic / Other Comm tructions Fuel selection Bank APii trcraft configuration ch (attitude) YawVertical Velocity RRunway aim point unway remaining

Mean Priority

In-Flight Planning Exp In-Flight Planning Nov
20

Table 7. Major novice-experienced priority differences.

Phase Information Element

Higher Priority

by: p

Takeoff Vertical Velocity Nov .015

Climb Track Exp .025

Traffic/Other Comm Exp .043

Vertical Velocity Nov .016

Transition to Cruise Ground Speed Nov .046

Cruise RPM (power) Exp .032

Time - ETA/ETE Nov .011

Vertical Velocity Nov .043

Descent Traffic Exp .022

Traffic/Other Comm Exp .005

Course Exp .028

Airport Configuration Exp .013

Approach General Weather Exp .056

Landing Runway Aim Point Exp .027

Traffic/Other Comm Exp .005

In-Flight Planning ATC Comm Exp .043

Table 8. Highest priority information by phase of flight.

Flying Planning

Information Element TO Clmb TTC Cruse Desc App Land Num PP* IFP+ Num

Aircraft configuration XX XX 4 X 1

Engine health XXXXXXX 7 0

Fuel quantity X 1 X X 2

Fuel selection X X 2 0

RPM (power) XXXXXXX 7 0

Altitude - AGL XXXXXXX 7 0

Altitude - MSL X XXX 4 X X 2

Distance 0 X 1

Bank X XX 3 0

Pitch (attitude) XX XXX 5 0

Yaw X XX 3 0

Course XXXX 4 X 1

Heading XX X X 4 X 1

Track X XXX 4 0

Waypoints X 1 X X 2

Airspeed XXXXXXX 7 X 1

Ground speed X 1 X 1

Vertical velocity X X 2 0

Time - ETA/ETE 0 X X 2

Airport configuration XXX 3 X X 2

Runway aim point X X 2 0

Runway remaining X 1 0

ATC comm XXXXXXX 7 X 1

Traffic / other comm XX XXX 5 0

Obstructions XX XX 4 X 1

Traffic XXXXXXX 7 0

Airspace X XX 3 X X 2

General weather X XXXX 5 X X 2

Wind X XX 3 X X 2

* Preflight planning

+ In-flight planning
21

line with their capacity rather than with what might

be the ideal priority. Some information elements

show experience differences in more than one phase

of flight. Traffic/Other Communication consistently

receives higher priority ratings from the experienced

pilots. In contrast, novices assign higher priority

ratings to Vertical Velocity in Takeoff, Climb, and

Cruise. Perhaps experienced flyers focus on factors

such as pitch and power and let the vertical velocity

take care of itself, while novice flyers focus directly on

vertical velocity. Instructor pilots may find it useful

to consider these priority differences as they think

about what topics to discuss with their students.

Table 8 gives another view of the priorities in the

phases of flight. The information elements with a

mean priority rating of 2 or better are shown for each

phase of flight. The items that are critical in all of the

flying phases are often available in the central instrument cluster, but not all of the critical information

elements are found there. Some items have high

priority in planning, but not in flying. It may be

worth considering how to deal with planning versus

flying information as displays are designed.

Information Priorities and Workload

Considerable attention has been devoted to characterizing and assessing workload both under normal

operating conditions and under various unusual situations (Corwin, 1992; Gopher & Donchin, 1986;

Moray, Johanssen, Pew, Rasmussen, Sanders, &

Wickens, 1979). Pilots generally report that workload

is greater in some phases of flight than in others.

Takeoff, approach, and landing are usually identified

as the phases involving the highest workload under

normal operating conditions. The priority ratings we

collected in the present study offer an additional way

to measure workload. We propose that the average

priority rating for data within a phase of flight will

reflect workload, namely, the higher the average

priority rating, the higher the workload. Another

measure from our study is also worth examining for

its relation to workload: the proportion of the highest

priority ratings in the set of ratings for a phase of

flight. Presumably, the higher the proportion of

highest-priority (i.e., priority 1) information elements, the higher the workload.

To test these ideas, we examined previous studies

of workload in the literature. One study, Corwin

(1992), was particularly appropriate for our purposes

because that study assessed workload both in-flight

and post-flight for three simulator flights. One flight

was a 30-minute flight between two airports. A second 30-minute flight was similar, with the addition

of communications. The third 1.5-hour flight involved malfunctions with two diversions, an engine

failure, and a hydraulic system failure. Using the

Subjective Workload Assessment Technique (SWAT),

workload was assessed both in-flight and post-flight

for each of several phases of flight (takeoff, climb, top

of climb, cruise, top of descent, approach, and landing). These phases match up very well with the flying

phases we defined for our priority ratings (takeoff,

climb, transition to cruise, cruise, descent, approach,

and landing). The values of the various measures are

shown in Table 9, along with the correlations between these measures. Interestingly, the measures

derived from the priority ratings of the present study

correlate about as well with the measures from the

Corwin study as those measures correlate with each

other. SWAT measures were more highly correlated

with the percentages of priority ratings falling in the

top priority category than they were with the mean

priority ratings. This result suggests that workload

may derive primarily from high-priority data requirements, rather than from the total potential data

requirements.

While this is a rather appealing notion, it should

be kept in mind that this is based upon correlational

data, and thus no causality can be directly inferred

from these observations. A slightly stronger case can

be made, however, if one looks at additional sources

of data, i.e., the data requirements per flight phase

and the distribution of accidents by flight phase.

Accidents are distributed disproportionately across

phases of flight such that the frequency of incidents

and accidents on take-off and approach/landing is far

greater, by a significant amount, than the times of

exposure to these phases of flight. Although several

factors are thought to influence this distribution, it is

widely accepted that workload is higher during takeoff or approach than it is during level cruise. Examinations of information requirements by phase of

flight indicate that changes in aircraft configuration,

increases in communications requirements, increases

in traffic in the terminal area, and requirements for

maintaining separation all contribute to an elevated

workload during the approach and landing. It is

documented that more data are needed per unit time

during an approach and landing than are accessed

during a cruise flight segment. If we can assume that
22

higher-priority data are “required” whereas lowerpriority data are optional in some cases, then we can

forge a link between the percentage of information

elements rated as high-priority and the expected

workload resulting from the increased informationprocessing requirements.

Project 3: The Mental Organization of

Information Accessed in Flight

In this project, we used network scaling-methods

to analyze how pilots conceptually organize the information used in flight. First, we present some background on network scaling.

Network Analyses of Relatedness Data

The techniques used in the present analysis are

generally known as Pathfinder Network Scaling (see

Schvaneveldt, 1990). The method uses individuals’

judgments as a source from which to extract underlying network structures. This method has been used to

capture expert-novice differences in conceptual structure (USAF pilots – Schvaneveldt, et al., 1985; also

see Cooke & Schvaneveldt, 1988 and Rowe, Cooke,

Hall & Halgren, 1996), to assess student knowledge

(Goldsmith, Johnson, & Acton, 1991), and to analyze and design user-system interfaces (McDonald &

Schvaneveldt, 1988).

Graph Theory and Relations

Among Information Elements

The abstract model underlying the proposed model

of relationships among information elements is graph

and network structures. In mathematical graph theory,

a graph is an abstraction consisting of a set of nodes

and a set of pairs of the nodes (Harary, 1969). Each

such pair of nodes is called a link. The links can be

directed or undirected. The nodes represent some

entity (concept, document, individual, process, etc.)

and the links represent relationships between nodes.

A network is a graph with weights (or costs) associated with the links. We often expand these basic

definitions to include distinguishing different types

of nodes and different types of links. The present

application can benefit from the ability to systematically distinguish different node and link types. For

example, node types can be used to represent collections of related nodes, which, for some purposes, may

be treated as a single category (e.g., the various

parameters relating to the status of engines).

Table 9. Workload measures and correlations.

Phase

A

Exper.

Mean

Priority

Rating

B

Exper.

Percent

Top

Priority

C

Corwin

Basic

InFlight

D

Corwin

Basic

PostFlight

E

Corwin

Comm

InFlight

F

Corwin

Comm

PostFlight

G

Corwin

Malfn

InFlight

H

Corwin

Malfn

PostFlight

Takeoff 2.06 53.7% 24.9 25.6 24.5 27.8 20.0 16.3

Climb 2.15 41.6% 11.1 14.4 19.6 15.6 14.7 12.7

Trans'n to Cruise 2.50 24.4% 9.1 3.9 11.7 10.2 1.4 2.9

Cruise 2.30 28.9% 2.9 1.3 8.6 11.4 14.1 21.3

Descent 2.10 36.4% 16.8 16.9 15.2 16.8 13.1 18.2

Approach 1.74 59.1% 15.9 14.9 19.3 17.1 32.2 33.7

Landing 1.83 62.9% 30.4 26.5 29.4 29.0 46.4 59.9

Correlations ABCDE FGH

A -0.94 -0.67 -0.71 -0.72 -0.66 -0.89 -0.77

B -0.94 0.82 0.84 0.90 0.83 0.91 0.77

C -0.67 0.82 0.96 0.93 0.95 0.73 0.65

D -0.71 0.84 0.96 0.94 0.95 0.68 0.56

E -0.72 0.90 0.93 0.94 0.93 0.78 0.66

F -0.66 0.83 0.95 0.95 0.93 0.74 0.64

G -0.89 0.91 0.73 0.68 0.78 0.74 0.96

H -0.77 0.77 0.65 0.56 0.66 0.64 0.96
23

METHOD

Pathfinder scaling requires judgments of relatedness for the pairs of elements to be scaled. Pathfinder

produces a network showing the connections between the various elements. The network structures

also reveal clusters of elements that are interrelated.

Because the relatedness judgments require pilots to

evaluate all pairs of the elements, the number of

elements must be limited. We selected the information elements shown in Table 10 for study in this

project. To limit the number of items, we excluded

information more related to airports, while retaining the

items related to flying the aircraft and to communications.

Participants

Thirty-four certified pilots of varying age and

experience were used to obtain the relatedness rating

data. A number of subjects overlap from the priorityrating task to the relatedness-rating task. The participants were obtained from different areas around the

United States. To examine the impact of experience

on judgments of relatedness, we separated the pilots

into two groups based on their total flying time. The

hours of the 15 less experienced pilots ranged from

100 to 950 with a mean of 372 and a standard

deviation of 323. The 19 more experienced pilots had

a range of flying time from 1,000 to 20,400 hours

with a mean of 5,320 and a standard deviation of

5,143. Thirteen percent of the less-experienced pilots

held the Instrument Rating while 74% of the moreexperienced pilots were instrument rated.

Materials

The relatedness-rating task was conducted using

two Hewlett-Packard OmniBook 600c laptop computers. A MS/DOS program presented instructions

and all of the 231 pairs of the 22 information elements for relatedness ratings.

Design and Procedure

Collection of the relatedness data occurred over an

eight-month period. The participants completed a

demographic questionnaire (see Appendix 1). After

the demographic form was completed, the participants were directed to the computer for the relatedness ratings. After reviewing the 22 information

elements to be rated, the pilots rated the relatedness

for each of the pairs of elements. The 231 pairs were

presented in a different random order for each participant. At the conclusion of the rating task, the

project was discussed with the pilots, and they were

encouraged to provide feedback about the rating task

and the objectives of the study.

RESULTS AND DISCUSSION

The Pathfinder network derived from the average

of all of the relatedness rating data is shown in Figure

13. The figure generally reveals which items of information are most closely related to which other items

by virtue of the links in the figure. Linked nodes

indicate a close relationship between the items linked.

Judged by the number of links connected to the

nodes, Course, Altitude, Airspeed, and Power are the

central nodes in the figure. Arguably, these items are

also central to the flight task. Airspeed is required for

lift. Power is essential to produce Airspeed. Course

and Altitude are essential parts of navigation. The

other information elements all tie to these core elements to give a more complete picture of the information required in flight. Thus, the picture of the

organization of the information revealed by the network analysis generally corresponds with intuitive

impressions, as it is intended to do.

Table 11 gives more detail about the close connections among information elements. The table shows

the most closely related pairs of items based on

judged relatedness (rel) and/or the frequency with

which the pairs are linked in the networks of 34

individual pilots (net).

Table 10. The 22 information elements used for

relatedness ratings.

Aircraft Configuration Airspeed

Engine Health Ground Speed

Fuel Status Vertical Velocity

RPM (power) Time (ETA/ETE)

Altitude ATC Comm

Distance to Waypoint Traffic Comm

Bank Obstructions

Pitch Traffic

Yaw Airspace

Course General Weather

Track Wind
24

Aircraft

Configuration Engine

Health

Fuel

Status

RPM

(power)

Altitude

Distance to

Waypoint

Bank

Pitch

Yaw

Course

Track

Airspeed

Ground

Speed

Vertical

Velocity

Time

(ETA/ETE)

ATC Comm

Traffic

Comm

Obstructions

Traffic

Airspace

General

Weather

Wind

Figure 13. The pathfinder network for average data (q=2, r=8).
25

Table 11. Most highly connected pairs of information elements.

Net Rel Pair Net Rel Pair

32 8.8 Time (ETA/ETE) Ground Speed 16 6.5 Wind Distance to Waypoint

31 8.6 Wind Ground Speed 11 7.0 General Weather Altitude

30 8.6 Time (ETA/ETE) Distance to Waypoint 13 7.0 Track Distance to Waypoint

30 8.4 Obstructions Altitude 11 7.0 Ground Speed Track

29 8.4 RPM (power) Engine Health 15 7.0 Vertical Velocity Aircraft Configuration

28 8.4 Track Course 11 6.9 Wind Altitude

27 8.3 Traffic Traffic Comm 13 6.9 Ground Speed Course

27 8.4 Airspace ATC Comm 13 6.9 ATC Comm Altitude

27 8.1 Wind Track 13 6.8 Ground Speed Aircraft Configuration

26 8.2 Traffic Comm ATC Comm 9 6.8 General Weather Course

25 8.3 Airspeed RPM (power) 15 6.3 Pitch Bank

25 8.3 Time (ETA/ETE) Airspeed 15 5.5 Yaw Bank

25 8.3 Wind Time (ETA/ETE) 14 6.8 Obstructions Vertical Velocity

25 8.2 Wind General Weather 13 6.7 Traffic Altitude

24 7.9 Airspeed Pitch 10 6.7 Course Fuel Status

24 8.2 Vertical Velocity Pitch 7 6.7 Airspeed Altitude

24 8.0 Time (ETA/ETE) Fuel Status 14 6.6 Pitch Aircraft Configuration

24 7.8 Airspace Traffic 13 6.6 Airspeed Engine Health

23 7.9 Airspeed Aircraft Configuration 11 6.6 Airspeed Fuel Status

23 8.4 Ground Speed Airspeed 12 6.6 Pitch RPM (power)

22 7.8 Airspace Altitude 7 6.6 Traffic Course

21 7.5 Distance to Waypoint Fuel Status 10 6.6 General Weather Time (ETA/ETE)

21 7.6 Time (ETA/ETE) Track 9 6.5 General Weather Airspace

20 8.0 Airspace Traffic Comm 13 6.4 Time (ETA/ETE) Engine Health

20 7.7 Traffic ATC Comm 12 6.3 Fuel Status Engine Health

20 7.7 Wind Course 10 6.3 Ground Speed Altitude

20 6.7 Wind Airspeed 11 6.3 Ground Speed Pitch

19 7.9 Wind Fuel Status 12 6.3 ATC Comm Course

19 7.9 Ground Speed RPM (power) 13 6.3 RPM (power) Aircraft Configuration

19 7.8 Obstructions Course 9 6.3 Course Altitude

17 7.7 Course Distance to Waypoint 10 6.3 Obstructions Pitch

19 7.7 Altitude RPM (power) 8 6.2 Traffic Comm Altitude

19 7.6 Ground Speed Distance to Waypoint 10 6.2 Airspace Obstructions

18 7.6 Airspeed Distance to Waypoint 6 6.2 General Weather Ground Speed

18 7.6 Vertical Velocity RPM (power) 12 6.1 Altitude Engine Health

17 7.4 Vertical Velocity Altitude 10 6.1 Airspeed Bank

17 7.3 Time (ETA/ETE) RPM (power) 10 6.1 Airspace Airspeed

17 7.3 Obstructions Track 6 6.1 Altitude Aircraft Configuration

17 7.3 Pitch Altitude 10 6.1Distance to Waypoint RPM (power)

16 7.2 Time (ETA/ETE) Course 8 6.1 General Weather Obstructions

14 7.2 Airspace Course 9 6.0 Airspace Track

12 7.2 RPM (power) Fuel Status 9 5.9 Ground Speed Fuel Status

12 7.2 Vertical Velocity Airspeed 11 5.9 ATC Comm Time (ETA/ETE)

13 7.2 General Weather Fuel Status 9 5.9 Airspace Ground Speed
26

Clusters of Information Elements

The results of a hierarchical cluster analysis are

shown in Figure 14. Bold lines surrounding the

information elements show clusters. The cluster analysis reveals larger groupings of the information

elements, which may be useful in suggesting organizations of information displays. The clusters generally identify meaningful hierarchical groupings of the

information elements.

The information clusters suggest how pilots reach

decisions by considering multiple factors. For example, the clustering of Distance to Waypoint and

Fuel Status illustrates the decision-making about the

likelihood of reaching a waypoint. This cluster is

closely connected to the cluster including Time and

Ground Speed and Wind, reflecting the relevance of

all of these factors in decision-making in flight.

Presenting such information in an arrangement and

a format that facilitates decision-making should be of

considerable value to pilots.

Experience and Networks

We examined the differences between the network

derived from the ratings of novice pilots and that

derived from the ratings of experienced pilots. The

networks were generated with r = ¥ and q = 2 which

yields networks with the maximum number of links

given ordinal data (Schvaneveldt, Durso, & Dearholt,

1989). The experienced-pilots’ network had 28 links

while the novices’ network had 29. In general, the

similarities in the two networks were more striking

than the differences. The two networks had 21 links

in common. The differences between the networks

are shown in Figure 15. Many were simply differences in the linking among three items. Such differences in triples of items are not particularly significant

because the three items are joined in both networks

albeit with different links. It is possible, however,

that the different linking points to subtle differences

in the way novices and experienced pilots think about

the information involved. In Figure 15, the triangles

Figure 14. Hierarchical clusters in the pathfinder network.

Aircraft

Configuration Engine

Health

Fuel

Status

RPM

(power)

Altitude

Distance to

Waypoint Bank

Pitch

Yaw

Course

Track

Airspeed

Ground

Speed

Vertical

Velocity

Time

(ETA/ETE)

ATC Comm

Traffic

Comm

Obstructions

Traffic

Airspace

General

Wind Weather
27

shown include the links shared by both networks to

highlight the triangles. The other link differences

cannot be accounted for by such triples suggesting

greater differences in the perception of relatedness

between the two groups of pilots.

CONCLUSIONS, APPLICATIONS,

RECOMMENDATIONS, AND CAVEATS

The data presented in this report provide a picture

of how information functions in the National Airspace System, including how the information is prioritized and organized by pilots. The views of information

priorities show how priorities change with phases of

flight and how pilot experience relates to the perception

of priorities. The views of information organization

reveal some of the ways pilots inter-relate the various

elements of information. It is difficult to determine

the extent to which the experience differences in this

study are due to experience per se as opposed to

selection that occurs over the flying careers of pilots.

We know that pilot attrition may be due to a number

of factors, ranging from choosing to stop flying to

having a fatal accident. Is it that the pilots who

endure have different perceptions of priorities to

begin with, or do pilots learn to develop these priorities as a result of experience? It may well be that both

of these factors operate. A longitudinal study would

provide some answers to these questions. Regardless

of how the experience issue is resolved, the data

reported here should be of value in designing information display systems.

Figure 15. Differences in novice and experienced pilots’ networks.

Altitude

Bank

Course

General

Weather

Ground

Speed

Pitch

RPM

(power)

Time

(ETA/ETE)

Track

Vertical

Velocity

Yaw Wind Aircraft

Configuration

Distance

to Waypoint

Airspace

Airspeed

Traffic

Traffic

Comm

ATC

Comm

B

E N

N E

B

E

N B

N

E

B

E

B

E N

N

N

N

E

E - Link in Experienced Network

N - Link in Novice Network

B - Link in Both Networks
28

This study has focused on information priorities,

information organization, and information sources.

There are certainly several other factors that are

important in the design of information displays,

including issues of perception, format, analog vs.

digital, etc. As new display solutions are sought, it is

important to keep in mind the value of redundancy.

Limited panel real estate places a severe constraint on

just how much can be placed before the pilot, but

some solution to the problem of providing alternative sources of information must be found in the

interest of safety. We have not emphasized this issue

in the report, but our analysis of sources using the

instruments found in most general aviation aircraft

shows the redundancy existing in the present system.

New systems must strive to maintain redundancy to

ensure pilot confidence.

There are numerous ways that data on pilot information organization and information priorities

could be used to enhance the safety of operations in

the national airspace. These data can potentially be

used to position instrumentation in the cockpit,

design integrated cockpit information systems (ICIS),

restructure the logic of and displays for flight management systems (FMS), modify communications

between pilots and ATC/ATM (both voice and

datalink), reduce workload, and even to refine methods and curricula for flight instruction.

Instrumentation Layout and Integrated

Cockpit Information Systems

Given that one has a good idea of how information

elements are prioritized and related in the perception

of the pilot, it should be helpful to use these findings

to determine how data are presented in the cockpit in

a way that would enhance processing of those data,

both in terms of speed and accuracy. For example,

one can see from the hierarchical clusters shown in

Figure 14 that fuel status and distance to waypoint

are clustered, and are closely tied to ETA/ETE and

ground speed. This might lead to recommendations

that displays for related parameters be closely grouped

or, better yet, integrated for more effective presentation. Thus, one could argue the case that one way to

facilitate data assimilation would be to present a

graphic showing aircraft location on a plan-view

map, with range rings or vectors indicating waypoints/

destination and instantaneous “fuel” range, both

absolute and with 45-minute reserve. The clustered

data are now effectively presented in an integrated

display that allows direct and rapid interpretation of

the situation.

Flight-Management Systems

Several studies (i.e., Funk et al., 1999; Degani,

Shafto and Kirlik, 1999) have found that pilots

perceive flight-management systems as problematic

in a number of ways. Two contributing factors to this

attitude have been identified: (1) the organization

and grouping, or lack thereof, of functions and data;

and (2) the failure of the system to adequately communicate intent to the pilot. The first is probably the

most amenable to modification using clustering/

connections data, and some efforts are presently

underway (Riley, DeMers, & Misiak, 1998) to restructure the data-entry interface more along the

lines of natural language, aping the way in which

instrument clearances are verbally delivered and read

back. This, to some degree, involves a reorganization

of the data in ways that are more consistent with pilot

mental models and constructs. The present data can

be used in such endeavors to group data for input to

and retrieval from the FMS in a more natural and

expected fashion, which should reduce entry times

and errors.

The second problem of communicating intent

may also involve restructuring reports of FMS activity in such a way as to be more easily associated, by the

pilot, with the greater context of the flight. Although

there is a significant argument that the best FMS

should be one in which there are no modes but,

rather, an apparently seamless operation from one

flight regime to the next, the FMS intent still must be

communicated in the context of current flight conditions (i.e., messaging priorities are likely to change

from takeoff through cruise to landing, and the

system should organize and present action-related

data in the appropriate context and with the appropriate priority weighting).

Controller-Pilot Communications and

Datalink

Our argument is that the organization of data is

important across the board to achieve efficient, timely,

and accurate responses by both the airborne and

ground-based systems. Thus, the same groupings and

organization applied to flight-control data can also

be applied to many other forms of flight-related data
29

(e.g., Riley et al., 1998). Domain-specific analyses

will be necessary, however, to verify that each of the

users is receiving data in formats appropriate to their

task. This should apply to data regarding weather,

airports, traffic, obstacles, and clearances, to name a few.

Flight Instruction Curricula

One question that has not been addressed directly

here is the issue of how data should be organized in

the most abstract sense versus how data are organized.

It is clear that the actual task structure (how we fly

aircraft and how the data are presented as a function

of aircraft certification requirements) largely determines how instruction is structured.

However, it is not at all clear that the way in which

data are presently organized and presented is necessarily optimal. The question, which could be pursued

here, is what organization may be imposed upon

fundamental information regarding aircraft flight

(rate of travel, altitude, heading, position relative to

desired location in space or final goal) by nonpilots as

a function of their concept of the natural environment and the inherent implied organization via physical features and properties. Cases have already been

documented (Beringer, 1978) in which pilot behaviors were being established during training that were

counter to nonpilot stereotype responses, leaving the

door open for reversion to those now inappropriate

behaviors under stress.

The same can be asked of the organization of data

for the flight task: To what degree may we be distorting any inherent organizational structures that exist

in the general population in subsequent flight training, and how might task and data structures be modified

to take better advantage of extant knowledge?

Regardless of how these issues are resolved, flight

instructors may benefit from knowing how more and

less experienced pilots differ in information priorities

and in the mental organization of that information.

An Example: Applying the Data to

Display Allocation/Design

One specific example of how these data might be

applied can be found in the design of display suites

for the new GA glass cockpit, which are expected to

include a Primary Flight Display PFD) and a Multifunction Display (MFD). The data demonstrate that

pilots require different information sets depending

on the mode of flight. Thus, it is expected that

although common information may be present across

two or more modes, the relative importance of the

information will change. Consequently, a displaydecluttering scheme should take into account the

most salient pieces of data and preserve those over

other less task-relevant data.

Assume, for example, that we are allocating display

space in such a way that we can depict only the six

highest-priority items (the assumption of display

limitations is real; the choice of 6 is artificial). Concentrating on takeoff and drawing from the experienced-pilots’ rankings presented in Figure 4, we

would select RPM/power, wind, engine health, aircraft configuration, airspeed, and general weather for

presentation. Next, we can examine Figure 14 to

determine those elements that naturally cluster together and may be suited for integrated presentation.

We find that airspeed and aircraft configuration

cluster at a high level, with RPM/power and engine

health forming another high-level cluster, with these

two clusters in close proximity and with a direct link.

Wind and general weather can be seen as another

cluster with a link to the first (the one containing

airspeed). This suggests that there could be three

display subgroups derived from these data, each consisting of two data variables. The degree to which the

individual pieces can be woven into an integrated

presentation will, of course, affect formatting, and it

should be kept in mind that the data in this report do

not specifically designate a particular format. Rather,

the data reveal the priority of information elements

from the pilot’s perspective.

One can then apply some human factors engineering in this example, however, and see that an integrated presentation of applied power or thrust that

includes some status indicator depicting engine health

would be a good start. [Example: A bargraph presentation showing percent thrust with color of the

bargraph or texture fill indicating the status of the

power plant.] Thus, one component of the display is

defined. Second, a component presenting airspeed

and aircraft configuration (airspeed is an index of

when gear and flaps can be extended/retracted) would

be a next step, with direct graphical indices or icons

representing the status of gear/flaps/spoilers and somehow directly associated with the airspeed scale or indication. Finally, one could have a third representation
30

depicting surface wind graphically related to direction of travel (expected to be runway heading on

takeoff). These elements should therefore satisfy the

requirements given the restrictions assumed.

One may have the opportunity, however, to approach the display problem from the other direction

given effective integration techniques and boundless

display area. This approach would focus on providing all of the information elements ranked in the first

two categories (1 = critical and/or frequently accessed; 2 = important and/or usually accessed). Going back to our previous example, one might then

choose to include any data that achieved a ranking

value of 2.5 or smaller (everything from 1’s all of the

time to 2’s half of the time and 3’s half of the time).

This approach would then cause the first 22 items (as

rated by experienced pilots) in Figure 4 to be included for display-system presentation (all the way

down to “airport configuration”). This approach

would be more suited to selecting data for the entire

display suite, as opposed to selecting items for a small

subset of the display system. Grouping would be

pursued as previously by looking at clusters and links.

Again, it should be kept in mind that the exact form

of the data displays will be governed by established

human factors display-formatting principles, guidelines, and ultimately the standards presently in force

for the specific systems hosting the displays.

Prospectus

In this research, the emphasis has been on identifying information priorities and information organization. It would be valuable to take the work reported

here as a starting point for an examination of the kind

of decisions pilots make in the course of flight. In

particular, the decisions could be examined from the

perspective of the information that goes into the

decisions. Our analysis of clusters of information

elements points in that direction, but it would be

informative to pursue the involvement of information in decisions in greater depth.

Finally, this research effort has focused on normal

flight operations in contrast to unusual situations or

emergencies. Clearly, the work is incomplete without

including these situations. We are in the process of

remedying this lack in a second investigation that will

focus on a related analysis of information priorities in

emergencies.

REFERENCES

Andre, A. D., & Wickens, C. D. (1991). Display

formatting techniques for improving situation

awareness in the aircraft cockpit. International

Journal of Aviation Psychology, 1, 205-18.

Beringer, D.B. (1978). Collision avoidance response

stereotypes in pilots and nonpilots. Human Factors, 20(5), 529-36.

Cooke, N. M., & Schvaneveldt, R. W. (1988). Effects

of computer programming experience on network representations of abstract programming

concepts. International Journal of Man-Machine

Studies, 29, 407-27.

Corwin, W. H. (1992). In-flight and postflight assessment of pilot workload in commercial transport

aircraft using the subjective workload assessment

technique. International Journal of Aviation Psychology, 2(2), 77-93.

Degani, A., Shafto, M., & Kirlik, A. (1999). Modes in

human-machine systems: constructs, representation, and classification. International Journal of

Aviation Psychology, 9(2), 125-38.

Funk, K., Lyall, B., Wilson, J., Vint, R., Niemczyk, M.,

Suroteguh, C., & Owen, G. (1999). Flight deck

automation issues. International Journal of Aviation Psychology, 9(2), 107-24.

Goldsmith, T. E., Johnson, P. J., Acton, W. H. (1991).

Assessing structural knowledge. Journal of Educational Psychology, 83, 88-96.

Gopher, D., & Donchin, E. (1986). Workload–An

examination of the concept. In K. Boff, L.

Kaufman, & J. Thomas (Eds.), Handbook of Perception and Human Performance, Vol II (pp. 41-1

- 41-49). New York: Wiley.

Harary, F. (1969). Graph theory. Reading, MA: AddisonWesley.

McDonald, J. E., & Schvaneveldt, R. W. (1988). The

application of user knowledge to interface design.

In R. Guindon (Ed.), Cognitive Science and Its

Applications for Human-Computer Interaction,

Hillsdale: Erlbaum.
31

Moray, N., Johanssen, J., Pew, R. W., Rasmussen, J.,

Sanders, A. F., & Wickens, C. D. (1979). Mental

workload: Its theory and measurement. New York:

Plenum Press.

Nagel, D. C. (1988). Human error in aviation operations. Chapter 9 in E. Wiener & D. Nagel (Eds.),

Human Factors in Aviation. New York: Academic

Press, pp. 263-303.

Riley, V., DeMers, B., and Misiak, C. (1998). The

cockpit control language: a pilot-centered avionics interface. In Proceedings of the 1998 International Conference on Human-Computer Interaction

in Aeronautics. Montreal: May 27-29. (In press)

Roscoe, S. N. (1980). Aviation Psychology. Ames, IA:

The Iowa State University Press.

Rowe, A. L., Cooke, N. J., Hall, E. P. & Halgren, T. L.

(1996). Toward an online assessment methodology: Building on the relationship between knowing and doing. Journal of Experimental Psychology:

Applied, 2, 31-47.

Sarter, N. D., & Woods, D. D. (1992). Pilot interaction with cockpit automation: Operational experience with the flight management system. International Journal of Aviation Psychology, 2, 303-21.

Schvaneveldt, R. W. (Ed.) (1990). Pathfinder associative networks: Studies in knowledge organization.

Norwood, NJ: Ablex.

Schvaneveldt, R. W., Durso, F. T., & Dearholt, D. W.

(1989). Network structures in proximity data. In

G. Bower (Ed.), The psychology of learning and

motivation: Advances in research and theory, Vol.

24 (pp. 249-84). New York: Academic Press.

Schvaneveldt, R., Durso, F., Goldsmith, T., Breen, T.,

Cooke, N., Tucker, R., & DeMaio, J. (1985).

Measuring the structure of expertise. International

Journal of Man-Machine Studies, 23, 699-728.

Sutcliffe, A. (1997). Task-related information analysis.

International Journal of Human-Computer Studies, 47, 223-37.
32
A1

APPENDIX A

Demographic Questionnaire

Aviation Research Group

New Mexico State University

1. Please estimate your logged hours in each of the following categories of aircraft:

Single Engine Land ___________

Multi Engine Land ___________

Other ___________ Types _____________________

2. Please list the year in which you obtained the following ratings or certificates:

Private Pilot ___________

Instrument ___________

Commercial ___________

CFI ___________

CFII ___________

ATP ___________

3. Approximately how many hours have you logged in the last 12 months?___________

4. How, where, and when did you first learn to fly?

Year Location

___ Individual Flight Instructor _____ ______________________________

___ Civilian Flight School _____ ______________________________

___ Military Flight School _____ ______________________________

___ Other _____ ______________________________
A2
B1

APPENDIX B

Instructions for Priority Ratings and the Priority Rating Form

Aviation Research Group

New Mexico State University

Las Cruces, NM

Roger Schvaneveldt, Director

The Priorities and Organization of Information

Accessed by Pilots in Various Phases of Flight

Task Description

We are developing an analysis of the priorities associated with various kinds of information in various phases of

flight. At present, we are focusing on the priorities for normal operations as opposed to emergencies. You will

greatly assist us in this effort by providing us with your judgments of information priorities.

On the next page, you will find some questions about your experience in aviation. These data will help us

compare the priorities of pilots with varying experience. On the page following is a list of the information

elements along with our definitions of these elements. Study the list so you will understand what we are trying to

get at with your priority ratings.

Following the list is a sheet with the information elements on the rows and phases of flight on the columns. You

are to enter your priority ratings in the cells. Most pilots find it best to work down each column rating the

importance of the element for that phase of flight. We have also included two planning phases: Preflight

Planning and In-flight Planning. Because you can give priorities associated with planning in these two columns,

please rate the importance of the information elements in the flying phases relative to flying as opposed to

planning. The flying phases include: Takeoff, Climb, Transition to Cruise, Cruise, Descent, Approach, and

Landing. Try to assign priority ratings that cover both IFR and VFR flights. That is, give the highest priority

for the information element considering both IFR and VFR flights.

Use priority ratings as follows:

Priority Description

1 Critical and/or Frequently Access

2 Important and/or Usually Accessed

3 Relevant and/or Sometimes Accessed

blank Not Relevant or Rarely Accessed

If you have comments or questions, you can reach Dr. Schvaneveldt via e-mail: schvan@crl.nmsu.edu. You may

also write any comments below or on the back.

Thank you very much for your assistance.
B2

Priority Rating Form

Information Element

Preflight

Planning Takeoff Climb

Transition

to Cruise Cruise

In Flight

Planning Descent Approach Landing

Aircraft configuration

Engine health

Fuel quantity

Fuel selection

RPM (power)

Altitude - AGL

Altitude - MSL

Distance

Bank angle

Pitch (attitude)

Yaw

Course

Heading

Track

Waypoints

Airspeed

Ground speed

Vertical Velocity

Time - ETA/ETE

Airport configuration

Runway aim point

Runway remaining

ATC Communications

Traffic / other Comm

Obstructions

Traffic

Airspace

General Weather

Wind
C1

APPENDIX C

 Procedure for Collecting Relatedness Ratings

Collection of the relatedness data occurred over an eight-month period. The participants completed a

demographic questionnaire (see Appendix A). After the demographic form was completed, the participants were

directed to the computer for the relatedness ratings. The first screen consisted of these instructions:

Greetings, thank you for participating in our study of the information pilots need in the course of flight. In

other parts of the study, we have determined how pilots view the priorities of various elements of information

in each phase of flight. Now we are trying to characterize how pilots mentally organize flight-related

information. What we need from you in your judgments about how closely related the different elements of

information are to each other.

We will show you all the pairs of the elements listed below, and for each pair, you should enter a number from

1 to 9 indicating how related you think the two elements in each pair are. A rating of 9 indicates that two

information elements are very closely related, and a rating of 1 indicates that two information elements are not

very related at all. Feel free to ask if you have any questions.

The second screen gave a list of the 22 information elements (see Table 10).

The appearance of the rating task is shown in Figure 16. Ratings were selected by pressing a number key (1 to

9); the choice could be changed until the space bar was pressed when the next pair would be presented. The 231

pairs were presented in a different random order for each of the participants.

Figure 16. The Relatedness Rating Screen

The final screen presented a short debriefing:

That’s it. Thanks again for your help. We hope to contribute to improving aviation with this work. We

appreciate help from pilots such as yourself to accomplish our objectives.
C2