July 2001

NASA/TM-2001-211034

Cognitive Task Analysis of Business Jet

Pilots’ Weather Flying Behaviors:

Preliminary Results

Kara Latorella

Langley Research Center, Hampton, Virginia

Rebecca Pliske, Robert Hutton, and Jason Chrenka

Klein Associates Inc., Fairborn, Ohio
The NASA STI Program Office ... in Profile

Since its founding, NASA has been dedicated

to the advancement of aeronautics and space

science. The NASA Scientific and Technical

Information (STI) Program Office plays a key

part in helping NASA maintain this important

role.

The NASA STI Program Office is operated by

Langley Research Center, the lead center for

NASA’s scientific and technical information.

The NASA STI Program Office provides

access to the NASA STI Database, the largest

collection of aeronautical and space science

STI in the world. The Program Office is also

NASA’s institutional mechanism for

disseminating the results of its research and

development activities. These results are

published by NASA in the NASA STI Report

Series, which includes the following report

types:

• TECHNICAL PUBLICATION. Reports

of completed research or a major

significant phase of research that

present the results of NASA programs

and include extensive data or theoretical

analysis. Includes compilations of

significant scientific and technical data

and information deemed to be of

continuing reference value. NASA

counterpart of peer-reviewed formal

professional papers, but having less

stringent limitations on manuscript

length and extent of graphic

presentations.

• TECHNICAL MEMORANDUM.

Scientific and technical findings that are

preliminary or of specialized interest,

e.g., quick release reports, working

papers, and bibliographies that contain

minimal annotation. Does not contain

extensive analysis.

• CONTRACTOR REPORT. Scientific and

technical findings by NASA-sponsored

contractors and grantees.

• CONFERENCE PUBLICATION.

Collected papers from scientific and

technical conferences, symposia,

seminars, or other meetings sponsored

or co-sponsored by NASA.

• SPECIAL PUBLICATION. Scientific,

technical, or historical information from

NASA programs, projects, and missions,

often concerned with subjects having

substantial public interest.

• TECHNICAL TRANSLATION. Englishlanguage translations of foreign

scientific and technical material

pertinent to NASA’s mission.

Specialized services that complement the

STI Program Office’s diverse offerings

include creating custom thesauri, building

customized databases, organizing and

publishing research results ... even

providing videos.

For more information about the NASA STI

Program Office, see the following:

• Access the NASA STI Program Home

Page at http://www.sti.nasa.gov

• E-mail your question via the Internet to

help@sti.nasa.gov

• Fax your question to the NASA STI

Help Desk at (301) 621-0134

• Phone the NASA STI Help Desk at

(301) 621-0390

• Write to:

NASA STI Help Desk

NASA Center for AeroSpace Information

7121 Standard Drive

Hanover, MD 21076-1320
National Aeronautics and

Space Administration

Langley Research Center

Hampton, Virginia 23681-2199

July 2001

NASA/TM-2001-211034

Cognitive Task Analysis of Business Jet

Pilots’ Weather Flying Behaviors:

Preliminary Results

Kara Latorella

Langley Research Center, Hampton, Virginia

Rebecca Pliske, Robert Hutton, and Jason Chrenka

Klein Associates Inc., Fairborn, Ohio
Available from:

NASA Center for AeroSpace Information (CASI) National Technical Information Service (NTIS)

7121 Standard Drive 5285 Port Royal Road

Hanover, MD 21076-1320 Springfield, VA 22161-2171

(301) 621-0390 (703) 605-6000

Acknowledgments

We gratefully acknowledge the business jet pilots who served as subject

matter experts for this investigation, Mr. James P. Chamberlain's helpful

comments on the simulation interview scenario, and the useful reviews of

the manuscript provided by Mr. Chamberlain and Dr. Lance J. Prinzel.
iii

Abstract

This report presents preliminary findings from a cognitive task analysis

(CTA) of business aviation piloting. Results describe challenging

weather-related aviation decisions and the information and cues used to

support these decisions. Further, these results demonstrate the role of

expertise in business aviation decision-making in weather flying, and

how weather information is acquired and assessed for reliability. The

challenging weather scenarios and novice errors identified in the results

provide the basis for experimental scenarios and dependent measures to

be used in future flight simulation evaluations of candidate aviation

weather information systems. Finally, we analyzed these preliminary

results to recommend design and training interventions to improve

business aviation decision-making with weather information. The

primary objective of this report is to present these preliminary findings

and to document the extended CTA methodology used to elicit and

represent expert business aviator decision-making with weather

information. These preliminary findings will be augmented with results

from additional subjects using this methodology. A summary of the

complete results, absent the detailed treatment of methodology provided

in this report, will be documented in a separate publication.
iv
v

Table of Contents

1.0 Introduction------------------------------------------------------------------------------------------------------ 1

1.1 Weather Flying------------------------------------------------------------------------------------------- 1

1.2 Business Aviation Operations ------------------------------------------------------------------------- 1

1.3 Cognitive Task Analysis-------------------------------------------------------------------------------- 4

2.0 Methods ---------------------------------------------------------------------------------------------------------- 6

2.1 Participants ----------------------------------------------------------------------------------------------- 6

2.2 Protocol --------------------------------------------------------------------------------------------------- 7

2.3 Methods of Analysis and Representation ------------------------------------------------------------ 9

3.0 Results ---------------------------------------------------------------------------------------------------------- 12

3.1 General Observations --------------------------------------------------------------------------------- 12

3.2 Cognitive Demands ----------------------------------------------------------------------------------- 12

3.3 Indicators of Expertise -------------------------------------------------------------------------------- 17

3.4 Weather Information Sources Model --------------------------------------------------------------- 17

3.5 Assessing Weather Information Reliability ------------------------------------------------------- 19

3.6 Scenarios and Measures for AWIN Simulation Experiments----------------------------------- 20

3.7 Design and Training Interventions ------------------------------------------------------------------ 21

4.0 Conclusions---------------------------------------------------------------------------------------------------- 25

5.0 References ----------------------------------------------------------------------------------------------------- 26

6.0 Acronyms & Abbreviations --------------------------------------------------------------------------------- 28

7.0 Appendices ---------------------------------------------------------------------------------------------------- 29

7.1 Appendix A: ACTA Toolkit Methods------------------------------------------------------------- A-1

7.2 Appendix B: Preliminary Screening Questionnaire --------------------------------------------- B-1

7.3 Appendix C: Phase 1 Interview Guide -------------------------------------------------------------C-1

7.4 Appendix D: Weather Products Survey ----------------------------------------------------------- D-1

7.5 Appendix E: Phase 2 Interview Guide ------------------------------------------------------------ E-1

7.6 Appendix F: Simulation Interview Scenario and Materials ------------------------------------ F-1

7.7 Appendix G: Post-Experiment Questionnaire---------------------------------------------------- G-1

7.8 Appendix H: Individual Cognitive Demands Tables-------------------------------------------- H-1

7.9 Appendix I: Summary Cognitive Demands Table & Information Requirements ----------- I-1

7.10 Appendix J: Simulation Interview Summary Tables --------------------------------------------- J-1

7.11 Appendix K: Critical Decision Incident Tables-------------------------------------------------- K-1

7.12 Appendix L. "Scenarios from Hell" Incidents----------------------------------------------------- L-1

7.13 Appendix M: Scenario Elements and Behavioral Markers------------------------------------ M-1

List of Figures

Figure 1. Task Diagram: Phases of Flight............................................................................................... 8

Figure 2. Sources of Business Aviation Weather Information. .............................................................. 18

List of Tables

Table 1. Aircraft Types in the NBAA Fleet ............................................................................................ 2

Table 2. Participant Information............................................................................................................. 6
vi

Table 3. Simulation Interview Materials................................................................................................. 9

Table 4. An Example of an Annotated Incident .................................................................................... 11
1

1.0 Introduction

On February 12 of 1997, President Clinton called for a five-fold reduction in the rate of fatal aviation

accidents within a decade, and directed NASA to support this national safety goal. Recognizing that

weather contributes to approximately thirty percent of aviation accidents, Aeronautics Safety Investment

Strategy Team (ASIST) workshops suggested that a NASA program should focus on improving the

quality of National Airspace System (NAS) users' weather information (NASA 1997). NASA created the

Aviation Safety program to accomplish this presidential aviation safety initiative. This program includes

the Weather Accident Prevention project, which addresses a variety of the weather-related safety

initiatives suggested by ASIST workshops. The objective of the constituent Aviation Weather

Information (AWIN) element is, specifically, to improve pilots' weather situation awareness and the

quality of weather-related flight decisions. This research project supports AWIN's objective by

investigating decision-making of business aviators in weather flying.

1.1 Weather Flying

It is not surprising that weather is a major contributor to aviation accidents. Weather represents perhaps

the most dynamic and least predictable aspect of the NAS environment. The dynamics of weather

phenomena are multi-dimensional. Its characteristics vary in three-dimensional space, over time, and in

intensity. Pilots' views on weather phenomena are limited by the weather information products that are

available on the ground, and dramatically limited by the far fewer available once airborne. A pilot's

weather situation awareness is further constrained by the limited usability of available information (e.g.,

coded METAR text). The reliability of forecasted weather information is diminished by the

unpredictability of the phenomena. Both forecasted and immediate weather information are only as

reliable as the sensors that obtain and, in some cases, algorithms that interpret raw atmospheric data.

Thus, even perfect information, perfectly perceived does not guarantee safe flight in difficult weather

conditions. Finally, as aviation accidents and incidents result from a confluence of events, even perfect

weather information that is understood perfectly must be considered in light of its implications for a

particular flight. Weather information is only one aspect of the multifaceted decisions pilots must make.

Pilots must integrate their understanding of weather conditions with understanding of the terrain, aircraft

performance characteristics, airport facilities, their own skills and capabilities, airspace constraints, etc. to

arrive at safe decisions in situ. Underlying these concerns is the normal multitasking nature of aviating,

and the commensurate undulation of pilot workload. Pilots' susceptibility to both overload and

complacency therefore further exacerbates problems of weather information interpretation, and aviation

decision-making with weather information. This investigation focuses on the weather flying knowledge,

skills and decisions of business jet pilots to better understand how to improve, and improve the use of,

aviation weather information.

1.2 Business Aviation Operations

Business aviation has an excellent safety record. Since the mid-1980's, accident rates among

corporate/executive operators have been lower than those of other operators in FAR part 135 operations

and of any segment in general aviation, and comparable to FAR part 121 operators (NBAA 2000a).

Despite this exemplary safety record, AWIN, as well as other NASA Langley efforts (Schutte & Willshire

1996), have recognized that business aviation offers several advantages for developing and implementing

advanced flight deck concepts. The variety of business jet operational contexts serves as a broad basis

from which to extend technologies to both general aviation and transport aviation. This breadth of

operational contexts presents many of the most challenging conditions for flying.
2

There are also implementation advantages. Business aviation owners and operators are more likely to,

and quicker to, adopt new technologies than those of other general aviation or commercial transport

aircraft (Kauffmann & Pothanun 2000). The design cycle of business aviation jet aircraft is much shorter

than that of commercial transport jets (Perry 1999). Business jets are redesigned more often than are

other general aviation airplanes because their users demand new technology. These design cycle

considerations allow more frequent opportunities for, and less resistance to, introducing new

technologies. Finally, the market for business aircraft is increasing dramatically. In a poll of U.S.

turbine-powered aircraft operators, use of business aircraft increased 44% in the past 12 months, and 32%

of firms using business aircraft expected employees to make increasing use of these aircraft (NBAA

2000b). Business aircraft manufacturers anticipate this trend to continue (Phillips 2000). This study

focuses on business aviation for reasons of operational variety, and market penetration. The following

sections summarize characteristics of business aviation operations according to features originally

described in Rogers et al. (1998).

Mission Characteristics

Typically, business jet pilots do not fly for compensation directly, but as an employee in the service of, or

as contracted by, a professional organization. Because the goal of business jet operations is to serve

business needs, trips are scheduled with relatively short notice. Most trips are scheduled with about two

weeks notice, but more urgent trips also occasionally occur. Trip destinations are also dictated by the

needs of business. As a consequence, and as allowed by aircraft characteristics, business jet pilots fly into

a wider variety of airports and airfields than do either smaller general aviation or commercial transport

pilots. Frequently, business aviation missions use smaller airfields to minimize costs and distance to

business destinations. These smaller airfields are less likely to provide complete weather information

available at larger airports. Perhaps as a result of mission differences, business aviators tend to discount

preflight weather information relative to weather information and observations available in flight (Lapis

1998). Business jet pilots not only fly to a variety of destinations, as dictated by their passengers, they

also may need to make in-flight deviations to serve the needs and desires of their passengers.

Aircraft Characteristics

The National Business Aviation Association (NBAA) compiles a description of the aircraft fleet used by

business aviation pilots. Table 2 presents the percent of the NBAA fleet for each type of aircraft (NBAA

2000c).

Table 1. Aircraft types in the NBAA fleet.

Aircraft Type Percent of NBAA Fleet

Light Jets (< 29,999lbs) 35%

Piston-powered (reciprocating) 20%

Heavy Jets (> 30,000lbs) 16%

Light Turbo-props (<12,500lbs) 11%

Heavy Turbo-props(>12,499lbs) 10%

Helicopters (most) (< 12,500lbs) 7%

In addition to the basic weather information services available to all aviators, business jet aircraft usually

have on-board weather radar systems, and may have ACARS-like capability to uplink textual weather

information and ground services to provide additional weather information (Lapis 1998). Business

aviation aircraft are equipped to serve the business needs of passengers, and typically include: computer

docking stations, fax machines, air phones, and other office technology.
3

Organizational Characteristics

The organizations that business aviators serve vary widely, and include commercial/industrial,

government, academic, and not-for-profit agencies. Essentially, the worth of business aviation is

established by the opportunity cost of not being able to make a trip when and where it is advantageous for

that organization, and the opportunity cost of personnel work time absorbed by less efficient travel.

Business airplanes that serve these needs can also vary in the infrastructure supporting operations (Rogers

et al. 1998). Small companies, or those that share aircraft, typically fly smaller aircraft and use FixedBase Operators (FBOs) to conduct maintenance. Larger corporations, celebrity pilots, and fractional

ownership businesses tend to use larger aircraft, and typically have full maintenance, flight operations,

and ground support infrastructure. Large, global companies with multiple aircraft type fleets,

international flights, and heavy aircraft most resemble commercial carrier airlines, and have similar

standards for dedicated training, maintenance, and operational control departments.

Pilot Characteristics and Roles

Business pilot experience generally increases with the size of the business' aviation operations and

duration of missions (Rogers et al. 1998). A small survey of business aviation owner/operators, found

that approximately half have college degrees, and most are type rated in more than one aircraft (Rogers et

al. 1998). Training is usually governed by internal operating procedures, unless operating under FAR

Part 135. Business aviators typically are very familiar with the particular airplane they fly. This contrasts

with commercial transport pilots, or rental general aviation pilots who may fly different instances of the

same airplane, or even different airplanes. However turnover rate is high among business jet pilots, as

they graduate to larger aviation operations. While most business aircraft are certified for single pilot

operations, they are typically flown with pilot and co-pilot. Due to high turnover and use of "rental copilots," it is not unusual for crewmembers to be unfamiliar with each other’s skills, predilections, or

experiences; as well as unfamiliar with the business' operating and crew resource management

procedures.

Business aviation pilots must do more than simply pilot the aircraft to the destination. Business aviation

pilots are particularly conscious of comfort of their passengers, their ability to conduct business during a

trip, and their ability to make meeting times. Business flights must be timely, cost-efficient, comfortable,

and enjoyable. Depending on the size of operations, and extent of auxiliary service departments, a

business aviator’s job can include many other tasks, such as: planning point-to-point transportation and

accommodations, overseeing cabin cleaning and catering, maintenance, ensuring appropriate facilities at

destinations for servicing and de-icing, fuel determinations, cost/benefit analysis of fleet augmentation,

assessing and developing training materials and procedures, and performing passenger briefings and inflight "sight-seeing" tours (Rogers et al. 1998).

Weather Flying and Business Aviation

In summary, business aviation operations are a useful platform to consider the effect of weather

information on piloting for many reasons.

• Missions can arise on fairly short notice, minimizing the opportunity to carefully watch

weather trends.

• Mission destinations vary significantly, reducing the familiarity of pilots with local weather

phenomena.

• Mission destinations can be to small airfields that provide little weather advisory information.

• Mission destinations and flight plans can change dynamically due to passengers'

requirements.
4

• Passenger comfort and timeliness are extremely important to business aviation.

• Smaller business aviation aircraft have performance limitations in adverse weather

conditions.

1.3 Cognitive Task Analysis

To affect aviation safety by improving the use of weather information, one must: 1) provide the

appropriate information to support aviation weather decision-making; 2) present this information in an

intuitive manner; and 3) aid pilots, especially inexperienced pilots, in gathering information, interpreting

conditions and selecting appropriate responses. Prior efforts have typically approached this problem by

asking NAS users to comment on existing, and desired weather information products and services. While

user suggestions are extremely valuable, and user acceptance is of paramount import, user preferences are

typically anchored by current conditions and may be inconsistent with performance improvements (e.g.,

Antin 1988). Thus, it is imperative that, in addition to valuable user preferences, we more objectively

ascertain how weather information is used for safe and effective operations in the NAS. In addition to

ongoing traditional task analyses to objectively identify information requirements, this research used a

cognitive task analysis (CTA) to better understand the most challenging decisions associated with, and the

indicators of expertise in, weather flying for business jet pilots.

Cognitive task analysis differs from traditional task analysis by focusing on operators' cognitive

processing and knowledge base, or experience. CTAs are typically used when tasks are complex, or ill

structured; and when these tasks occur in dynamic, uncertain, multi-tasking, real-time operational

domains (Gordon and Gill 1997). To conduct a CTA, one uses knowledge acquisition tools to elicit and

represent general and specific knowledge. Typically researchers use these methods with highly

knowledgeable and experienced operators that are considered subject matter experts (SMEs). The

knowledge elicitation phase of CTA uses a set of interview techniques to explore these experts' decisionmaking processes. The knowledge representation phase of CTA guides documentation and codification

of data into formats that support systems design and training recommendations. Many of the CTA

methods used in this study were adapted from Klein Associates' ACTA (Applied Cognitive Task

Analysis) toolkit. In addition, this study employed the Critical Decision methodology. These tools and

techniques are described below as they are generically used. The methods section of this report

elaborates on how these tools and techniques were adapted for this study.

Applied Cognitive Task Analysis (ACTA) Methods

ACTA (Militello & Hutton 1998; Militello, Hutton, Pliske, Knight & Klein 1997) is a streamlined CTA

method that was developed to be less resource intensive than traditional methods. Traditional CTA

methods are extremely time-intensive, are therefore typically used with only a few participants, and are

best used by experienced interviewers. Although ACTA was developed for less-experienced

practitioners, experienced researchers also use ACTA to obtain a broad understanding of a domain.

ACTA consists of three interview methods to conduct information elicitation about the cognitive demands

and skills required for a task. The generic ACTA procedures are briefly described below. Details of the

ACTA tools are provided in Appendix A.

The three ACTA interview tools are: the Task Diagram, the Knowledge Audit, and the Simulation

Interview. The Task Diagram elicits a model of how a SME parses the way they perform a job. The Task

Diagram results in a set of component tasks for a job, indicates where cognitive skills (interpretation,

judgements, assessments, problem-solving) predominate, and identifies particularly difficult decision

points. The Task Diagram structures the remainder of the CTA. The Knowledge Audit focuses on the

role of expertise in the challenging decisions identified by the Task Diagram. For each of these tasks or
5

decisions, the interviewer asks SMEs to identify how they recognize that they need to make this difficult

decision, the strategies they employ, and to explain why this is a difficult situation. The Simulation

Interview provides a view of SME problem solving in the context of actual operation. Context certainly

guides behavior (Suchman 1987), and as such it is important to provide this context when eliciting SME

problem-solving processes. Simulation scenarios should be challenging situations that require SMEs to

exercise their expertise. These scenarios can be provided as text-based descriptions, video depictions, or

as a scenario in a full-mission operational simulator facility. ACTA developers suggest that high-fidelity

simulations are not required to obtain useful information. During the Simulation Interview SMEs are

asked to interpret a situation, explain the cues and strategies they would employ as well as actions they

would take. Finally SMEs are asked to consider what errors a less experienced and skilled operator might

make in the simulated situation.

CTA also requires a format for representing elicited information in order to facilitate using this

information in design and training. ACTA provides the Cognitive Demand Table as an initial

representation framework. The Cognitive Demand Table describes why each of the cognitive demands is

difficult, as well as the cues and strategies used, and errors that may occur in performing it. Other

representation formats can be developed to support thematic analyses, for example to focus on the role of

expertise.

Critical Decision Method

Klein Associates developed the Critical Decision method based on Flanagan's (1954) critical incident

technique (Hoffman, Crandall & Shadbolt 1998). The Critical Decision Method uses a specific openended question to elicit an incident account from a SME. The nature and content of the opening query is

determined by the research goals of the particular study, but is always asked in terms of an event that the

SME has personally experienced. For example, in a study of Neonatal Intensive Care Unit nurses' clinical

judgments, each nurse was asked to select an incident in which her patient assessment skills had made a

difference to the patient's outcome (Crandall & Getchell-Reiter 1993). In several studies of fire ground

command decision-making, participants were asked to recall an incident in which their expertise as a fire

ground commander was particularly challenged (Klein, Calderwood, & Clinton-Cirocco 1988;

Calderwood, Crandall, & Klein 1987).

Once a SME identifies a relevant incident, he recounts the episode in its entirety, without interruption.

The SME's account, solicited in this non-interfering way, focuses and structures the following interview.

Requesting personal accounts of a specific type of event maximizes response validity, and minimizes

potential interviewer biases. Once the incident report has been completed, the interviewer revisits the

incident with the SME several times, using probes designed to focus attention on particular aspects of the

incident and solicit information about them. Probes elicit details about the recalled event and deepen the

discussion to provide particular emphasis on perceptual cues (e.g., what was actually seen, heard,

considered, remembered) and strategies employed, rather than general value assessments, explanations or

rationalizations about performance. Solicited information depends on the purpose of the study, but might

include presence or absence of salient cues and the nature of those cues, assessment of the situation and

the basis of that assessment, expectations about how the situation might evolve, goals considered, and

options evaluated and those chosen.
6

2.0 Methods

This section characterizes participants in this investigation, provides the CTA protocol adapted from the

ACTA and Critical Decision methods and explains the approaches used to analyze the resulting interview

data.

2.1 Participants

Business jet pilots were recruited by advertisement in business aviation magazines and by direct contact

to local businesses. Applicants were screened using the preliminary questionnaire (Appendix B) to find

participants who had approximately five years of experience as corporate business jet pilots, and who had

a minimum of 500 hours flying business jet aircraft. After this initial selection, preference was given to

pilots who fly a wide range of missions, fly at least four or more times each month, and fly to some

destinations that require three or more hours of flight time.

Eight male, business jet pilots were selected as participants (Table 2). Pilots’ ages range from 33-61

years. All pilots are highly experienced and have flown business jets for more than a decade,

accumulating well over six thousand flying hours in various light, medium, and heavy aircraft.

Participants flew aircraft which included a variety of Cessna Citation and Gulfstream (II, III, IV, V) jets,

DeHavilland DHC-8, Rockwell Sabreliner 65; Raytheon Be-200; Learjet 35; Fairchild F27; Dassault

Falcon, and others. The most predominant type of aircraft flown were light jets (cf. Table 1).

Table 2. Participant Information

Participant # Age Years Business Jet Flying Total Hours Hours as PIC*

1 33 10 6800 2500

2 61 20 14000 10000

3 37 15 6200 4500

4 55 29 11500 6800

5 51 25 14500 8600

6 50 7100* 9700 3100

7 51 22 14500 10000

8 40 14 11000 10500

Median 50.5 20 11250 7700

Mean 47.25 19.28 11025 7000

* This figure was given in hours and is not included in the summary statistic calculations.

The participants typically fly throughout the United States and North America, and only occasionally fly

overseas to Europe, Asia, and the Middle East. Participants reported they always have a co-pilot and on

infrequent, long international flights may carry an extra pilot, flight attendants, and a mechanic. Typical

flight missions are one-day flights to and from a domestic meeting site. These flights may carry

customers to corporate offices or may transport corporate executives to customer sites. In addition to

passengers, business aviation flights occasionally carry small sales demonstration materials, and aircraft

parts and equipment.

Each pilot was interviewed for approximately 3-4 hours and asked to fill out additional questionnaires

following the interviews1

. Participants were assured that the content of the interviews would be

referenced anonymously. All pilots signed a voluntary consent form and were paid $200 plus per diem

for their participation.

1 The data from the post-interview questionnaires supports a different study.
7

2.2 Protocol

In Phase 1 interviews, CTA procedures, materials, and questionnaires were administered to the first three

participants. The remaining five subjects participated in Phase 2 interviews, using the materials and

procedures refined based on Phase 1 interview experience and feedback. Interviews were conducted in a

conference room at the NASA Langley Research Center. Two researchers participated as interviewers for

each SME interviewed. One researcher led the interview. The other researcher primarily recorded

participant responses and monitored the first researchers' adherence to the experimental protocol.

Interviews were audio taped for subsequent transcription and verification of notes.

Phase 1 Interviews

An initial set of semi-structured interviews was conducted with participants 1, 2, and 3, as SMEs. These

interviews obtained additional background and mission information from pilots and used the Task

Diagram, Knowledge Audit, and Critical Decision methods. Appendix C provides the initial interview

guide. Typically, the Task Diagram method begins with asking participants to note the major tasks in the

job of interest. For this investigation, these steps were assumed to be the generally accepted phases of

flight. Participants were provided with a Task Diagram of flight phases (Figure 1), and asked to identify

the most cognitively challenging tasks that weather information affects. Knowledge Audit probes further

explored participants' expertise in handling these challenging tasks with challenging weather conditions.

Participants were then asked to list and characterize the weather information sources they currently use by

completing the Weather Products Survey (Appendix D). Finally participants were asked, according to the

Critical Decision method, to recollect a specific weather-related situation in which weather had a role that

demonstrated their expertise. They were provided the following specific probe:

“You can probably remember some flights in which you had to make difficult decisions due to the

current weather conditions. We would like you to describe an incident in which your experience

made a difference in how you handled the situation—a situation in which a less-experienced pilot

might have made a different decision than you did.”

This incident was then further examined to identify their performance goals, decision points, cues and the

weather information sources they used.
8

Figure 1. Task Diagram: Phases of Flight.

Phase 2 Interviews

The second set of interviews was conducted with participants 4, 5, 6, 7, and 8. Phase 1 interview data

were used to revise the interview guide for the second set of interviews. Appendix E contains the Phase 2

Interview Guide. Rather than asking participants about their background and mission characteristics in

the interview, the preliminary questionnaire was augmented to include these queries. The Task Diagram

and Knowledge Audit methods were employed as previously described. In addition, pilots were provided

with a low-fidelity “simulation" to support the Simulation Interview knowledge elicitation technique.

This simulation was based on actual weather conditions along the Eastern U.S. coast on January 23, 2000.

Table 3 lists the information provided to participants for this simulation. The constructed materials and

captured weather products describing weather conditions are provided in Appendix F. A confederate

researcher, in another conference room, acted as both Flight Service Station (FSS) and Flight Watch (FW)

personnel. He was provided with scripted responses for anticipated questions as well as scenario weather

information to answer unanticipated questions. Participants were told to use whatever information they

had, and to ask for any additional information that they would consult during this mission. Not one

participant requested information from FSS or FW. While this may have been an artifact of the

environment, other research shows pilots frequently under-utilize these services in full mission simulation

experiments as well (Yucknovicz et al. 2000).
9

Table 3. Simulation Interview Materials

Preflight Enroute Approach Charts

Scenario Mission PIREPS - ride reports ATIS recordings Approach Charts

Aircraft Description Visual descriptions PIREPS - terminal area Airport Diagrams

Standard Briefing

(DUATs encoded text)

FW scripts METARS IFR enroute low alt.

Weather Graphics:

Current Surface

24 Hour Surface

48 Hour Surface

Current Flight Rules

Doppler Radar

Lifted Index

Satellite Image

Weather Hazards

IFR high altitude

Sectional charts

Essentially, the simulated mission required the pilot to fly passengers to a meeting during questionable

weather conditions that included icing, limited visibility, and possible high wind conditions. After

participants read the mission scenario and reviewed the other materials, they were queried for decision

points. Experimenters further probed these decision points by asking participants to assess the situation at

the decision point, explain critical cues they would seek, strategies they would use and actions they would

take. Additionally, participants were asked to describe the difficulty of the decision point, and if difficult,

how a novice pilot's behavior may differ from that of an expert.

In addition to the inclusion of a weather simulation, several other changes were made to the interview

procedure based on the results of the first set of interviews. An additional Knowledge Audit question was

developed to augment results from the Critical Decision method. This question asked participants to

conjure a “Scenario from Hell;” that is, a situation that would be extremely challenging, and would

benefit greatly from expertise and experience. To compensate for the addition of the Simulation

Interview and "Scenario from Hell" queries, the Weather Products Summary questions were included in

the Post-Interview Questionnaire (Appendix G).

2.3 Methods of Analysis and Representation

Data analyses were based on notes of all interview sessions. While the procedures for Phase 1 and Phase

2 interviews differed, the authors determined that the effects of these differences were far outweighed by

the relevance of the resulting data. Information from the interviews was extracted as follows. The scribe

researcher prepared a detailed summary of the interview content. The interviewing researcher reviewed

the summary and noted revisions as needed. Audiotapes were consulted and transcribed when researchers

were unclear about interview notes, or when researchers did not recall the interview content consistently.

Interview notes and transcripts were then systematically reviewed to develop knowledge representations

to support analyses.

Several different data representations were developed. The Cognitive Demands table, as defined by the

ACTA toolkit, served as the primary knowledge representation, and an adaptation of this table was used

to represent Simulation Interview results separately. In addition, we developed a model of weather

information sources and reliability assessment, annotated incidents, and also conducted additional

thematic analyses from the raw interview data. The methods for developing these representations are

described below.
10

Cognitive Demand Tables

Recall that Cognitive Demand tables (CDTs) tabulate the most demanding cognitive aspects of a task, and

for each of these "demands," identify why it is difficult, the cues and factors that are assessed, strategies

and actions that would be taken, and what errors novices might make in these situations. We generated a

CDT for each participant. The CDTs were based on all data obtained across the various knowledge

elicitation methods. That is, they integrate data from standardized methods (e.g., the Simulation

Interview) with recollections of personal experiences (the Critical Decision method). Individual CDTs

represent the cognitive demands identified by each SME according to phase of flight. Sometimes a

similar cognitive demand was reported in different phases of flight, but different cues or weather

information were used to make the assessment of the situation, or the same cues may have had a different

meaning in the different phases of flight. In these cases, the same cognitive demand is listed under all

relevant flight phases. Appendix H contains these individual CDTs.

Combining similar cognitive demands across individual CDTs generates two summary CDTs (Appendix

I). The first table identifies indicated information requirements identified by SMEs for these decisions,

and summarizes the sources used to obtain weather information and the primary performance goals for

each decision. The second summary table also lists cognitive demands by phases of flight, but focuses on

strategies the SMEs employ and errors they suggest novices may make. Together, these tables summarize

information requirements for flight planning and in-flight decisions and highlight where current

information sources are inadequate in helping pilots make these decisions. Additionally, they suggest

challenging weather scenario elements and behavioral markers for erroneous/novice performance in these

circumstances.

Simulation Interview Tables

Simulation Interview data was explicitly represented in a separate table, as well as being incorporated in

the individual and summary CDTs. Rather than generalized cognitive demands, the Simulation Interview

tables (Appendix J) presents Phase 2 SME responses to key decision points in a scripted, simulated

weather-flying scenario.

Information Sources & Reliability Assessment

A diagrammatic model of information sources perceived as available to the business jet pilot, and a

framework for how these pilots assessed information reliability was developed by reviewing individual

CDTs and responses to the "Your Use of Weather Products" survey. This representation depicts the

information sources as expressed by the subjects and is not a comprehensive model of all weather

information that pilots may access.

Critical Incidents

Most participants described one or more specific "critical" situations in which their skills as an

experienced pilot were challenged by weather-related factors. These incidents were intentionally elicited

using the "Scenario from Hell" inquiry and the Critical Decision method probe. Each incident was

analyzed to identify the critical decisions contained in the incident and the relevant cues and factors. An

example of an annotated incident is shown in Table 4. The complete set of annotated Critical Decision

probed incidents is included in Appendix K. Appendix L describes the "Scenarios from Hell" responses

from Phase 2 interviews. These analyzed incidents provide actual scenarios that may be recreated in fullmission simulation evaluations of aviation information weather systems.
11

Table 4. An Example of an Annotated Incident

Narrative of Incident Analysis

SME 4 had to miss an approach into a small, unsupported

airfield near Traverse City, MI. He could not see the airfield

even though the ASOS (Automated Surface Observation

System) was telling him that the airfield was above

minimums. SME 4 knew that the temperature and dew-point

spread was close, and that the wind was in the “wrong

direction.” But the automated weather observation equipment

was telling him that the airfield was within the limits. He said

he “smelled a rat.” He made the approach, but the visibility

was only 2 1⁄2 miles and the ceiling was under 700 feet. He

could not see the runway, so he made a missed approach. He

did not really know how the automated observation equipment

worked, but he sensed that it might be wrong based on his

assessment of the conditions, and his intuition proved him

correct. He ended up flying to a close alternate airfield.

Cognitive demand:

Can I accept low ceiling and limited

visibility at destination?

Information sources:

ASOS at destination; pilot’s observation of

conditions at destination; pilots’ experience with

similar weather situations

Cues:

temperature/dewpoint spread; wind direction

Actions:

attempt approach; observe ceiling and

visibility

Cue:

couldn’t see runway

Action:

abort approach; fly to alternate destination
12

3.0 Results

These results are presented as general observations of expert business jet pilot responses to weather

flying, and more specifically the cognitive demands and indicators of expertise of these pilots in difficult

weather flying scenarios. Results were compiled to provide a representation of how these subjects

perceive the weather information sources available to them, as well as how they assess weather

information reliability. The results of this study are further interpreted for pragmatic purposes. They

identify scenario characteristics and behavioral markers to support full mission simulation studies to

evaluate AWIN systems. They also indicate both design and training interventions to improve business

jet pilot decision-making for weather flying.

3.1 General Observations

Based on the entire corpus of data, we observed these general findings:

• Twenty-two weather-related critical decisions were identified for the business jet pilot

community. Weather conditions impact decisions at all phases of flight and levels of

decision-making, from general planning to tactical runway selection in changing winds.

• Three high-level pilot goals were identified: Flight Safety, Flight Efficiency, and Passenger

Satisfaction. The pilots we interviewed emphasized that flight safety is never compromised

to improve efficiency or passenger satisfaction. However, these pilots note that these goals

are all influenced by weather and can potentially conflict, potentially requiring in-flight replanning and communication with passengers.

• Individual differences were observed across the different pilots we interviewed. They varied

in the amount of effort and time they expended to understand the weather picture in the

Simulation Interview, and in their weather information acquisition and en route decisionmaking. There were also differences in the degree to which they relied on reported weather

information (such as METARs, TAFs, PIREPs, ATC, ATIS, etc.) versus their own senses.

• Business jet pilots seek out and rely on the most timely weather information. The older the

data, information, or forecast, or the further away from the pilot's position, the less reliable it

is perceived to be. Business jet pilots tend to rely on the reports of experienced humans over

automated or processed data and information. They have the most confidence in what they

can observe directly.

• Weather-related decisions made by business jet pilots differ from those made by general

aviation and transport pilots due to differences in mission and aircraft characteristics.

Business jet pilots have more flexibility in achieving their mission than do transport pilots.

They can accommodate weather situations by leaving earlier/later, flying into a different

airfield, detouring around a wider area, etc. However, they have impoverished weather

information to the extent that fewer have dedicated weather ground services, and they may

have to fly into unsupported airfields. Business jet pilots tend to rely more on their piloting

skills, and on the higher performance capabilities of their aircraft and possibly auxiliary

equipment (e.g., onboard weather radar) to tactically maneuver around weather they

encounter. The pilots we interviewed suggested that they would be less likely to do this, and

be more reliant on preflight and strategic weather information, when flying smaller general

aviation aircraft.
13

• In the Simulation Interview, most SMEs exhibited frustration with the standard FAA/ICAO

encoded DUATS text preflight information. Many admitted they did not typically use this

type of preflight information. Many of our participants were accustomed to having tailored

weather briefs from private weather providers, and were not facile with standard DUATS text

reports. Reports tailored to the departure point, route of flight, and destination supported by

graphical representations were preferred.

• Experienced business jet pilots are able to accurately judge the severity and dynamic nature

of the weather most of the time. They have a good understanding of the performance

characteristics of their aircraft and are typically able to “fly around” significant weather

situations. They have well-developed self-monitoring skills that enable them to make

effective judgments about their ability to fly in difficult weather conditions.

3.2 Cognitive Demands

The identified cognitive demands and decisions faced by business jet pilots in making weather-related

decisions and judgments are described below according to these flight phases: preflight planning, taxi &

take-off, climb, cruise, descent & approach, land & taxi.

Preflight Planning

• What’s the general weather picture? Will it be affecting my flying in the next few days?

This judgment is a very broad, early, assessment of expected weather problems for operations in

the subsequent day or days, and the kinds of weather phenomena that the pilot will need to pay

attention to for detailed planning. This judgment relies heavily on knowledge of atmospheric

dynamics and regional trends.

• What are the weather problems for this flight?

- Departure: Is it a Go/No Go ? VFR/IFR? Will the weather affect my departure time?

Pilots consider whether the weather is severe enough to cancel the mission. Concerns about

weather hazards for a mission lead pilots to consider alternate flight plans. For business

aviation operations, this requires a conference with passengers to determine viable options for

alternative departure times and/or destinations. In special circumstances, this requires

planning for different departure points. For example, one study participant flew under Visual

Flight Rules (VFR) conditions from the location of his business jet’s hanger to a small private

airfield in order to pick up the passengers. If he anticipated weather conditions would not

support the VFR portion of his mission, he would have to arrange to meet his passengers at

the public airfield. Weather also affects the flight rules that a mission is operated under.

Considering the weather in context of departure concerns leads pilots to consider general

problems that can occur in taxiing, takeoff, and climb. These considerations include, for

example, the effect of possible icing on climb performance.

- En route: Which route will I take? Will I need to detour around weather?

This includes decisions relating to the planned route of flight, potential detours and added

time required to reach the destination, any specific safety concerns, etc.

- Destination: Is it still viable? Do I need to consider alternative destinations? If so, where?
14

This includes planning decisions related to the destination itself. These include: Is it even

feasible to land at the preferred airfield? Should I consider an alternative now and/or plan to

deviate en route? If so, which will be the most feasible and convenient for my passengers?

• How does the weather affect the fuel I carry?

Pilots assess whether extra fuel will be needed for possible deviations from route, delays, holds,

or diversions to an alternate airport due to weather. The desire to carry additional fuel for mission

flexibility must be considered in light of the costs of doing so. These costs include: expense of

extra fuel, and reduced aircraft climb-rate and payload capability.

• How much confidence do I have in the co-pilot's weather flying abilities?

This assessment affects how comfortable a pilot might be in making a mission, the workload

distribution he expects during the mission, and how often he might engage in monitoring and

cross-checking his co-pilot. This assessment is based on familiarity with the co-pilot, knowledge

of their general flying experience as well as region-specific and weather-exposure experience.

Taxi & Take-off

• Can I taxi/take off in this weather?

The weather concerns relate directly to the ability of the pilot to safely taxi and takeoff, but also

affect passenger comfort and schedule. Pilots consider weather factors such as ceiling, visibility,

crosswind, windshear, runway conditions, and icing probability, in these assessments.

• Will I need to consider a rejected takeoff?

Pilots consider the likelihood of having to abort a takeoff and the contingency actions if a takeoff

is aborted. Cross wind speed and direction, thunderstorms, runway conditions, and other weather

phenomena can influence these plans.

• Do I need to delay my takeoff?

Sometimes weather conditions can change significantly between preflight planning and actual

departure. Pilots must reconsider the safety of departing at the planned time, and may consider

delaying departure. As previously mentioned, such a decision requires a conference with

passengers to preserve, if possible, success of the ultimate business mission. Delays can also be

required to ensure that the aircraft is properly configured for weather conditions. Deicing, and

ground use of the onboard weather radar are specific examples of such delays.

• Do I need to plan post-takeoff actions?

Pilots think beyond takeoff to plan for actions based on weather factors that may impact the

aircraft immediately during the takeoff. For example, they may plan a specific maneuver to avoid

a weather cell in the vicinity. They may consider wind conditions when planning for engine

failure contingencies.

Climb

• Do I anticipate icing problems?

Pilots anticipate the likelihood of icing on the climb-out and judge the impact of icing on flight

safety and aircraft performance. The pilot must continually be aware of the aircraft’s

performance in potential icing conditions.

• Do I anticipate wind-related problems (turbulence, windshear, crosswinds)?
15

Pilots are also aware of wind conditions that impact aircraft performance and ability to maintain

lift. Pilots also consider wind conditions' effects on passenger comfort.

• Do I anticipate any weather obstacles on climb out?

Hazardous weather phenomena, such as convective cells, rain shafts, wind shear activity, and

reported lightning create obstacles that the pilot tries to avoid, for safety and passenger comfort

reasons. Deviations around these obstacles should be considered in advance and with respect to

other constraints such as terrain and airspace restrictions.

Cruise

• Is my destination still viable?

These decisions relate directly to judgments and assessments made during the cruise phase of

flight. If the weather at the destination is uncertain or less than anticipated, pilots will more

frequently sample weather data for the destination during cruise. This information is used to

continuously assess the viability of the destination, and alternate destinations. Viable alternates

must be appropriate not only in terms of weather conditions, but also for the aircraft's fuel

available, and performance capabilities. Further, alternates should be acceptable for the business

mission (i.e., passengers will still be able to accomplish the business purpose of the mission).

• Is the flight proceeding according to expectations?

This is a general assessment of actual progress against planned progress; of unexpected

occurrences or weather, of planned ground speed versus actual ground speed (unexpected

headwinds aloft), of estimated-time-of-arrival (ETA) versus the passengers’ schedules. The

pilots we interviewed mentioned using PIREPs as well as checking with FSS and ATIS just after

the hour, when new information may be available. Deviations from expected occurrences and

performance spur pilots to acquire additional information about environmental conditions and

engage in replanning. Pilots may also decide they need to alert their passengers if the flight is not

proceeding as planned (e.g., ETA changes).

• If I see an unexpected weather hazard, how can I avoid it?

Perfect weather information would enable perfect strategic flight planning to avoid weather

hazards. However, because weather dynamics cannot be perfectly modeled and therefore

forecasted, and because weather information may not be immediate or precise enough, some

pilots "wait and see" what the weather looks like before attempting to avoid it. Tactical

avoidance strategies include flying around or over the weather, and most conservatively, simply

turning around. Tactical avoidance of weather hazards requires pilots to consider weather

dynamics and intensity, aircraft performance characteristics, and other airspace constraints (e.g.,

terrain, airspace). In addition to simply tactically avoiding weather they could see, several

interviews suggested that pilots query ATC to better understand how pilots closer to the

anticipated hazard are choosing to avoid it.

• How can I provide passengers with the most comfortable ride?

En route decisions also must consider the impact on passengers: Are they comfortable? Can they

work or sleep comfortably? Turbulence from weather cells, and the jet stream is the primary

concern with regard to passenger comfort. If turbulence is unavoidable, pilots attempt to notify

passengers prior to onset. Advance notice of such turbulence is therefore valuable information.
16

Descent & Approach

• Do I anticipate significant weather on descent?

When in the vicinity of the destination airfield, pilots obtain local weather information (e.g.,

ceiling, visibility) and reconsider the safety of landing at this airfield and on the designated

runway. Pilots consider suboptimal conditions (e.g., icing, windshear, turbulence, crosswinds,

runway conditions) that require additional contingency planning for missed-approach planning.

• How do I deal with weather hazards on descent?

This judgment is difficult because the options narrow for the pilot as the airfield gets closer.

Different obstacles require different avoidance strategies. Also, the closer to the ground the

aircraft is, the more severe the consequences of loss of lift. Expert pilots develop these strategies

in advance and consider airspace and terrain, as well as aircraft performance characteristics.

• Do I want to hold or divert to an alternate destination?

This decision may be instigated by weather conditions, but must also be evaluated in light of

weather conditions. While one might decide to hold to see if an airfield's visibility improves, it is

unwise to hold in what may be icing conditions. The decision to hold or divert also affects

passengers’ schedule and ground transportation logistics; and requires additional fuel and

therefore expense. If the pilot chooses to not carry extra fuel, for improved payload capacity or

climb performance, s/he may eliminate the option of holding at the destination. Provided safety

is not compromised, decisions regarding changes to the final destination are often shared with

passengers.

• Do I anticipate a missed approach and go-around?

Pilots consider the likelihood of a missed approach or go-around due to weather conditions, and

the effect of weather conditions on the ability to perform these maneuvers.

• What kind of approach do I need to make (Visual, ILS)? Which runway? Is it suitable?

This more specific judgment addresses selection of approach and runway. Even if local weather

information defines the airfield conditions to be safe, pilots still must consider the

appropriateness of the approach and runway for the weather conditions. For example, icecontaminated runways and runway length must be considered in light of aircraft landing

requirements.

Landing & Taxiing

• What kind of landing do I need to make?

This decision is made based primarily on safety factors, although it influences passenger comfort.

If the runway is very wet, it may be necessary to bring the aircraft down hard to avoid skidding

on the wet surface. To inform such decisions, the pilot needs to have an accurate appraisal of the

runway surface conditions and surface winds.

• Do I need to prepare the aircraft for a quick turn-around for my next leg/trip?

This cognitive requirement is faced when the aircraft has to be turned around quickly for the next

leg of a trip. Pilots must think about how the weather will change while they are on the ground

and how they may have to accommodate that into their plans.
17

3.3 Indicators of Expertise

Consideration of expert/novice performance differences is useful to design supportive aiding technologies

and training programs. In addition, these differences and the errors that novices may make define

behavioral markers for assessing performance. In this study, we only interviewed experienced business

jet pilots, so our conclusions about these differences are derived from these individuals' perceptions of

their own skill development and contrasts of their behavior with less experienced colleagues. Study

participants were specifically asked to consider the types of errors less-experienced pilots might commit.

The benefits of piloting expertise and deficiencies of novices are as follows:

• Expert business jet pilots accurately judge the severity and dynamic nature of the weather most of

the time. Novice business jet pilots are more likely to misinterpret weather severity or fail to

anticipate changing weather patterns. Novices, therefore, may not be able to adjust properly to

changing conditions, either disrupting safety, efficiency, and/or passenger comfort.

• Expert business jet pilots build a dynamic, "big picture" of a situation by understanding how a

situation developed, assessing relevant current and forecasted data, and projecting the situation.

Novice business jet pilots build incorrect or incomplete pictures by either not thinking far enough

into the future (e.g., not contingency-planning), failing to continuously monitor changing weather

conditions, and failing to use all relevant resources to check the reliability among them.

• Expert business jet pilots have a good understanding of the performance of their aircraft and

onboard equipment, the limitations imposed by weather conditions, and strategies for recognizing

and compensating for these limitations. Novice business jet pilots may not consider how changing

weather affects aircraft performance and therefore are forced into a reactive mode of control.

Novices may not have a refined knowledge of how to use instruments to their fullest potential to

best acquire weather information and recognize effects of weather on aircraft performance.

• Expert business jet pilots have strong self-monitoring skills that enable them to judge when they

should avoid weather rather than manage through it. They more accurately judge their own

abilities to fly in certain conditions, and more readily catch their own judgment errors before they

are enacted. Novice pilots who have not experienced difficult weather flying, have not developed

diagnostic skills for identifying hazards or decision-making skills for selecting appropriate

responses, can have a cavalier attitude and venture into hazardous situations. This deficiency in

skills, in conjunction with the delayed realization that one is in a novel and hazardous situation,

can quickly overwhelm and confuse a novice pilot.

• Expert business jet pilots are more aware and attentive to all details of flight, from passenger

schedules to atmospheric pressure changes to providing passengers services such as catering.

Novices tend to be more focused on piloting and making the next point in the flight plan.

3.4 Weather Information Sources Diagram

The Weather Information Sources diagram (Figure 2) represents all the sources of weather information

and types of weather information that the SMEs referred to in interviews. The diagram does not represent

all possible sources of weather information that are available to the pilot community. The sources

depicted are those that are most commonly reported by the business jet pilots we interviewed to help them

understand the weather picture and its implications for flying. These sources, the ovals, are described in

terms of the weather information, the rectangles, for which they are referenced.
18

18

Figure 2. Sources of Business Aviation Weather Information.

Bizjet Pilot Weather

Information Sources

Senses

ATC

Automated

Services

e.g., ATIS,

ASOS/AWOS

FSS/Flight Watch

(en route)

Private Wx

Provider

Internet

(Wx.com)

FSS/DUATS

Television

Wx Channels

Datalink fromground sensors

Flight

Management

System

Detailed Wx departure poalternates;suAIRMETs / Sreports, wind

General conditioradar & outlooks

Current terminal Wx e.g. wind,IFR, MVFR, ceilings, temp,dew-point, precipitation, runwayinformation, Altimeter settings,

obstructions to visibility.

Eyes

En Route

Departure

Destination

Vibrations,

“feel”

Ears

Wx at destination(warningsand hazardous, severe Wxinformation), SIGMETs , icingreports etc.

Cp(t

Pilot reported problems(turbulence & icing),precipitation.

Tailored Wx WSI, Universynopsis, dop

severe Wx, t

outlook forec

General Wx patterns:localconditions, current dopplerradar & satellite picture,outlooks/forecasts

Doppler radar returns of

precipitation intensity

Temp, wind, fuelconsumption, ETA

Clouds shapes, edges, types,tops, & color; lightning, rain,wind indicators, icing

Turbulence, aircraft performance

Thunder, rain, hail, wind

Precipitation en route infront of aircraft (intensity,severity/ gradients),storm cells (electricalactivity)

Onboard

Sensors

e.g. weather radar,

lightning detector
19

3.5 Assessing Weather Information Reliability

The Weather Information Sources diagram illustrates that pilots use a variety of weather information

sources to arrive at their "weather picture." SME responses also were analyzed to determine the

characteristics that these pilots use to assess the reliability of weather information, and thereby prioritize

weather source usage. These interrelated factors include: directness of perception; timeliness and usefullife of data; spatial dislocation of the pilot/sensor from the weather, sensor and transmission equipment

quality, and spatial resolution, and meta-cognitive interpretive skills.

Pilots in this study indicated that they are more likely to trust their own direct perceptions than those of

others, or of automatically-sensed information, and more likely to trust the perceptions of other humans in

the NAS than that of automated weather information services or sensor-driven displays. Of course not all

weather information of significance is available to direct visual perception. To the degree that presented

data is isomorphic with the atmospheric conditions, it is more "direct." For example, a satellite picture of

cloud cover is considered more direct than a mosaic radar return of the same area.

The timeliness and useful-life of the data are interrelated. Subjects assessed the credibility and reliability

of current weather reports based on how long ago the reading was taken, and on the dynamism of the

observed weather phenomenon. For example, a report of a low-lying fog over an airfield may be valid for

several hours after the report was made. Conversely, a turbulence report at flight level 310, may rapidly

become obsolete or may be valid for several hours, depending on the weather phenomenon's dynamics.

Generally speaking, pilots trust weather reports that are more recent than those that are less recent, and

use their knowledge of weather dynamics to assess when reports are likely to be obsolete. Weather

forecasts are similarly assessed. Pilots are unlikely to rely on forecasts to the degree that the weather they

predict is dynamic. Some subjects in this study mentioned that they assess the validity of forecast

information by comparing past forecasts and current conditions.

SMEs in this study also mentioned spatial dislocation of the sensor (either human or automated) from the

phenomenon as a significant factor affecting their assessment of weather information validity. In essence,

this factor can be described as either due to the effects of the prior factor, "timeliness and shelf-life" vis a

vis atmospheric dynamics, or to inaccuracies that increase in sensor and transmission capabilities with

increased distance. This factor is described next.

SMEs are aware that the weather information available to them is only as good as the equipment used to

detect, transmit, and present this information. Directly perceived weather information is subject to, for

example, the limitations of visual perception. Similarly, for automated sensing systems, the sensor

quality (sensitivity, false alarm rate) and the transmission quality (timeliness and completeness of

uplinks) must be considered when assessing the validity of the resulting weather information. Both

human vision and automated sensors are likely to be degraded with distance. Many of the pilots

interviewed reported that their onboard radar provided varying degrees of reliable and accurate data.

These data are dependent on the range of the phenomenon from the aircraft, the power of the radar, the

radar range setting, the presence of atmospheric or ground clutter, and the intensity of the weather

phenomenon itself (which may block the extent of the radar's effective range).

The spatial resolution of the weather information is relevant to SMEs reliance on this information. These

pilots rely on weather information to the degree that it is spatially precise and appropriate for their

flightplan. For example, the general outlook for a 24-hour forecast may predict “half-mile visibility in the

vicinity of Greenville Airport” but a pilot may still fly out because the hazard is localized to the South

side of the airport. This forecast does not provide the relevant level of detail to fully trust this information

to make such a decision.
20

Finally, meta-cognitive skills are important in determining how pilots use weather information sources.

The subjects in this study demonstrated a range of comfort in understanding atmospheric phenomena,

weather hazards and dynamics, and technical details of weather information sensors and services. These

pilots determined appropriate reliance on abstracted and sensed weather information sources to the degree

that they are confident in their weather assessment skills. Their self-confidence in interpreting weather

data, detecting false indications in weather products and predicting weather behavior allowed them to

more intelligently assess the reliability of individual reports and forecasts. These assessments were often

derived from regional experience and history of exposure to the interaction of atmospheric conditions and

terrain. Some of our participants made their own assessments of raw information, and generated their

own forecasts. They were able to generate a reasonable explanation of the weather in certain areas for

certain periods. If the existing weather data, information, and forecasts matched their own predictions,

they could place more faith in it. They “owned” their weather data, rather than relying on someone

telling them what was happening and what was going to happen. Less meteorologically skilled

participants were more likely to either blindly trust preflight weather information or tended to completely

dismiss this information and be more reliant on, and reactive to, in situ direct and PIREP observations.

3.6 Scenarios and Measures for AWIN Simulation Experiments

AWIN researchers will be evaluating aviation weather information systems in flight experiments and fullmission simulation evaluations. One critical facet of such experiments will be defining, and in the case of

simulation evaluations designing, scenarios that stress pilots' weather-flying skills and decision-making.

It is equally important to evaluate pilot performance using sensitive, operationally significant measures.

Data obtained from the Critical Decision and "Scenario from Hell" probes provide a foundation for

defining scenario elements that stress weather-flying skills and decision-making and indications of good

performance in these conditions. Cognitive Demand tables provide behavioral markers of novice pilot

errors.

Appendix M parses Critical Decision and "Scenario from Hell" incidents into scenario elements. These

elements are further classified by phase of flight, type of element (weather condition, system malfunction,

environment, and infrastructure), the sources of information to cue the pilot, and behavioral markers for

good performance. Some weather situations that may be used as scenario elements include:

• an uncontrolled airport as a destination with weather near minimums and non-precision

approaches,

• unreliable or uncertified AWOS,

• highly dynamic weather conditions at the destination,

• difficult crosswinds on the assigned runway,

• weather cell in vicinity of airport and further constraints from terrain,

• ground sensors or personnel provide optimistic weather intensity and closure rate

assessments,

• runway ice contamination in conjunction with anti-skid failure,

• engine failure at take-off with crosswinds,

• bleed air failure and icing conditions,

• icing during a night flight with pitot-static occlusion,

• onboard radar clutter and attenuation,

• dense fog occurs just after reaching rotate speed.

Subjects mentioned other factors that are relevant to business jet operations that exacerbate bad weather

condition situations. These include:

• obvious alternates that don't provide sufficient ground transportation for passengers,
21

• late and anxious passengers,

• uncooperative co-pilot,

• ground switches runways at last minute,

The expert subjects in this study were specifically asked to consider what errors novices would make in

challenging weather flying situations. These data are presented in their entirety in the Critical Decision

Tables. A summary of these novice aviator errors follows:

• inappropriate fuel loading: failure to consider need to reach alternates & performance

implications

• over-reliance on forecasted weather in dynamic conditions

• failure to intermittently monitor changing weather conditions at appropriate sampling rate,

• failure to acquire information from multiple weather sources to cross check for validity,

• over-reliance on automatically sensed weather information,

• acquiescing to passenger pressure to make trip when concerned about weather,

• failure/inability to develop trend information; to predict if conditions are worsening or

improving,

• failure/inability to consider, temporally project weather intensity and location,

• failure to appreciate limitations of, and/or appropriately manipulate onboard radar sensors,

• insufficient understanding of aircraft performance and implications for weather avoidance,

• failure to appreciate effects of weather, and weather equipment use, on aircraft performance,

• failure to accurately predict effects of weather on passenger safety and comfort,

• failure to continue to monitor fuel in weather avoidance,

• staying on a visual flight plan (VFR) in instrument meteorological conditions (IMC),

• over-confidence in abilities,

• failure to assess runway crosswinds until too close in.

3.7 Design & Training Interventions

The cognitive task analysis of business jet pilot weather-related decision-making generated several

recommendations for this niche in the flying community. Some of the recommendations are very specific

to the business jet community, whereas others are probably applicable across other groups within the

flying community. We have identified two areas for our recommendations: business jet flight deck

design and business jet pilot training.

Design Interventions

The following recommendations address system design issues for the business jet pilot community. They

address only issues specific to decisions and judgments related to weather.

Integrated Source Displays. Currently, pilots must integrate multiple types of information from

multiple sources. These include up-linked radar pictures, up-linked satellite, textual ATIS

reports, voice reports of turbulence over common broadcast radio frequencies, automated

observations from specific locations, weather alerts issued by various agencies, etc. Flight deck

displays should provide an integrated representation of weather information to support the

cognitive demands of situation assessment and response planning.

Big Picture Radar Representations. Due to the flexibility required of business jet missions, it

is particularly important that these pilots understand the big picture of the weather situation.
22

Augmenting onboard aircraft radar capabilities with up-linked "big picture" radar information

would aid challenging en route re-routing decisions.

Radar Image and Usage. Pilots need weather hazard displays, rather than radar return displays.

Subject matter experts should be used to develop rules for interpreting onboard weather radar data

and rules for understanding when that data might be suspect. In addition, subject matter experts

could serve as the basis for codified guidance on how to more expertly and sensitively manipulate

the onboard weather radar system to gather the most appropriate weather information, as well as

be able to assess the reliability of that information. Onboard weather radar systems, as well as on

board aviation weather information systems more generally, would benefit pilots by relieving

workload if they performed automated monitoring for weather hazards and changes, and provided

indications of these events in a manner consistent with their import.

Significant Weather Alerts. Participants frequently mentioned problems associated with

unreliable weather information in terms of timeliness, and relevance in terms of proximity to their

own position. Pilots do not always know when to seek information, or when the weather

situation changes, for better or worse. Remembering to obtain weather information and actively

doing so are significant workload impositions on a pilot. Onboard aviation weather information

systems as well as ground-based weather information systems, would benefit pilots by relieving

workload if they performed automated monitoring for relevant weather hazards and changes, and

provided indications of these events in a manner consistent with their import. Currently, ATC

broadcasts weather alerts to pilots on IFR flight plans or who are using VFR flight following.

However, these alerts are weather-centered, i.e., they describe the location and severity of

weather, not pilot-centered, i.e., considering a specific route, aircraft performance capabilities,

etc. These weather-centered reports provide necessary information, but in a manner that has the

following disadvantages: they may be irrelevant to a particular route and/or aircraft, they may be

delayed, and they rely on radio frequency availability - which becomes saturated in difficult

weather conditions.

Decision-Centered, Action-Oriented Alerts. The goal of in-flight aiding systems should not

necessarily be to tell pilots what to do, but to support their own situation assessment and decisionmaking process. One aspect of this support is to provide interpretations and implications of

different weather situations on the pilot’s intended course of action. For example, by considering

the implications of weather alerts to the possible alternative runways in an approach and landing

decision, one might receive a warning such as:

“Crosswinds at destination X conflict with chosen runway Y. .

Wind speed and direction are above aircraft landing minimums...

Selected runway Y is now marginal or non-viable.

Suggest runway X, or suggest alternate airfield.”

This warning provides the current wind conditions at the destination airfield in relation to the

intended landing runway, indicates the current status of the runway for landing, and suggests

alternate courses of action. Ultimately, pilots would be responsible for acting on this information,

or they may choose to wait, and reassess the intended runway conditions when closer to the

airport.

Winds Aloft Information System. Pilots reported that winds aloft information is particularly

unreliable. This information is critical to business jet pilots who are trying to maximize

passenger satisfaction and flight efficiency. Pilots suggested that the winds aloft forecasts could

be improved by using data from aircraft in the air that are sensing wind speed and direction in
23

real-time. These data could be transmitted to a central location, aggregated and integrated into

current wind readings. This would improve current observations of winds aloft that are currently

collected by the 12-hour weather balloon recordings.

Weather-Integrated Route Planning Decision Support. One of the challenges faced by the

business jet community is to understand the impact of weather on their route of choice and to

select alternate routes based on this information. Because the business jet community has flexible

routing and scheduling, route planning is more challenging than it is in the transport aviation

world. One solution would be to integrate route selection planning with current weather

information to provide the most feasible routes given current weather hazards. Pilots reported

that the flight planning systems they typically use calculate routes to their destination without

taking into account the current weather situation. Only after the routes are planned are weather

concerns superimposed on this route and the route evaluated for safety. An improved flight

planning system would incorporate routing around such weather hazards and consider the

dynamics of weather system movement and intensity, and consider these hazards in conjunction

with terrain and airport capacity constraints.

Tailored Weather Planning Information. Many private weather information providers provide

the aviation community with weather information tailored to specific routes and aircraft

capabilities. These tailored weather briefings are preferred over standard DUATS text preflight

information. Pilots expressed preference for graphical weather information. Reference locations

used in AIRMETS and SIGMETS are often difficult to interpret in terms of the pilot's route, a

problem exacerbated when a pilot is unfamiliar with the region.

Anomaly Detection Support. Pilots would benefit from decision support that would flag

deviations from expected flight progress or weather status. Rather than a wide area broadcast,

this information would be specific to a mission. It would relate to the current flight parameters

(e.g., route and ETA), and would reflect any changes above a “threshold” (to be empirically

determined or user-set), to alert pilots that their situation awareness is based on old information,

and therefore would more sensitively alert them to slowly developing problems that may

otherwise go undetected. For example, an alert would draw the attention of the pilot to variations

from the ETA based on projected route, speed, altitude, and wind conditions. If any of these

parameters altered the ETA significantly the pilot could be alerted and the winds aloft projections

could be recalculated in order to assess their fuel situation. Aiding could monitor and assess the

impact of de-icing equipment on aircraft performance and alert the pilot to any significant

impacts on fuel consumption.

Training Interventions

The following recommendations address training issues for the business jet pilot community. They

address only issues specific to decision-making and judgment related to weather.

Radar Usage. Onboard weather radar provides very helpful, real-time information, however,

pilots are not adequately or routinely trained to manage this technology or interpret radar return

images. Previously, we recommended aiding technologies to support these management and

interpretation tasks. This recommendation does not obviate the necessity of training pilots to

manually manage radar, interpret radar images, and, in particular, understand the limitations of

this technology.
24

Training Scenarios. During training, weather problems are often provided as a single obstacle

that introduces sudden flight problems. Training scenarios should also address more complex,

often seemingly benign, weather situations that may be encountered by pilots. These scenarios

should include conflicting passenger comfort and flight efficiency goals based on the

interpretation and assessment of uncertain weather conditions. Scenarios should also present

situations that have combinations of weather-related problems, and combinations of weather and

equipment failure scenario elements. Training scenarios should be designed to sharpen the pilot’s

ability to gather weather information, interpret the weather situation, identify the time course of

weather dynamics and response requirements, and select and evaluate options.

Goal Deconfliction. Training for the business jet pilot community needs to focus on the

efficiency and comfort goals as well as the safety goals. The pilots we interviewed did not

describe situations in which they had compromised the safety of their passengers. Many of the

difficult judgments pilots reported were related to conflicts between organizational and

passenger-related goals. For example a participant reported “Ceiling and visibility restrictions

may not allow me to get my clients to their preferred destination—should I go ahead and attempt

the trip, even though I may have to turn around and bring them back to the departure point?”

Pilots could be explicitly trained to identify goal conflicts, and effectively use weather

information to evaluate options. Scenarios designed to pit these goals against each other or

introduce factors that may generate several conflicts within a goal would help the pilots make

better use of their weather products, recognize situations where these conflicts are likely to arise,

and develop strategies for identifying the best course of action.
25

4.0 Conclusions

This study conducted a cognitive task analysis with eight expert business jet aviators to better understand

challenging weather-related decisions. Several CTA methods, developed by Klein Associates were used.

In addition, we extended the Simulation Interview method by providing participants with a description,

and weather information materials from an actual challenging weather day.

Several representations were developed to codify interview data and facilitate analyses. Results distilled

to 22 different cognitive demand categories faced by experienced business jet pilots related to weather

conditions. The expert participants suggested how novice pilots might have difficulty with these

challenging conditions. We developed a representation for the information sources the participants in this

study mentioned while describing challenging weather-flying situations, and developed a framework to

describe the factors that these pilots consider when evaluating trust in weather information sources.

Results were considered in light of supporting empirical simulation evaluations, through scenario and

measurement development, and in light of identifying design and training interventions.

The objective of this report is to document the CTA methodology developed for this study and to present

these preliminary findings. While many CTAs are conducted with as few as eight SMEs, the complexity

of the flight mission and variety of themes addressed in this study necessitated a broader treatment in the

interview. For this reason, additional CTAs should be conducted with the documented protocol to fortify,

or perhaps extend, these preliminary findings.
26

5.0 References

Antin, J.F., 1988. An empirical comparison of menu selection, command entry, and combined modes of

computer control. Behavior and Information Technology 7(2): 173-182.

AOPA Air Safety Foundation. 1996. Safety Review: General Aviation Weather Accidents - An Analysis

and Preventive Strategies. Frederick, MD: AOPA Air Safety Foundation.

Calderwood, R., Crandall, B. W., and Klein, G. A., 1987. Expert and Novice Fireground Command

Decisions. Final Report for the U.S. Army Research Institute, contract MDA903-85-C-0327. Fairborn,

OH.: Klein Associates, Inc.

Crandall, B., and Getchell-Reiter, K., 1993. Critical decision method: A technique for eliciting concrete

assessment indicators from the intuition of NICU nurses. Advances in Nursing Sciences 16(1): 42-51.

Flanagan, J. C., 1954. The critical incident technique. Psychological Bulletin 51: 327-358.

Gordon, S.E., and Gill, R.T., 1997. Cognitive task analysis. In C. Zsambok and G. Klein (Eds.)

Naturalistic Decision Making. Mahwah, NJ.: Lawrence Erlbaum Associates.

Hoffman, R. R., Crandall, B. W., and Shadbolt, N. R., 1998. Use of the Critical Decision Method to elicit

expert knowledge: A case study in cognitive task analysis methodology. Human Factors 40(2): 254-276.

Kauffmann, P. and Pothanun, K., 2000. Estimating the Rate of Technology Adoption for Cockpit Weather

Information Systems. Technical Report SAE 2000-01-1662. Warrendale, PA: Society of Automotive

Engineers.

Klein, G. A., Calderwood, R., and Clinton-Cirocco, A., 1988. Rapid Decision Making on the Fireground

United States Army Research Institute Technical Report DTIC-AD-A199 492. Alexandria, VA: U.S.

Army Research Institute.

Lapis, M.B., 1998. AWARE Report 1. Interim Report for NASA Langley AWIN Program. Palo Alto,

CA: Rockwell Science Center.

Militello, L. G., and Hutton, R. J. B., 1998. Applied Cognitive Task Analysis (ACTA): A practitioner's

toolkit for understanding cognitive task demands. Ergonomics, Special Issue on Task Analysis 41(11):

1618-1641.

Militello, L. G., Hutton, R. J. B., Pliske, R. M., Knight, B. J., and Klein, G., 1997. Applied Cognitive

Task Analysis (ACTA) Methodology. Final Report prepared for Navy Personnel Research and

Development Center, contract # N66001-94-C-7034. Fairborn, OH.: Klein Associates, Inc.

NASA, 1997. NASA Aeronautics Safety Investment Strategy Weather Investment Recommendations.

Washington, D.C: National Aeronautics and Space Administration. Available from World Wide Web:

(http://www.aero-space.nasa.gov/library/asist/images/WX_Summa.pdf).

National Business Aviation Association, 2000a. NBAA Business Aviation Fact Book: Excellent Safety

Record. Available from World Wide Web: (http://www.nbaa.org/factbook/2000/section4.htm, Accessed

11/9/00).
27

National Business Aviation Association, 2000b. J.D. Power Associates Survey. Available from World

Wide Web: (http://www.nbaa.org/data/, Accessed 11/9/00).

National Business Aviation Association, 2000c. NBAA Business Aviation Fact Book: The NBAA Fleet.

Available from World Wide Web: (http://www.nbaa.org/factbook/2000/section2.htm#02, Accessed

11/9/00).

Perry, R., 1999. Personal Communication. NASA Langley Research Center.

Phillips, E.H., 2000. Strong market prevails for business aircraft. Aviation Week and Space Technology:

53-54.

Rogers, B., Sly, J., Leard, T., Clark, L., 1998. A Descriptive Analysis of “Business Jet” Operations with

Comparisons to Airline Operations and Implications for Flight Deck Design. Final Report for NASA,

contract NAS1-20219, Task 18. Minneapolis, MN: Honeywell Technology Center.

Schutte, P.C. and Willshire, K.F., 1997. Designing to control flight crew errors. IEEE International

Conference on Systems, Man, and Cybernetic: 1978-1983.

Suchman, L., 1987. Plans and Situated Actions: The Problem of Human-Machine Communication.

Cambridge, UK: Cambridge University Press.

Yucknovicz, D., Burgess, M., Heck, M., Novacek, P., 2000. Assessment of the effects of delayed weather

information datalinked to the cockpit on pilot navigation decision making. Presented at IEEE/AIAA

Digital Avionics Systems Conference. Philadelphia, PA: IEEE/AIAA.
28

5.0 Acronyms & Abbreviations

ACARS - Aircraft Communications and Reporting System.

ACTA - Applied Cognitive Task Analysis toolkit (copyright, Klein Associates, Inc.).

AIM - Aeronautical Information Manual.

AIRMET- Airman’s Meteorological Information.

ASIST - Aeronautics Safety Investment Strategy Team.

ASOS - Automated Surface Observing System.

ATC – Air Traffic Control.

ATIS – Automatic Terminal Information Service.

AWIN - Aviation Weather Information.

AWOS - Automated Weather Observing System.

CDT - Cognitive Demands Table.

CTA - Cognitive Task Analysis.

DUATS - Direct User Access Terminal System.

ETA - Estimated Time of Arrival.

FBO - Fixed Base Operator.

FMS – Flight Management System

FSS - Flight Service Station.

FW - Flight Watch.

GPWS – Ground Proximity Warning System

IFR - Instrument Flight Rules.

IMC – Instrument Meteorological Conditions.

METARs - roughly translates from French as Aviation Routine Weather Report.

NBAA - National Business Aviation Association.

PIREP- Pilot Report.

SIGMET – Significant Meteorological Information.

SME - Subject Matter Expert.

SPECI - roughly translates as Aviation Selected Special Weather Report.

TAF - Terminal Area Forecast.

NAS - National Aerospace System.

NASA - National Aeronautics and Space Administration.

TV – Television.

UNICOM – Universal Communications station.

VFR – Visual Flight Rules.

WSI – Weather Services International, Corp.

WX – Weather.
7.0 Appendices

29
7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.)

A-1

ACTA - TASK DIAGRAM

Purpose: The Task Diagram is intended to serve as a road map to the rest of the CTA. The Task

Diagram acts as an advance organizer, providing an overview of the task and identifying the

cognitively complex elements of the task.

How to get started: Before you begin, have clearly in mind what the task is you intend to investigate.

In this interview, you want to find out about the interviewee’s processes as they perform the task of

interest.

CONDUCTING THE TASK DIAGRAM INTERVIEW

* Write the Task of Interest at top of whiteboard.

* Elicit the steps required to do the task. Record them across the board from left to right in

chronological order. Use arrows to indicate the order in which the steps occur.

- Ask your SME, “Think about what you do when you (Task of Interest) .

Can you break this task down into between three and six steps?”

* Elicit information regarding which of the steps require cognitive skills. Circle the elements that

require cognitive skills.

- Ask your SME, “Of the steps you have just identified which require difficult cognitive skills? By

cognitive skills I mean judgments, assessments, problem solving—thinking skills.”

At this point, you should have a very broad overview of the task, with an indication of where the

complex cognitive skills lie. If the task seems too big or the steps you have identified are too broad for

further investigation, you may choose to focus on one or two of the subtasks you have identified as

requiring cognitive skills. In this case, you should complete a Task Diagram on the step(s) you have

chosen to focus the rest of the cognitive task analysis.

TIPS FOR DOING THE TASK DIAGRAM INTERVIEW

* Your interviewee may immediately start talking at a very fine level of detail. Make it clear early on

that you are looking for a very broad overview with this interview. You will be interested in

hearing lots of stories and details later in the session (with the Knowledge Audit and the

Simulation Interview).

* If your interviewee begins listing things to consider rather than the steps of the task, help reframe

the topic for him/her. “What do you do when you (Task of Interest) ?”

* This may be a new way for the interviewee to think about the job. Give him/her time to think. You

may need to repeat or rephrase the question.

* The Task Diagram serves as a road map to the rest of the cognitive task analysis. You are not trying

to elicit detailed, specific cognitive information with this interview. You are trying to get a sense of

which parts of the task require complex cognitive skills.
7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.)

A-2

EXAMPLE: Task Diagram of Fireground Commander's Job in Commanding Crew

Getting the call Pre-assess

the

situation

A sse ss th e

scene upo n

arrival

Assign

task s to

crew

The interviewer decides this is too broad — really wants to focus on the assignment of tasks

during an incident.

EXAMPLE: Task Diagram of Assign Tasks

Do a quick

Search and

Rescue

Set up

hose line

Assign

overhaul

crew

Do a secondary

Search and Rescue

Ventilate the

building

G etting the call
7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.)

A-3

ACTA - KNOWLEDGE AUDIT

Purpose: The Knowledge Audit provides details and examples of cognitive elements of expertise; it

contrasts what experts know and novices don’t.

How to get started: You used the Task Diagram to identify parts of the job that require skilled

judgment, decision-making and evaluation. In the

Knowledge Audit you will elicit the expertise necessary to

do each of those tasks. Use the Task Diagram to help you

decide which tasks and subtasks you want to explore with

the Knowledge Audit. Go into the Knowledge Audit

interview knowing what you want to analyze.

CONDUCTING THE KNOWLEDGE AUDIT

* Write the Task of Interest at top, center of whiteboard.

*Divide the space below into three columns; label as shown.

* Elicit an example of one element of expertise, using the definitions and probes provided. Start with

the first probe, (e.g., “Is there a time when you walked into the middle of a situation and knew

exactly how things got there and where they were headed?)

* Elicit information for the remaining two columns before proceeding to another element:

- Ask your SME, “In this situation, how would you know this? What cues and strategies are you relying

on?” Record answers in middle column under “Cues and Strategies.”

- Ask your SME, “In what way would this be difficult for a less-experienced person?” What makes it hard

to do?” Record answers in final column under “Why Difficult?”

* It is important that you cover the six basic Knowledge Audit probes; you may also want to use some

or all of the optional probes.

TIPS FOR DOING THE KNOWLEDGE AUDIT

* Examples allow you to get at specifics and help you understand the task better. Ask for an example

for each element of expertise.

* Don’t try to write everything; but write enough so you will know later what was said and meant.

With practice you will develop a sense of the level of detail you need.

* Some of the questions may take a few minutes for the SME to answer thoughtfully; don’t rush; give

the SME time to think over what you are asking about.

* Confusion about what to write and in which columns may be a signal that the SME has

misunderstood your question; the information you are getting is not what you expect. You may

want to take a timeout, restate the question, and check that your SME understands what you are

trying to get at.

Task of Interest

Example

Cues

&

Strategies

Why

Difficult?

Past & Future

Example...

Big Picture

Example...

Noticing

Example...
7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.)

A-4

ELICITING INFORMATION WITH THE KNOWLEDGE AUDIT

Provide an explanation of the type of information you want; then ask the probe questions. You can

read the definitions below or paraphrase them.

BASIC PROBES:

* Past & Future Experts can figure out how a situation developed, and they can think into

the future to see where the situation is going. Among other things, this can

allow experts to head off problems before they develop.

Is there a time when you walked into the middle of a situation and knew exactly how

things got there and where they were headed?

* Big Picture Novices may only see bits and pieces. Experts are able to quickly build an

understanding of the whole situation—the Big Picture view. This allows the expert

to think about how different elements fit together and affect each other.

Can you give me an example of what is important about the Big Picture for this task?

What are the major elements you have to know and keep track of?

* Noticing Experts are able to detect cues and see meaningful patterns that less-experienced

personnel may miss altogether.

Have you had experiences where part of a situation just “popped’ out at you; where you

noticed things going on that others didn’t catch? What is an example?

* Job Smarts Experts learn how to combine procedures and work in the most efficient way possible.

They don’t cut corners, but they don’t waste time and resources either.

When you do this task, are there ways of working smart or accomplishing more with

less—that you have found especially useful?

* Opportunities Experts are comfortable improvising—seeing what will work in this particular

situation; they are able to shift directions to take advantage of opportunities.

Can you think of an example when you have improvised in this task or noticed an

opportunity to do something better?

* Self Experts are aware of their performance; they check how they are doing and

make Monitoring adjustments. Experts notice when their performance is not

what it should be (this could be due to stress, fatigue, high workload, etc.) and

are able to adjust so that the job gets done.

Can you think of a time when you realized that you would need to change the way you

were performing in order to get the job done?

OPTIONAL PROBES:

* Anomalies Novices don’t know what is typical, so they have a hard time identifying what is

atypical. Experts can quickly spot unusual events and detect deviations. And, they

are able to notice when something that ought to happen, doesn’t.

Can you describe an instance when you spotted a deviation from the norm, or knew

something was amiss?

* Equipment Equipment can sometimes mislead. Novices usually believe whatever the

* Difficulties equipment tells them; they don’t know when to be skeptical.

Have there been times when the equipment pointed in one direction, but your own

judgment told you to do something else? Or when you had to rely on experience to

avoid being led astray by the equipment?
7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.)

A-5

ACTA - SIMULATION INTERVIEW

Purpose: The Simulation Interview provides a view of the expert’s problem-solving processes in

context. The interview provides specific detailed information on expert cognitive processes.

How to get started: You will need to obtain a simulation of the task. The simulation you choose

should address difficult, challenging elements of the job. It does not have to be high fidelity; it can be

a paper and pencil simulation, a video depicting a scenario, or whatever is available. It is important

that the simulation you choose presents a challenging scenario.

CONDUCTING THE SIMULATION INTERVIEW

• Divide a whiteboard into five columns, labeled as shown on the next page.

• Have the SME experience (i.e. read, watch, interact with) the simulation.

- Tell the SME, “As you experience this simulation, imagine you are the (Job you are investigating) in

the incident. Afterwards, I am going to ask you a series of questions about how you would approach

this situation.”

• Elicit a list of the major events in the simulated incident and record in the first column.

- Ask your SME, “Think back over the scenario. Please list the major events that occurred during the

incident. These events could include judgments or decision points. As you name them, I am going to list

them in the left column of the board.”

• Begin with the first major event and elicit information for the remaining four columns before

proceeding to the next major event.

- Ask your SME, “As the (Job you are investigating) in this scenario, what actions, if any, would you

take at this point in time?” Record answers in the second column under Actions.

- Ask your SME, “What do you think is going on here? What is your assessment of the situation at this

point in time?” Record answers in the third column under Situation Assessment.

- Ask your SME, “What pieces of information led you to this situation assessment and these actions?”

Record answers in the fourth column under Critical Cues.

- Ask your SME, “What errors would an inexperienced person be likely to make in this situation?”

Record answers in the fifth column under Potential Errors.

TIPS FOR DOING THE SIMULATION INTERVIEW

• Eliciting major events is critical to this interview. The major events should be turning points or

segments of the story. You do NOT want a recount of the entire scenario.

• People often want to critique the simulation. Assure your interviewee that you are interested in their

critique, but that for the first part of the interview, you would like to work with the scenario as it

has been presented. Be sure to follow up and ask for a critique at the end.

• Don’t try to write everything; but write enough so you will know later what was said and meant.

With practice you will develop a sense of the level of detail you need.

• Confusion about what to write and in which columns may be a signal that the SME has

misunderstood your question; the information you are getting is not what you expect. You may

want to take a timeout, restate the question, and check that your SME understands what you are

getting at.
7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.)

A-6

EXAMPLE: EXCERPT FROM A FIREGROUND SIMULATION

Events Actions Situation Assessment Critical Cues Potential

Errors

On-scene

arrival

- Account for people

(names)

- Ask neighbors (but

don’t take their

word for it, check it

out yourself)

- Must knock on

doors or knock it in

to make sure

people aren’t there

- It’s a cold night,

need to find place

for people who have

been evacuated

- Night (time)

- Cold — 15 degrees

- Dead space

- Add on floor

- Poor materials

(wood) punk board

metal girders —

buckle and break

under fire

- Common attic in

whole building

- Not

keeping

track of

people

(could be

looking

for people

who are

not there)

Initial attack - Watch for signs of

building collapse

- If signs of building

collapse, evacuate

and throw water on

it from the outside

- Faulty construction:

building may

collapse

- Signs of building

collapse include:

what walls are doing,

cracks (building

ready to collapse),

floor groans (floor

ready to cave in),

metal girders

(click— coming out

of wall—popping),

cable in old buildings

holds wall together,

fire collapses walls

-Ventilating

the attic,

this draws

the fire up

and spreads

it through

the pipes

and

electrical

system
7.2 Appendix B: Preliminary Screening Questionnaire

B-1

1. General Information

Full Name: ________________________________________

First, Middle, Last

Address:

Street and Number, or P.O. Box

City, State, Zip Code, and Country (if not USA)

Home Phone: ( ) Work Phone: ( )

Birth Date:

Month/Day/Year

Do you wear corrective lenses when you fly?(circle one) Yes No

2. General Experience Information

Current/Most Recent Airline:

Current/Most Recent Position:

Captain, First Officer, Engineer, etc.

Are you currently flying military?(circle one) Yes No

Years Flying Commercial (approximate):

Years Flying Military (approximate):

Total Hours Flying (approximate):

Total Hours Flying as Pilot-in-Command (approximate):

Years of formal education: (e.g., high school graduate = 12)
7.2 Appendix B: Preliminary Screening Questionnaire

B-2

3. Specific Aircraft Experience Information

Please list the types of aircraft on which you have experience, beginning with the most recently flown.

For each aircraft, please check the columns to indicate your approximate number of hours flying

experience and approximate number of hours simulator experience.

If you were an Instructor (I) or a Check Airman (CA) on any of these aircraft, please indicate by checking

the last column.

Aircraft Type Hours in Type Simulator Hours I/CA ?

< 300 300-1000 > 1000 0 < 50 > 50

Please check the appropriate column to indicate the approximate number of years of experience you have

for each of the following categories:

Specific Aeronautical Experience Years Experience

< 1 1-5 > 5

Long-range, Over-water (Class II) Operations ( 2 engines)

Long-range, Over-water (Class II) Operations ( > 2 engines)

Total Multi-Engine (Captain or F/O, Military or Civil)

Glass Cockpit (i.e. EFIS/CRT or FMS)
7.3 Appendix C: Phase 1 Interview Guide

C-1

1. Inform the pilot that what they tell us will be confidential. Ask for permission to tape record

the interview. (Take good notes!)

2. State purpose of our research. Say something like:

Becky, Rob, and Jason work for a small R & D company called Klein Associates. Klein

Associates studies decision makers in many different domains in order to develop improved

training programs and to design decision support systems. Kara works for NASA Langley's

AWIN (Aviation Weather Information) program. We want to learn how pilots use weather

information when they make decisions before and during their flights.

3. Obtain background information on pilot [review preliminary questionnaire from

recruitment].

• Where, geographically, do you usually fly?

• How varied are the routes you fly?

• Are there any typical weather systems along your typical routes?

• What degree of pressure do you feel to make a trip?

• Under what FAR Parts do you operate?

• What types of "cargo" do you carry? Do you have any sensitivity associated with this

cargo?

4. Focus the interview. We’ve read that weather is identified as the causal factor in 30% of all

general aviation accidents. We have also read that most fatal weather-related accidents occur

during the cruise phase of the flight.

• Do you think these facts hold “true” for business jet operations?

• Why? Why not?

5. Use basic questions from the Task Diagram figure along with the list of the 12 phases of

flight to identify the most cognitively challenging tasks for the pilot that weather conditions

may impact.

6. Use the Knowledge Audit to follow-up on the tasks identified with the Task Diagram. Try

to elicit the pilot’s expertise for decision situations impacted by weather conditions. Try these

probes:

• Big picture

• Noticing

• Past & future

• Opportunities/Improving

• Anomalies

• Equipment difficulties

7. Obtain a list of all the weather products the pilot currently uses or has used in the recent past.

You can use the Weather Products Summary sheet to help collect and/or record this

information.
7.3 Appendix C: Phase 1 Interview Guide

C-2

8. Collect an incident. Try a probe something like this:

You can probably remember numerous flights in which you had to make difficult decisions

due to the current weather conditions. We would like you to describe an incident in which your

experience using weather products made a difference in how you handled the situation—a

situation in which a less-experienced pilot might have made a different decision than you did.

[Explore this incident for about 20-30 minutes, have some colored pencils & blank paper handy

to get diagrams.]

a. Construct the incident timeline. Be sure to relate this timeline to the different stages of

flight operations.

b. Review the timeline to verify the content and sequence of events.

c. Identify key decision points involving weather.

• Did you consider other alternatives?

• Might someone else in this situation have responded differently?

d. Situation Awareness probes:

• What was it about this situation that let you know what was going to happen?

• What led up to this decision?

e. Cue probes:

• What were you noticing at this point?

• What were you seeing (or hearing) at this point?

f. Knowledge probes:

• What information did you use in making this decision?

• How did you get that information?

g. Goal probes:

• What were your specific goals at this time?

• What were you hoping/intending to accomplish at this point?
7.4 Appendix D: Weather Products Survey

D-1

Pilot _____________________

Date _____________________

Wx

product

What

information

does this

product

provide? How

do you use this

product?

Where do

you get this

product/

When is it

used (preflight, on

flightdeck)?

How

credible/

trustworthy

is the

information

provided?

How

current do

you need

this

information

to be?

How

important is

this

information?

What impact

does it have

on your

decisions?

Additional

comments/

notes
7.5 Appendix E: Phase 2 Interview Guide

E-1

Introductions

• Background information on AWIN

• Consent forms

• Klein Associates Inc.

• Schedule for the day, facility layout, overview of interviews

Generating Cognitive Demands

• “We want you to tell us about what you do when you fly a mission. We are interested in the

judgments and decisions you make throughout your typical mission that are affected by weather.

We have identified eight different phases of flight. For each phase we will ask you to describe

the judgments and decisions you make that are directly affected by the weather.”

• Use the phases of flight diagram to go through each of the steps listed on the following pages

(e.g., “For the General Planning phase, what decisions/judgments do you make that are impacted

by weather?”) Keep the participant focused on judgments and decisions that are related to

weather. If they cannot generate their own cognitive demands, use prompts from the list.

{Additional probes for eliciting cognitive demands for the different Phases of Flight }

• General Planning (24-48 hrs prior)

What time should I depart/should my passengers be ready to depart?

What are the general Wx problems for the day going to be? Factors to be aware of in Specific

Planning?

Is the destination feasible? Go/No Go?

Is departure point feasible?

Is there any reason why I shouldn’t fly at all? (Ex: menacing fog at departure or destination point)

• Specific Planning (30 mins – 2 hrs prior)

[When is flight plan filed? How long does it take to put together a flight plan?]

How long will the flight take?

How much fuel will I need? (wind considerations)

What’s my best route? (wind or other Wx problems)

Are there any Wx obstructions? (fronts, t-storms)

Is it safe to takeoff? If not, when?

Will it be safe to land?

Do I need to plan for alternates? Which alternates?

Will I be able to get back in sufficient time?

• Taxi/Take-Off

IFR or VFR?

Can pilot see runway? Runway contaminated? (ice, snow, foreign objects)

Is there risk of icing?

Are there dangerous wind conditions – downdrafts, windshear, etc? (aircraft configuration for

taxi-ailerons)

Request alternate runway to takeoff from?

Are there aircraft configuration issues to think about for taxi due to wind?

• Climb

Is there risk of icing?

Is aircraft performance affected by any current Wx condition?
7.5 Appendix E: Phase 2 Interview Guide

E-2

• Cruise

What’s my destination like, will it still be safe to land? Is destination still viable?

Will I need to divert to an alternate? Which alternate? (Holding and Wx problems)

Is my route still appropriate (obstacles)? Change route?

Is this altitude comfortable? (turbulence)

Is my heading appropriate for navigation (wind)?

Am I getting appropriate fuel consumption? (speed and altitude)

Am I getting expected ground speed? Will I have enough fuel?

Will I need to refuel on the way? If so, where?

Is there risk of icing?

Is there risk of lightening strike?

Will I be there on time? Do I need to reschedule anything or communicate with work to

reschedule meetings, etc.?

Do I need to communicate with passengers and/or 3rd parties about passenger schedules?

Do I need to reschedule ground transportation at primary destination; arrange ground

transportation at alternate airport?

• Descend

Holding? Is it safe to enter holding pattern? Do I need to reschedule/coordinate with ground

transportation?

Are there weather phenomena that will affect aircraft performance (icing, severe winds, t-storms)?

Is there severe windshear? Turbulence, microburst/downdraft activity?

Is my destination still viable? Do I need to divert to an alternate?

Call FBO? Call tower?

• Approach/Land

Am I going to land?

Is my destination viable? [at this stage is the pilot already committed to land?]

Do I need to go-around? Do I need to divert?

Is the runway clear (visibility, ice, snow)? Request alternative runway due to Wx? (wind, runway

obstructed)

Is the ceiling OK? Is the visibility OK?

Effects of crosswinds?

Are there downdrafts/microbursts? Windshear?

Do I need to request an alternate runway?

• Taxi/Deplane

Obstructed runway? (ice, snow, other objects)

Is the wind a problem (Passenger safety; am I going to get blown away; aircraft configuration)?

Keep the plane inside or outside hanger?

• Concurrent Tasks in all Phases of Flight

[These issues may be present during all phases of flight, but are they always equally important

considerations? Are there some times when these issues are more critical and therefore require

more apparent decision support?]

Is the Wx affecting my aircraft performance? What are my limits before I act, related to Wx (e.g.,

route, fuel, altitude, speed, schedule, safety, comfort, aircraft performance)

Icing considerations?
7.5 Appendix E: Phase 2 Interview Guide

E-3

Conduct the Simulation Interview using the ACTA Simulation Interview Guide

Knowledge Audit Probes

• Anomalies - New pilots don’t know what is typical, so they have a hard time identifying what is

atypical. Experienced pilots can quickly spot unusual events and detect deviations. And they are

able to notice when something that ought to happen, doesn’t.

Can you describe an instance when you spotted a deviation from the norm (either in weather

information on the flight deck or out the window), or knew something was amiss, that a new pilot

wouldn’t notice? Example: After takeoff a pilot immediately noticed that the winds were not in

the forecast direction on which his flight plan was based . . .

• Equipment difficulties / unreliable information - Equipment and information sources can

sometimes mislead. Information received from sources such as distanced operators is sometimes

unreliable. New pilots often believe whatever the equipment or other information source tells

them; they don’t know when to be skeptical.

Have there been times when weather-related equipment (weather information and/or weathercompensating equipment, e.g., deicing boots) pointed in one direction, but your own judgment

told you to do something else? Or when you had to rely on experience to avoid being led astray

by weather information or weather-related equipment? Example: While enroute to his

destination, a pilot became concerned about the reliability of the radar image he was getting on

the flight deck . . .

• Big Picture - New pilots may only see bits and pieces. Experienced pilots are able to quickly

build an understanding of the whole situation – the big picture view. This allows the experienced

pilot to think about how different elements fit together and affect each other. This can allow them

to anticipate problems that might develop in the future.

Can you give me an example of what the Big Picture looks like for this task? What are the major

elements of the weather that you need to know about and keep track of ? How does this help you

know where things are headed? Example: A pilot needed to fly around a hurricane to get to his

destination, which had an uncertified AWOS operating. He had to rely on a variety of different

sources of information to build his understanding of the weather situation including his past

experience with similar airfields and other weather information.

• Noticing - More experienced pilots are able to detect cues and see meaningful patterns that lessexperienced pilots may miss altogether.

Have you had experiences where you noticed things going on that indicated a potential weather

problem that others didn’t catch? Example: When he was descending through a cloud, a pilot

noticed a drop in temperature and the presence of precipitation. This pattern of cues alerted the

pilot to potential icing conditions—normally he would expect temperature to increase as he

descended.

Elicit an incident using the Critical Decision Method

“You can probably remember some flights in which you had to make difficult decisions due to the current

weather conditions. We would like you to describe an incident in which your experience made a

difference in how you handled the situation—a situation in which a less-experienced pilot might have
7.5 Appendix E: Phase 2 Interview Guide

E-4

made a different decision than you did.” Note: If participant has difficulty recalling an appropriate

incident—use the Scenario from Hell technique below.

• Construct incident timeline, relating the timeline to stages of flight operations

• Review the timeline to verify the content and sequence of events

• Identify key decision points involving weather

- Did you consider other alternatives?

- Might someone with less experience have responded differently?

- To what degree was your decision based on past experiences, on formal training, on

other’s experiences?

• Situation Awareness probes

- What was it about this situation that let you know what was going to happen?

- What led up to this decision?

• Cue probes

- What were you noticing at this point (visual-external, visual-instruments, auditory,

aircraft response)

- Did you actively retrieve any additional information?

- Did you retrieve any additional information for the purposes of validating information

you had?

• Knowledge probes

- What information did you use in making this decision?

- Did you have any information that you didn’t trust enough to use it in making this

decision?

• Goal probes

- What were the specific goals you hoped to achieve by making this decision?

- What were you hoping to accomplish?

- Besides weather, what other factors were concerns?

Elicit a "Scenario from Hell"

“How would you design a weather-related “scenario from hell” for pilot simulation testing that would be

difficult, but not impossible for another pilot. In particular, we are looking for weather-related events

that are difficult to detect, ambiguous, insidious conditions, or conditions that, in conjunction with some

other event or system failure, may result in an unsafe condition if not considered.”

• Review the incident to verify the content and sequence of events

• Identify key decision points involving weather

- Might someone with less experience have responded differently?

• Situation Awareness probes

- What about this situation that would let a pilot know what was going to happen?

• Cue probes

- What would an experienced pilot have noticed at this point (visual-external, visualinstruments, auditory, aircraft response)

• Knowledge probes

- What information would an experienced pilot use in making this decision?

• Goal probes

- What would the pilot be hoping to accomplish?

• Besides weather, what other factors would an experienced pilot be concerned with?

• Why would this situation be difficult for a less-experienced pilot? Where might they go wrong?
7.6 Appendix F: Simulation Interview Scenario and Material

F-1

Mission:

Current time is 4:00 pm. You have a pop-up trip taking the CEO and four other passengers to

Jacksonville, FL. You arrange to depart from Norfolk airport at 5:30 pm. There is an evening meeting 10

miles from the terminal at 8:00 pm that the passengers must make. You will also need to make a return

trip the following morning to be back in Norfolk for a noon meeting. At the meeting, the CEO will

discuss the bonus plan for the pilots and you must be there to collect.

You are on the runway and making final preparations for take-off. Outside your window you see

a windsock flying horizontal indicating that winds are gusting out of the Northeast at about thirty knots.

It has been drizzling off and on during the day, however, you see no drizzle on the runway. Visibility is

about seven statute miles and it is slightly overcast with a low ceiling at about one thousand feet. Current

temperature is two degrees Celsius with a dew point of one degree Celsius.

As you ascend to cruising altitude at flight level three five zero you're flying in and out of patchy

clouds. The weather outside your window is clear now, but there are t-storm cells to the South forming

large convective cumulonimbus clouds. The temperature outside is about negative forty-seven degrees

Celsius. The flight is getting a little bumpy and passengers are beginning to make comments about the

uncomfortable ride.

You are listening to pilot frequencies to try to get a feel for what's ahead. You hear, "Charleston

Northward at two two five zero, moderate turbulence at flight level three three zero, B757."

Above flight level two four zero, you can see towering cumulous cells along your flight path

Northwest of Wilmington, approximately 25nm South of Wilmington, a large system due East and 50nm

from Charleston, and a smaller cell almost directly over Charleston International.

Another pilot report indicates, "Dash Eight reported moderate mixed icing, forty miles East of

Brunswick between six thousand and one zero thousand feet at two three one zero Zulu."

When you start your descent towards Jacksonville, a pilot report conveys, "Moderate turbulence

at two three two zero Zulu, on approach to runway 31 inside VOR at Jacksonville International, reported

by DC8."

Current conditions at Jacksonville indicate winds out of the Northwest at thirteen knots and

gusting to twenty-one knots. There is a light rain and visibility is about two statute miles. Cloud ceiling

is at one thousand two hundred feet. Current temperature is five degrees Celsius, with a dew point of four

degrees Celsius.

Materials Provided:

Preflight Enroute Approach Charts

Scenario Mission PIREPS - ride reports ATIS recordings Approach Charts

Aircraft Description Visual descriptions PIREPS - terminal area Airport Diagrams

Standard Briefing

(DUATs encoded text)

FW scripts METARS IFR enroute low alt.

Weather Graphics:

Current Surface

24 Hour Surface

48 Hour Surface

Current Flight Rules

Doppler Radar

Lifted Index

Satellite Image

Weather Hazards

IFR high altitude

Sectional charts
7.7 Appendix G: Post-Experiment Questionnaire

G-1

Dear Participant:

This questionnaire complements the interview process in which you have just participated. Your

responses on this questionnaire will help us interpret the results of your interview data and help us address

some issues that we didn’t have time to cover in the interview process.

The questionnaire has the following sections: 1) Characterizing Business Jet Operations, 2) Personal

Limits, 3) Weather Flying & Aircraft Characteristics, 4) Weather Training, 5) Use of Weather Products,

and 6) Weather Information Requirements for Business Jet Operations. The Characterizing Business Jet

Operations section will help us get a better feeling for your operations and allow us to better interpret

your interview results. In addition, it will help us define the special requirements and challenges of the

business jet community. The Personal Limits section will help us understand how you might constrain

how you fly as a function of weather and atmospheric conditions. The Weather Flying & Aircraft

Characteristics section will help us understand how you use your knowledge of your aircrafts’

capabilities, and the equipage of your aircraft to make decisions in challenging weather situations. The

Weather Training section will help us interpret your interview and questionnaire responses. In addition,

we will better understand the breadth and depth of weather-specific training provided to business jet

pilots. The Use of Weather Products section will help us understand the weather products (services,

PIREPS, AIRMETS, graphics, ...) you use, and the degree to which you find these products reliable and

useful. Finally, the Weather Information Requirements for Business Jet Operations section will be used

to complete a study of pilot weather information requirements by adding the needs of business jet pilots to

those compiled for Commuter, General, and Transport pilot populations.

Please complete this survey and mail it back to me in the attached self-addressed and postage-paid

envelope. I hope to receive all responses back by February 29. If you are unable to return the survey

before then, please simply call me and tell me when you think you will be able to return it.

We are looking forward to developing prototype concepts for improving weather decision-making based

on the interview and questionnaire results. We hope to see you again in the development and evaluation

phases of these concepts. Thank you again for your participation in this study. Your time and efforts are

much appreciated.

Sincerely,

Kara Latorella, Ph.D.

Crew/Vehicle Integration Branch, AWIN Operator Support

M/S 152 NASA Langley Research Center

Hampton, VA 23681-0001

k.a.latorella@larc.nasa.gov

Phone: 757-864-2030, Fax: 757-864-7793
7.7 Appendix G: Post-Experiment Questionnaire

G-2

1. Characterizing Business Jet Operations

1.1 Owner/Operator Characteristics:

1.1.1 Size of Fleet: _________ aircraft owned _________ leased

1.1.2 Frequency of fleet use: _________ trips per _____________

1.1.3 Support structure (circle those you have in-house):

dispatcher, initial training, recurrent training, maintenance, procedure/ checklist development

1.1.4 Would you characterize the aircraft you fly as: < 20,000lb 20,000-40,000lb > 40,000lb

1.1.5 Do you share aircraft with other organizations? Yes No

1.1.6 Pilots are: Owners/Low Time Professional Crew

1.1.7 FAR Operations (circle all that apply):

121 135 91

1.1.8 Typical crew: ________ pilots + _______ non-piloting crew

1.2 Pilot Perspective:

1.2.1 Advantages of being a business jet pilot:

1.2.2 Disadvantages of being a business jet pilot:

1.2.3 Tasks you do beyond those of a commercial pilot:
7.7 Appendix G: Post-Experiment Questionnaire

G-3

1.3 Mission Characteristics:

1.3.1 Typical types of passengers:

1.3.2 Typical reasons for flights:

1.3.3 Amount of notice usually given for a flight: _____ to _____ (range), usually around ______

1.3.4 How often are last minute changes to the trip? every time 1/2time never

1.3.5 Typical trip length > 300nm, > 1000nm, > 2500nm

1.3.6 What % of your trips are to a novel destination? < 1% < 10% ~ 50% > 75%

1.3.7 To how many different “typical” destinations do you fly: _________________

1.3.8 Do you fly international trips? NO / YES: To where? _____________

1.3.9 Do you fly over water trips? NO / YES: How long over water? _________

1.3.10 Add any goals that are missing from the

following list, and rank order these goals according to

how important they are for your typical mission (i.e.,

more important goals may be attained by sacrificing less

important goals).

_____ precision of ETA at remote destination

_____ precision of ETA on return to home base

_____ fuel economy

_____ aircraft operating costs

_____ company transportation costs as a whole

_____ flexibility in mission

_____ passenger comfort

_____ ________________________ (other)
7.7 Appendix G: Post-Experiment Questionnaire

G-4

1.4 Operational Elements

1.4.1 Factors you consider in planning a flight (especially any unique to business operations):

1.4.2 Define a “good weather day.”

1.4.3 What is the ratio of flights that you fly in bad weather days (less than “good weather days”).

1.4.4 What would be “normal” bad weather? (you’d go, you think you’ll be fine, it’s just unpleasant).

1.4.5 What would be “alerting” bad weather? (you’d go, but think you might be coming back).

1.4.6 What would be the lower end of “prohibitive” bad weather (you wouldn’t go)?

1.5 Problems

1.5.1 What are the biggest problems associated with business jet operations within your operational center?

1.5.2 Please describe the biggest problems associated with the aircraft you usually fly.

1.5.3 What are the biggest problems of business jet operations as part of the NAS?
7.7 Appendix G: Post-Experiment Questionnaire

G-5

2.0 Personal Limits

Personal limits are constraints on whether you fly, how you fly (i.e., manage the aircraft), as well as how

you would perform other pilot duties during a mission (i.e., communications, systems management, task

management). As such, when the conditions of a Personal Limit are met, a pilot will operate at a point

inside the boundary established by “possible operations” (i.e., those permitted by FARS, those made

possible by aircraft capabilities). Personal limits can be expressed in terms of IF/THEN rules: the “IF”

describes the conditions under which you would exercise this limit, the “THEN” describes how you

constrain your behavior because of this limit. Personal limits can be distinguished from “Organizational

Limits.” A Personal Limit is one that you have determined through your own experience, by listening to

others’ experiences, or that you have derived from your basic knowledge of weather and atmospheric

phenomena, and aircraft performance characteristics. An Organizational Limit is one that has been

provided to you through formal training, company procedures, aircraft pilot operating handbook.

Please use the following table to describe the Personal and Organizational Limits that constrain your

behavior beyond what is aerodynamically possible and permissible by regulatory agencies: 1) Indicate

whether the limit is Personal (P) or Organizational (O), 2) Describe the task or decision that the limit

affects, 3) Describe any Weather/Atmospheric Conditions that pertain to this Limit, 4) Describe how this

Limit alters whether or how you conduct a mission.

P/O* Your Personal/

Organizational Limit (“IF”)

Weather / Atmospheric

Conditions

If weather/atmosphere is worse than limit,

what do you do? (“THEN”)

3.0 Weather Flying & Aircraft Characteristics

3.1 Please describe equipment (standard or special) that is on your aircraft that allows you to fly more safely in

different weather conditions.

Weather / Atmospheric Condition Aircraft Equipment

3.2 Please describe how performance characteristics of the aircraft you usually fly affect how you fly in weather

conditions.

Weather / Atmospheric Condition Aircraft Performance Characteristic
7.7 Appendix G: Post-Experiment Questionnaire

G-6

4.0 Weather Training

It is important for us to be able to interpret your responses during the interview and

on this questionnaire in terms of your knowledge of weather flying. In the areas

below, please list the formal classroom training, instructional videotape, computer

training, and readings from which you have acquired skills and knowledge for flying

in weather, for using weather services, and for using onboard weather-related

equipment (in particular, onboard weather radar).

classroom training videotapes computer-based

training

readings

flying in weather

using weather

services

using onboard

weather radar

using other

onboard

equipment

5.0 Your use of Weather Products

The purpose of this questionnaire is to survey how biz-jet pilots use weather products (e.g., surface-level

charts, the weather channel, internet) and weather resources to make decisions during flight missions. We

want to know which products and resources biz-jet pilots use, where they get access to these products and

resources, what information pilots look for in the products and resources they use, and in which stage of

the mission the information is useful. The next page shows a table with six columns and multiple rows

underneath. At the top of each column is a specific question related to specific weather products that you

as a biz-jet pilot would use before or during a mission. These columns are:

1. Wx product. This could be a specific product (e.g., pressure charts for the entire U.S.) or specific

resources of where you get your weather products (e.g., a weather internet site, on-board radar, ATIS).

2. What information does this product/resource provide? How do you use this product/resource?

3. Where do you get this product (from the internet)? / When is it used (during pre-flight)?

4. How credible/trustworthy is the information provided?

5. How current do you need this information to be?

6. How important is this information? What impact does it have on your decisions?

Please start by identifying each weather product or resource you use during a typical mission, and then

answer all subsequent questions about that specific product or source. We define a mission in the

following order: General planning (24-48 hrs prior), Specific planning (1-2hrs prior), Taxi out and take

off, Climb, Cruise, Descend, Approach and Landing, and finally Taxi in and deplane. We recognized that

this may not be exactly what you define a mission to be, but would like you to focus on these stages.
7.8 Appendix H: Individual Cognitive Demands Tables

H-1

Subject 1 - Cognitive Demand Table

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning General planning—24-

48 hours ahead--Whatwill wx be like enrouteand at destination?

Wx channelOn-line wx sources

Specific planning—12hours ahead—whatare current conditions atdestination?

Wx affects altitude,route selection and fuelrequirements

NWS forecastsOn-line wx sourcesReport from FBO atdestination

Contact FBO at destination 

Preflight planning—1-2hours ahead—should Ibrief Passengers aboutlikely delays?

Take-off Runway conditions,

winds, ceiling, visibility

Gets ATIS report before hestarts the engines

ClimbCruise Do I need to adjust my

speed?

Wx affects aircraftperformance (dependsin part on aircraftcapabilities); Wxinformation obtainedduring specificplanning may be “old;”On long flights, windsaloft affect fuelconsumption and routeselection

Fuel consumptionForecast windsDistance left to travel

May call FSS (for longerflights)Speed across ground isdifferent than air speed; heuses headwind instruments todetermine his ground speedWhen necessary to avoidrefueling stop, he will slowdown to reduce fuel use

How do I avoid t-stormcells?

On-board radar
7.8 Appendix H: Individual Cognitive Demands Tables

H-2

Descent/Landing Which frequency should

I use to get Wxinformation?

Uses ATIS information orAWOSHe will change his radio freqto get different sources ofWx info

Is wx okay for landingat destination?

Landing at small,uncontrolled airports ischallenging (becausethere may be no one toprovide wx update);Airport may haveuncertified AWOSoperating

PirepsWind speedWind directionCapabilities of his aircraft

Contact tower or FBO forWx update (for airfields w/otower he uses AWOS orUNICOM frequency)When flying to uncontrolledairport he does more in thespecific planning stageIn bad wx, he sets morerestrictive limits on windspeed/directionHe considers the capabilitiesof theAC he is flying;Thinks about landing atsimilar airports in the past(Ex: Hilton Head and St.Simons Is)

Do I need to changedestination?

Keeps alternate destinationsin mind

Should I go into aholding pattern ordivert?

Fog at originaldestinationPassengers needs

If diverting, need torearrange groundtransportation

Will I be able to see totaxi?

Moderate to heavy rainon the groundHis flying skills

Rely on non-flying pilot tomonitor Wx situation(although “chief” pilotmakes the decision)
7.8 Appendix H: Individual Cognitive Demands Tables

H-3

Subject 2 - Cognitive Demand Table

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning What’s the wx at my

destination?

Watches Wx channel or getsNOAA radar and satelliteimages on internetGets forecast for destinationfrom FSS for his flight plan

Takeoff Can I take off? Wind sheer

(downbursts)Icy runways

Gets information from ATIS

Cruise Avoiding turbulence Jet stream

Vertical clouds

Look out the windowListen to determine whatother pilots are doing

Avoiding t-storms Onboard radar

Vertical cloudformations

Look out the windowListen to determine whatother pilots are doing-HIWASTook a course on how tointerpret on-board radarpictures—at certain elevationsyou can pick up groundreturns (from his experience)

Is there an icing problem at lowaltitudes?

Low cirrus cloudsAircraft capabilities

Look out of window

Should I slow down to see ifWx at destination will clear?

Call ATIS 100 miles outGo into holding pattern or justreduce speed

Descent/landing Can I land? Wind sheer

Icy runwaysCeiling,visability

Observes from about 100miles out.Uses ATIS via radioCheck wx at airports near hisdestination.Call FBO at destinationRearrange when on groundtransportation if diverting

Is there an icing problem? Aircraft capabilities
7.8 Appendix H: Individual Cognitive Demands Tables

H-4

Subject 3 - Cognitive Demand Table

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning Is the route safe? Wx is changeable and

forecasts have limitedaccuracy (in terms ofresolution and shelf-life)

Wx briefing; wxchannel, internet wx

Is the destination safe? Wx is changeable and

forecasts have limitedaccuracy (in terms ofresolution and shelf-life)

Wx briefing; wxchannel, internet wx

Is destination close tominimums?

FSS not reliable becausethey rely on automatedweather obs equipmentand don’t have eyes on;need to check regularlyin case they change, esp.if the ATIS has a lot ofamendments (i.e. Wx

changeable)

Check more frequentlyif numbers are close tominimums; check afterthe hour for the latestupdated ATIS

Determining take offtime

Weather may causedelays

Leave an hour earlier ifweather is a possiblefactor (rule of thumb: ifsevere Wx approaching,leave one hour before itspredicted arrival)

Will aircraftperformance be anissue?

A/c performanceimpacts fuelconsumption and ETA

Temps (hot and cold)

Do I try to go throughthe gap in the Wx that Isee on my radar?

Don’t always know howbad the weather is inthat “gap”; don’t alwaysknow what’s on theother side of the gap

Radar picture;sensitivity of thepassengers
7.8 Appendix H: Individual Cognitive Demands Tables

H-5

Concurrent Will windshear be a

problem?

If you misjudge thepossibility of windshear,the consequences aresevere. No directindication of severity ofwind shear.

Wind speed, direction,changeability; GPWSindications; FMSindications (estimatedfuel consumption basedon winds aloft forecastis off); t-storm activityincreases likelihood;FW; ATIS, PIREPs;visual cues (treesblowing, rain direction,waves on water)

Carry extra speed toovercome loss in lift dueto change in winddirection; add power;recalculate fuel usage;change altitude forbetter winds

Will icing be an issue? In clouds it’s hard to

visually see icing; nodirect indications oficing; if there’s atemperature inversion,you may get unexpectedicing on descent; esp.tricky if flying slow (ina hold or on descent)because lift is impactedby icing on wingsurfaces

Temp; precipitation ormoisture (clouds, rain,fog);

Look for ice onwindshield, on wings,on boots; if skimming orgoing through clouds(and temps areappropriate 10 degreesC or below) put deicingon as a ROT; disconnectautopilot (so that itdoesn’t compensate forloss of trim, with-outpilot realizing it)

Can I trust my radarpicture?

Sometimes it “paints”ground clutter and thatcan be confused withprecipitation

Angle of tilt on the radar Adjust tilt of radar;

don’t trust it beyond 100miles
7.8 Appendix H: Individual Cognitive Demands Tables

H-6

Subject 4 - Cognitive Demand Table

Phase of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning How much fuel to carry,

Do I want to carry“bingo fuel” - extra fuelto give me options ofdifferent destinations.

Constrained by aircraftperformance; aircraftdoesn’t fly well at highaltitude with warm air –poor speedIt is difficult to predictwinds aloftIt is difficult to predict ifpilot will need tohold/divertHave to be able to planaccordingly so as to landwith a min. amount offuel

Temperature and windsat specific altitudesNumber of people flyingDistance of flight

Don’t carry “bingo fuel”if doesn’t need it – addsextra weight, whichmeans restricted tocertain altitudes

Fly in poor visibilityconditions (e.g., 300ft,1⁄2 mile)

Difficult to see runwayor other objects

What are the ceiling andvisibility readingsWhat is the temperatureif going thru clouds –icing problems

Set cockpit foremergency returnTurn on anti-ice beforetakeoff

Route planning Universal service gives

most direct flight, whichcould go right thrustorm center – have tobe aware of that

Where are storms, howare they moving, whatspeed are they moving?Where are fronts, squalllines, and pressure cells?

Call Universal FlightService to speak withexperiencedmeteorologist – morereliableFly around bad wx.Summer time watch forwx in “tornado alley”

Looks at general wxforecast concerns(departure and destin.)

Satellite, surface maps,Knowledge of LandTopography

Compare 10 minute-olddata to own big pictureCompare own numbers

with flight plan; if
7.8 Appendix H: Individual Cognitive Demands Tables

H-7

What are Minimums?Tops of clouds andheight of tropopause?

anomalies spotted, lookdeeperCall Universal FlightPlan Service – morereliable talking to realwx forecasterBuild big picture – withradar summary andgraphics.

Takeoff“business attitude”approach

Consider a “rejectedtakeoff” prior todecision speed

Hot weather – thermalscoming off runwaycreate easier lift, andtherefore less decisiontime and room forcorrectionShort runway – less timefor decision to be madeWinds in terminal area

Convection in terminalSpontaneous lightningMechanical issues

Checking forcontaminated runway(ice, deer)

Flying into anunfamiliar airport,perhaps no FBO ongroundShort runways are aproblem – may have tochange numbersmanually in own head ificing on runwayNeed more runwaywhen anti-ice is on

Ambient air tempTraction is poorBraking action reportsAltitude of runwayPIREPS

Visually inspectUse anti-ice for takeoffUse Field LengthRegulation Tables

Check that configurationof aircraft and cockpit isset [This is procedural]

Flaps up, gear up/down,etc...

Look out window, usegauges

Climb Looping around inbound

traffic

Sound abatement
7.8 Appendix H: Individual Cognitive Demands Tables

H-8H-8

Set radar scale Low watt radar is not

functional beyond50miles, it paints toomuch, doesn’tdiscriminate wellbetween precip andother clutter

Have to play aroundwith power level to getgood setting and useablepictureT-storms pop-up quicklyand they can be difficultto pick-up with low-wattradar

Keep on same settingand look for changes

Above FL250, ask fordirect route

May have to changeroute because of wxahead

Knowing where poorweather is

Stay on upwind side ofthe storm

Cruise Check to see if wx at

destin. is poor, may callFSS to check for currentwx

Some FSS have liveradar, some don’t

What are current trends;getting better or worse?

Check to see if theyshould slow down,change destin., reassessfuel, estimate hold timeTalk to FSS just afterthe hour – FSS amendsthe ATIS if wx is gettingworse or better duringthe hourTalk to another airport

Watching for turbulence Can’t see it, clear-air

turbulence - happensafter a front has passedthrough or due tomountains disturbing airflowWeight of plane andamount of fuel affecthow to react

Eddies are formed oneach side of jet streamTops of t-storms, overallcloud topsTurbulence also found atconverging jet streamsLand topography –mountain ranges createturbulenceWind and temp at eachaltitude

Ask ATC what altitudeis smoothListen for PIREPS
7.8 Appendix H: Individual Cognitive Demands Tables

H-9

Request wx avoidance,change in altitude

Aircraft performancerestrictions

Looking for thunderheadsAvoid icing conditionsTemps under 10 degreeCelsius

Don’t want to go toohigh, may stall outUse de-ice through“soggy clouds”Try to get 25-30 milesdownwind of convectivewx or stay on theupwind side of wx cell

Reroute Difficult to anticipate

changing wx

Poor weather en route(t-storms, icing,turbulence and winds)

Decide early if youthink you want tochange route

Descent/Landing Call for auto wx report

at destin. to see if theycan initiate landing

Perform this 150-180miles out (get skyconditions, precip.,pressure)

Non-precision approachat unsupported airfield

No help on the groundDon’t know whatrunway conditions are

What are minimums?Temp and dewpointspread, and wind(direction and speed)

Conducts visualinspection of runway“fly by”Better to fly onautopilot, better SA

Taxi How long to stay at

destination?

If no de-icingcapabilities, deplaningcan be tricky

Taxi/Deplane How long to stay at

destination?

If no de-icingcapabilities, deplaningcan be tricky
7.8 Appendix H: Individual Cognitive Demands Tables

H-10

Other T-storms Deep Low cell indicates

lots of turbulenceBackside of a low, hasresidual effectsLooking forthunderheads, if there isgrowth or dissipation- istowering then they are atmature stage and havemuch energyWhat’s the distancefrom the plane

Listen or look for severewx reportsPIREPSChange altitude or goaround
7.8 Appendix H: Individual Cognitive Demands Tables

H-11

Subject 5 - Cognitive Demand Table

Phase of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning What’s my route? Need to know where

any weather is, also anyspecial airspacerestrictions; may be apop-up trip with only acouple of hours notice

WSI Wx briefing; Wxchannel; look out thewindow

Route upwind of theweather, behind thefronts; carry extra fuelin case need to hold ordivert;

How much extra fuel will Ineed?

Need to anticipatepotential delays forholding en route and atthe destination; need toanticipate aircraftperformance based onpayload and use of antiicing

Wind; Payload; Potentialdelays at destination dueto Wx;Icing forecasts (precipand temps);Icing PIREPs and radiochatter

Carry extra fuel (halfhour or 500lbs)

Are there any weatherconcerns today?

Depends how recent theforecast is; PIREPS canbe unreliable; forecastscan be unreliable

WSI Wx briefing; pilotchatter on radio andPIREPs

Look for keyinformation in Wxforecasts (e.g. use ofthe words “severe”),anything relating todeparture or destinationfields, or en route; fastchanging weather; wxnear minimums

Is the destination viable? Have to anticipate Wx

at the destination atarrival time; forecastsare unreliable

METARS fordestination;AIRMETs/SIGMETs fordestination; PIREPs fordestination; delays atdestination

Read Wx briefing, listento radio chatter

What additional precautionsdo I need to take?

Don’t know all thepossible problems
7.8 Appendix H: Individual Cognitive Demands Tables

H-12

Taxi/Takeoff Can I handle the crosswind? Hard to anticipate gusts

or wind shear;crosswind component isdependent on angle &/speed of winds (nodirect crosswindcomponent indicator)

Wind direction, windspeed, pilot and ATCreports of crosswinds(chatter and PIREPs);can I see bad weather(dark menacing clouds)?

Request differentrunway; know aircraftlimits and personallimits; use “split”controls procedure (onepilot has throttle andrudders, the other hasthe yoke)

Should I turn on the antiicing equipment? Turning on anti-icing can affect aircraft

performance and makeyour ascent slower, thusproviding moreopportunity for icing,it’s trade-off

What is the temp?;Is there precip/moisture?Length of runway; iswindshield icing up (lookat windshield icing lightreflector)

Use radar to look forprecipitation in the area,use FMS or TAF to getair temperature; listenfor PIREPS, or ATCreports of icing; add.5hour (500lbs) fuel

Can I taxi out in this? Potential for damaging

brakes, wheels, flaps;potential for loss ofbraking ability

Snow, slush, or standingwater on the tarmac

Taxi with flaps in; listento PIREPs and brakingreports

Do I need to be careful ofrunway contamination?

Can’t see standingwater; no directindication of length ofrunway and aircraftperformance limits

Airport facilitiesdirectory for runwaylength; look for standingwater; get PIREPS andlisten to radio chatterconcerning snow, slush,water, othercontaminants)

Climb Should I turn on the antiicing? Anti-icing impacts aircraft performance

(see above)

Temp? Moisture?Aircraft performancecharacteristics and feel;length of runway;

Do I need to deviateimmediately to avoid a Wxcell?

Requires clearance fromATC to get a quickdeviation

Does radar or ATCindicate close t-stormcells/severe weather(wind shear); WSI;

Request (before takeoff) detour forimmediately after takeoff
7.8 Appendix H: Individual Cognitive Demands Tables

H-13

onboard radar;stormscope

Will wind shear be aproblem?

There is no directindicator of windshearin any Wx forecasts orindicators on the aircraft(GPWS provides someinformation)

Wind direction, speed;sudden or dramaticchanges in these values;convective weathersystems in the area (tstorms) from FMS,looking for ominousclouds, feel, andwindsock

Watch wind indicationsvery closely for suddenchanges or shifts (FMSprovides crosswindcomponent); use splitcontrol procedures

Cruise How should I avoid any Wx

cells?

Hard to know the rangeof the Wx (lateral andvertical) and the extentof the Wx around themain cell; squall linesmay require a largedetour; radar cannotalways penetrate Wxcells and may notprovide an accurateimage of the extent ofthe cells; hard toestimate cloud heights

Radar indications (frompreflight, onboard, andATC); look outside;listen for PIREPS andradio chatter;

Estimate the height ofcloud tops; manuallymanipulate radar toestimate cloud heights;ask FW for tops; 10-20miles lateral clearance,2000ft verticalclearance; look for“steep” gradients on theradar, avoid these areas.

How can I make the ride ascomfortable as possible forthe passengers? (i.e., avoidturbulence)

You cannot seeturbulence and there areno direct indicators ofturbulence in theweather forecasts orfrom indicators in theaircraft;AIRMETs/SIGMETsand PIREPs areunreliable

PIREPS; radio chatter;feel; ATC reports;AIRMETs/SIGMETs(not very reliable)

Request ATC reports;listen to chatter; slow airspeed; deviate aroundclouds/ sources ofturbulence

Is my destination stillviable?

Requires anticipationand projection, plusinterpretation ofPIREPS and radiochatter around the

Radio chatter fordestination; delays andholds at destination;ATC reports; FW reports

Listen to ATIS asap;contact FW; slow down,reroute, hold
7.8 Appendix H: Individual Cognitive Demands Tables

H-14

destination

Do I still have enough fuel tomake the trip? How am Idoing?

Headwind componentmay be different thanforecast; delays at thedestination may requireholding; aircraftperformance is affectedby anti-icing equipmentturned on

FMS; length of delaysand holds at destination;distance to alternate;length of detours toavoid weather; look forchanges in wind andimpact on speed andETA (on FMS) esp.compared toexpectations/planning

Slow down, reroute,hold; change altitude toreduce headwind(especially over 25-30kts)

Descend Do I need to turn on my antiicing equipment? Hard to tell if icing will be a problem METARS at destination; PIREPS; radio chatter No cost for aircraft performance by turning

it on (it’s almost a nondecision)

Is there a possibility ofwindshear that I shouldprepare for?

Windshear is hard topredict/anticipate andthere is no directindicator of it on theaircraft and you cannotsee it

Sudden or dramaticchanges in wind speedand wind direction overtime

Use radar to findconvective weather; ifcrew disagree onseverity, “goconservative route”;watch GPWS

Do I need to briefPassengers for a rough ride?Will there be turbulence?

Not difficult!! Usuallyeasy to do it, beconservative and don’tget caught out

Is the ride gettingbumpy? Will Passengerscomfort be affected?Will destination beaffected? Will schedulebe affected?

Always keep passengersinformed, for comfortand safety. Inform themif more turbulence thanexpected. Also, ifthere’s a need to divert,they may want to helpmake that decision

Approach/Land Is my assigned runway OK? Need to check several

sources of info: brakingreports, runwaycontaminations, airportfacility directory andNOTAMS

runway direction; winddirection/speed; windshear indications; lengthof runway; runwayequipment (ILS);visability and ceilingconditions

Check runway directionand crosswinds. Limitsare 25 knots crosswindcomponent

How is the visibility? Do Ineed an ILS approach?

Visibility and ceilingcan be dynamic andchangeable and are hard

runway equipment (ILS);visibility and ceiling(MVFR/IFR

Check airport facilitiesfor landing; check ATC

for current vis and
7.8 Appendix H: Individual Cognitive Demands Tables

H-15 H-15

to forecast andambiguous to report

conditions??) ceiling

Is the runway contaminated?Is braking going to be aproblem? (wet runway)

Need to checkNOTAMS, listen for orrequest PIREPS

PIREPS; braking reports;ATC; precipitation

Check ATC for brakingreports, and runwaycondition

Is Wx changing/changeableat the destination field?

Hard toanticipate/project whatthe destination Wx is,esp if info is sparse

ATIS; ATC; PIREPS;radio chatter

Listen to ATIS andfigure out how manyamendments have beenmade in previous hour

Will windshear be aproblem? (crosswind)

No indicators, hard topredict, cannot see it

PIREPS; ATC reports;radio chatter; FMSreadouts

Use radar to findconvective weather; ifcrew disagree onseverity, “goconservative route”;watch GPWS; Checkrunway direction andcrosswinds. Limits are25 knots crosswindcomponent, 10-15ktdiscrepancy in speedchanges, and rapid shiftsover ~ 20 degrees

Do I need to think aboutanything specific for apossible missed approach?

Need to judge howlikely a missedapproach could bebased on the weatherand other factors (e.g.something on runway)

Severe Wx indicators? Listen to radio chatter

and PIREPS for otherrecent missedapproaches

Taxi/Deplane How are my brakes? No indicator Braking reports;

How are the groundconditions?

Hard to tell by looking,hard to see black ice, ordepth of standing water.

PIREPS; braking reports;ATC reports

Listen to PIREPS

Do I need to make a quickturnaround?

aircraft must be kept“alive” while thepassengers deplane, andreduce wing icing etc.

Keep the engines alivewhile on the ground, sothey don’t get cold
7.8 Appendix H: Individual Cognitive Demands Tables

H-16

Concurrent Do I need to brief the

passengers?

Easy Is the schedule in

jeopardy? Will the Wximpact comfort? Will Ihave to divert to a newdestination/alternate

Ask for input ifschedule, destination, orcomfort may beaffected, if their safetymay be affected

What’s the buzz from othertraffic?

What do others report? Radio chatter; PIREPS Listen to radio chatter

on your route and atyour destination

What’s the destination like,will I have to hold?

How busy is thedestination, is it likelythat I will have to hold?

PIREPS; ATC reports;ATIS; reports of holdsand/or delays; call FW

Get forecasts from WSI;radar (onboard andWSI); listen to chatter atdestination; listen forhold/delays atdestination
7.8 Appendix H: Individual Cognitive Demands Tables

Subject 6 - Cognitive Demand Table

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning Determine best route to

destination

Must continually getupdated info andpossibly revise planenroute

RadarDeparture anddestination WxCloud tops, winds aloft,SIGMETs, turbulence

He builds his bigpicture of Wx startingwith radar and satelliteof USHe looks at movementsof fronts and locationsof t-stormsHe then reads hard textreadouts to check Wx atdeparture anddestination and 3-5stations “behind” theWx to see what’smoving in his direction

Take-off Should I delay takeoff? Have to plan around

hold over times

Winter: freezing rain,snowSummer: squall lines, tstormsWind sheer, crosswindsRunway condition

Wait awhile before takeoff, because wind sheertends to move quicklyDetermine if immediatereturn is possibleBrief passengers aboutturbulence

Take-off Which direction do I take

off from?

Cross windsWind sheer

Climb Avoiding turbulence Weather is dynamic

Corridors may besaturated with other AC

Reports of t-stormsReports of icing

Builds big picture (inhis head) of Wx aheadChecks for proximity ofconvective activity andhow it is movingUses TCAS todetermine how otherAC are moving aroundChecks PirepsAsk ATC for rerouteLooks for leveragepoints if he has toreturn soon
7.8 Appendix H: Individual Cognitive Demands Tables

H-18

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Cruise Avoiding turbulence/storms Usually it’s not

difficult, he can fly overitSometimes there is apressurization problemso he has to fly at alower altitude and goaround the Wx

Cloud linesMountain “waves”Radar—turbulenceindicatorPirepsFuel stateGreen clouds indicatehail

Fly over the WxCall tower for change inaltitude or vectorDon’t fly under cloud“anvil”

Thinking aboutapproach/destinationchange if weather is bad

Descent Should I put on the antiicing or wait for the

automatic de-icing to start?

He plans for the worseand starts the anti-icingif he thinks icing maybe a problem

Convective activity? location of activity,

width, height ofconvective activity?Comma shapedconvective activityReports from other pilots(from AFIS or overradio)

Checks radar forconvective activityMay fly aroundconvective activityAsk Center where otherpilots have gonethrough in last 30minutes
7.8 Appendix H: Individual Cognitive Demands Tables

H-19

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Landing Should I hold? Divert? Or

Land?

Wind sheerDown burstsRunway conditionsPotential forhydroplaning if excesswater on runway

Wind direction onrunwayWet runwayGusty windsDew point spread

He carries an extra hourof fuelHe won’t land on lessthan 5000ft runwayHe finds out about theavailability of deicingequipment and hangerspace during planningWaits until someonetries to landLands “hard” to avoidhydroplaning on wetrunwayMay hold at higheraltitude to wait outturbulence

Taxi/deplane Should I put the plane in a

hanger?

IcingHanger spaceDeparture time

Other Should I brief Px about wx

conditions
7.8 Appendix H: Individual Cognitive Demands Tables

H-20

Subject 7 - Cognitive Demand Table

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions 

Planning Where am I going?

(domestic or international)

International flights areusually long and havedifferent wx patternsDifficult to anticipateinternational wx; duringlong flights the wxchanges more frequentlyLanguage barrier andairspace laws are specificto certain countries; maynot be able to rerouteover or land in certainareas

Find out what tops ofclouds are

Call flight control(ATC) to see how otheraircraft are doing andwhat they are doing(jammed up)Try to get live wx radarbefore takeoffLook at cloud top reportFill aircraft full of fuelon international flights –can come in handy forunexpected events

What is the wx doing atdestination, departure, andalong the route?

Air traffic congestion (inand out of East Coast)

How much moisture isout there?Tight groups of t-stormsindicating pop-up tstorms, fast growing –may be hailWinds and especiallygusts of wind atdeparture and destin.

If domestic, getsbriefing over the phonefrom FAA on conditionsat departure anddestination, NOTAMS,and forecast.If wx looks bad callanother wx service toget more information orcall FAA the nightbeforeUse Universal Wx inHouston or Global Wxin Seattle (realforecasters)FAA relies on NWS forgeneral forecast info(private pilot service

provides more specific
7.8 Appendix H: Individual Cognitive Demands Tables

H-21

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions 

info)May speak to ameteorologist from (cando this on the ground orenroute)Puts radar on while stillon ground and points hisplane in direction oftakeoff so can seewhat’s out in front

Takeoff Where do I go if there is a

problem on takeoff (canyou get back?)

TurbulenceContaminated runwayWindshear concerns(how will aircraftperform)T-storms close to fieldmake it difficult to turnback

Direction and speed ofwindsTemperature (if lowtemps potential foricing). Temp-DewpointspreadTurbulence reportsWet runway

Takeoff with anti-ice onTurn continuous ignitionon so engine won’tflame out if fly througha small t-storm

What should I set takeoffpower setting to?

Gusty windsAircraft performanceconstraints (how will itaffect fuel load?)

Icing, winds

Is the runway toocontaminated?

Some part of the runwaymay be more slipperythan others, tough to seethis

Are the runways cleanand dry?What part of the runwayis slippery?

Look out window

Will the de-icing fluid lastlong enough? Do I needanti-ice on?

Delays caused by ice andsnow and sleet, build upon wingsDifferent fluids last
7.8 Appendix H: Individual Cognitive Demands Tables

H-22

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions 

longer/shorter.

Climb Can I go around or over tstorms or will I have to

penetrate?

Frontal systems movefast in summerUsually lightningassociated with stormsMay have delays (unableto go around)Embedded t-storms,can’t see themHail – very dangerousWarm air drafts above tstorms can affect aircraftperformance

How high are t-stormsColor of clouds (darkdark, half dark, white)Flat, anvil shapes orpeaked, growing, matureClouds lined-up in a rowor are spottyVisual assessment ofcolor on bottom of cloudRadar images:symmetrical cells are notas dangerous as “hooks”

Uses radar to spot them,Run radar down so thathe knows where bottomof beam is, then anglesup.Talks to ATCIf t-storm is at 40,000,then give 40 milesdistance; if at 30,000,then give 30 milesdistance

Where are the strongestwinds and will they be aproblem?

When you penetrate jetstream lots of turbulence

Strongest winds in jetstream are during thewinter monthsJet is somewhere near30,000, and around 120knots of wind

Listens to other aircraftwho have gone throughjet streamTalks to enroute ATCabout jet stream activity

Is there icing? Build up of ice reduces

lift on aircraft

Rime ice occurs between0 and minus 10 degreesC.

Turn Anti-ice on beforetakeoff

Cruise Can I stay at this altitude? Too much fuel or extra

weight will affect abilityto move to higheraltitude

Weight of aircraft makesa differenceWhat are the winds,steady or changing? Arethey favorable?Location of frontalcrossings and t-storms

Checks with ATC to seewhat other pilots areexperiencing

What kind of land/water Crossing mountains can High wispy clouds with
7.8 Appendix H: Individual Cognitive Demands Tables

H-23

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions 

will I be crossing? be dangerous

Airport in the valley maybe reporting little windand not have any ideawhat winds are like overthe mountain tops – maygive false impression

somewhat circular rolls(linticular clouds)indicateswinds/instability

Do I need to stop torefuel?

International flights overocean

High winds and tempaffect aircraftperformance

Find best route for fuelconsumptionFill up on internationalflights, never knowwhen you may have tohold or reroute

Do I need to slow down tofit ATC’s guidance?

Difficult to time when toget to a certain lat/long tofit into ATC flow

High winds and tempswill affect speed

For international flights,what wx information do Ineed?

Communication withground can be difficult,may not be willing toprovide pilot withnecessary info; bad radaror hostile country.May need to go way outof the way because of illcommunication, remoteairports

Middle East, EasternBlock, and SouthAmerica are difficultplaces to fly into

Use own Flight Service 

Descent Do I have to fly through a

storm system? How do Ido this?

Turbulence associatedwith stormsWindshear anddownbursts in summertime

Ceiling and visibility,tempsLow level fronts –produce turbulence

Listening to destinationairport wx informationListening to what otherplanes have done thathave been flying in thearea from the enrouteATC
7.8 Appendix H: Individual Cognitive Demands Tables

H-24

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions 

Listen to ATIS for windinformationHave one pilot on shortrange radar and one onlong rangeMay hold to wait for wxto pass

Is there freezing rain orice?

Weighs down aircraft,can’t get lift whenneededRunways are slippery,won’t be able to stop

Runway breaking actionreports

Use anti-ice equipmentbefore get into icingconditions

Do I have enough fuel?What is alternate destin.?

Difficult overseas – not alot of alternatives insome remote areas

What are the passengersdoing? – being droppedoff and leaving again, orare they staying on theground for a while

Planning for possiblecontingencies

How many runways are atdestination?

Fuel considerations maybe impacted if they haveto hold or divertCan only land in onedirection in one runway– consider crosswinds

What direction thewinds are coming from,are they aligned withrunway

Check ahead in advancefor number of runways,especially if landing at anew airport

Approach/Landing What are the minimums

and can we land in them?

In the winter can getblowing snow, ice, gustyWith strong crosswindsthe runway may not beexactly where you expectit to be.

Where will plane breakoutTurbulence at field

Listens to pilots aheadof himCheck to see if anyoneelse has landed at thatairport today

Will I have to gosomewhere else after Iland?

May be on the groundlong time, and if icing willhave to de-ice

Might not land if wx isbad
7.8 Appendix H: Individual Cognitive Demands Tables

H-25

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions 

Gusty winds, lots ofturbulence

Will I have to changedestination at last minute?

At this point in themission, may already havebeen committed to adestination and notplanned to divert

Forecast is differentthan what current wxis: winds, icing,turbulence, t-stormactivity that quicklydevelop

Get update on currentconditions atdestination, contact FSS

Are winds too heavy toland in?

Passengers start to getconcerned, causes morestressMay get better, may getworseCan’t always see thewinds coming

Getting banged arounda lot, heavy turbulence

Start to talk about otheroptions with 1st pilot

Other Do I have to fly to an

airfield with no tower?

Flying without vitalinformation about thedestination

Do not want to do this inthe evening when thereis low visibility and lowceiling – affectsdecisions throughoutflight

Don’t know how to flyaround or near certain wxsystems (e.g., hurricanes,tornados)

Little knowledge abouthow the wx system workscreates more uncertainty,don’t know what to expect

Call experiencedmeteorologist to find outabout what particularsystem is doing andwhat the best route is togo around it
7.8 Appendix H: Individual Cognitive Demands Tables

H-26

Subject 8 - Cognitive Demand Table

Phases of Flight Cognitive Demand Why Difficult? Cues & Factors Strategies/Actions Planning What’s the general

weather picture, what’sthe situation? What aremy “problems of theday”?

The weather picture isbased on what aircraft he’sflying, therefore has tointerpret the picture withrespect to the aircraft’scapabilities; DUATSformat is very hard tointerpret (he likespictures/graphics)

WSI Severe Wxbriefings; Wx channelsynopses, outlooks andgraphics; route relevantinfo; convectiveAIRMETs/SIGMETs;Where are the pressureareas, where are thefronts, which way arethey moving? Wherewill they be when Iwant to fly?

Call FSS for NOTAMS;if the weather is cruddy,carry extra fuel.

Who’s going to fly thisleg?

May not know experiencelevel of copilot and theextent of the Wx problems

If it looks like a cruddyday, he will want to flythe leg

How much do I need tobrief the copilot?

Doesn’t always knowcapabilities/experience ofcopilot

Level of copilotexperience; degree ofconcern for weatherbased on airmets,sigmets, icing andturbulence reports

He briefs the copilot ifhe is not familiar withhim/her - otherwisedoesn't, necessarily.

How will I fly thisroute based on theweather?

Not an issue of whether togo, but how to do it; wx isa challenge to him, not aproblem; has to be safe.

Severe weather; feasibledestination; aircrafttype; copilot

Because his aircraft(Citation X) is highpowered and has AFIS,he can fly throughanything; he just has tofind a good route and aplace to land and he canbe much more flexiblethan the airlines.

What kind of approachwill I need, ILS orvisual?

Hard to know what theconditions will be onarrival, but need to know ifILS facilities are availableat the destination

Current and forecastceiling/vis atdestination; AirportFacilities Directory forlanding facilities

Check forecasts andfacility directory

Taxi/Take-off Will icing be a

concern?

May be an issue forrunway contamination,
7.8 Appendix H: Individual Cognitive Demands Tables

H-27

depending on the type ofairport

Will Wx affect takeoff? Need to find t-storm cells

in take-off path, hard to doespecially if the cells areembedded

Storm cells; ceiling;turbulence reports

What will I do if I getan engine out?

Wx impacts the handlingcharacteristics of theaircraft differently fordifferent conditions

Crosswinds; turbulence;storm cells; wind-shearreports

Anticipate the problemand generate potentialcontingency plans

Cruise Will I be able to

maneuver around theweather?

May be embedded cells,hard to tell severity bylooking and sometimesfrom radar

Use radar and eyes toweave through stormcells (Stormscope,radar) and listen toPIREPS

Listens to PIREPS butdoesn’t trust them toomuch, just use as aguide; request info fromATC; choose route thatavoids major fronts, ort-storm activity (flybehind a moving front,upwind of it)

Will I need anti-ice enroute?

A/c self-deices at speed,but hard to tell how severethe icing conditions are andif you really need to switchdeicing equipment on(aircraft performance goesdown when deicing is on,uses more fuel)

If aircraft is flying fast,the airframe warms upand automatically deices, depending on howsevere the icing is (tempand moisture)

Approach/Landing Will icing on approach

be an issue?

Forecasts for icing andPIREPS for icing areunreliable; can’t always seeicing conditions

As aircraft slows down,it cools down and ismore susceptible toicing

Request icing reportsfrom tower

Will I need to shoot anILS approach? Onwhich runway?

Need to assess ceiling andvisibility conditions, andintegrate with knowledgeof runway facilities, as wellas personal capabilities

Will I need to carry Wind-shear hard to Call tower for wind Inform passengers of
7.8 Appendix H: Individual Cognitive Demands Tables

H-28

extra speed for landingto avoid wind-shearproblems?

recognize or see reports; listen for

PIREPS

bumpy ride if turbulenceis expected; carry 20ktsover normal approachspeed

If I need to go around,which way will I go?

Need to assess where theweather is

Radar, ATC reports;Where are t-storm cells,are they moving?Which way? How fast?;Which runway am Iusing?

Concurrent How can I make this

flight the mostcomfortable for thepassengers?

Need to be aware ofturbulence and its impacton passengers; can’t seeturbulence or alwaysanticipate it

PIREPS; tower reports Request turbulence

reports from tower; flyat a slower speed inturbulence
7.9 Appendix I: Summary Cognitive Demands Tables

I-1

Information Requirements and Goals Table

Phase of Flight Cognitive Demand Information Requirements

1

Goals2

Planning How much extra fuel do I need with

regard to wx?

Knowing if there are re-fueling facilities atdestination – (not specified)Airmets – (DUATS, ATIS, FSS, ATC)Sigmets – (DUATS, ATIS, FSS, ATC)Pilot reports – (ATC, HIWAS frequency,DUATS)Aircraft performance – (own experiencewith flying a particular aircraft and withrestrictions in the Aircraft Flight Manual)

EfficiencySafetyPassenger Comfort

What’s the general wx picture? Will itbe affecting my flying in the next fewdays?

International and Nationwide forecast –(Internet, Wx Channel, NWS, FSS, Privatewx provider, FAA phone brief, DUATS)

SafetyPassenger ComfortEfficiency

What are the wx problems for this flight– departure?(what time do I leave, go/no go?)

Forecast Wx– (Internet, Wx Channel,NWS, FSS, Private wx provider, FAAphone brief, DUATS)Current Wx – (Nexrad radar on internetand TV, FSS, Private wx provider)

SafetyPassenger ComfortEfficiency

What are the wx problems for this flight– en route?

Airmets – (DUATS, ATIS, FSS, ATC)Sigmets– (DUATS, ATIS, FSS, ATC)Turbulence – (DUATS, ATC, FSS,onboard radar)

SafetyPassenger ComfortEfficiency

1 Information Requirements are pieces of information a pilot uses to build a “big picture” of the weather, and are related cognitive dema

phase of flight. Pilots use various sources to retrieve weather information. The sources that were directly linked to information require

identified in parentheses, and sources that were not directly linked to an information requirement are identified as “not specified.”

2 Mission goals are prioritized and sometimes compromised during a mission. The purpose of this column was to prioritize general mis

passenger comfort, and efficiency) based on order of their importance for each cognitive demand specified in the table.
7.9 Appendix I: Summary Cognitive Demands Tables

I-2

What are the wx problems for this flight– en route? (cont)

Icing reports – (ATC, DUATS, FSS)Pilot Reports – (ATC, HIWAS frequency,DUATS)Height of cloud tops – (not specified)Winds aloft – (Wx channel, DUATS,internet, FSS, ATC, ATIS)Fronts – (Wx channel, private wx provider)Pressure cell movement and development –(Internet, DUATS)Topography – (Maps, own experience,look out window)Jetways/Airways – (Flight Charts)Current Conditions – (Nexrad radar fromTV and Internet, FSS, Private wx provider)Forecast Conditions – (Internet, WxChannel, NWS, FSS, Private wx provider,FAA phone brief, DUATS)

How much confidence do I have in theother pilot?

Experience flying with other pilot Safety

EfficiencyPassenger Comfort

Taxi/Takeoff Can I taxi/takeoff in this wx?

(crosswinds, windshear, runwayconditions, icing; go or no-go)

Wind and crosswind direction, speed -

(ATIS)Takeoff limitations (aircraft manual)METARS – (not specified)Temperature/Dewpoint spread – (notspecified)

SafetyPassenger ComfortEfficiency
7.9 Appendix I: Summary Cognitive Demands Tables

I-3

Precipitation – (onboard radar)Visibility – (ATIS, look out the window)Windshear – (not specified)Pilot Reports – (not specified)NOTAMS – (not specified)Runway contamination, braking actionreports– (ATIS, look out the window)Icing – (visually inspect plane)

Do I need to consider a rejected takeoff? Temperature - (not specified)

Humidity - (not specified)Pressure – (not specified)Length of runway and altitude – (notspecified)Wind speed and direction - (ATIS)Runway contamination - (ATIS, look outthe window)

SafetyPassenger ComfortEfficiency

Do I need to delay takeoff? Severity reports - (not specified)

METARS - (not specified)Current wx - (not specified)Forecast wx - (not specified)Ceiling - (ATIS, look out the window)Visibility - (ATIS, look out the window)

SafetyEfficiencyPassenger Comfort
7.9 Appendix I: Summary Cognitive Demands Tables

I-4

Crosswinds - (ATIS)Windshear – (ATC)Runway contamination -(ATIS, look out the window)Icing - (visually inspect plane)Lightning – (not specified)

Do I need to consider post- takeoffalternatives?(e.g., engine failure, wx cell blockingroute, can’t get back)

Severity reports - (not specified)METARS - (not specified)Ceiling - (ATIS, look out the window)Visibility - (ATIS, look out the window)Crosswinds - (ATIS)Windshear – (ATC)Runway contamination -(ATIS, look out the window)Icing - (visually inspect plane)

SafetyEfficiencyPassenger Comfort

Climb Anticipating icing problems Temperature, forecast and current – (ATC)

Pilot Reports – (not specified)Precipitation – (onboard radar)Icing – (look out the window)Cloud location – (onboard radar)

SafetyEfficiencyPassenger Comfort

Anticipating wind-related problems(turbulence, windshear, crosswinds)

Forecasted wx – (FSS)Winds aloft, speed and direction – (look

SafetyPassenger ComfortEfficiency
7.9 Appendix I: Summary Cognitive Demands Tables

I-5

out the window, onboard sensors FMS)Jet stream level – (enroute ATC)Turbulence – (enroute ATC, PIREPS)

Determine how to avoid wx obstacles(t-storms, hail, lightning)

Locating t-storms activity – (onboardradar, enroute ATC, PIREPS)Pilot reports – (not specified)Cloud color and alignment – (look out thewindow)

SafetyPassenger ComfortEfficiency

Cruise Is my destination still viable? Ceiling – (FSS, DUATS, ATC)

Visibility at Destination – (ATC, FSS,DUATS, ATIS)T-storm activity – (Call ATC, look outwindow, onboard radar)Surface winds – (ATC)Turbulence – (DUATS, ATC, FSS)

SafetyEfficiencyPassenger Comfort

How am I doing in general?(Do I need to refuel? Am I onschedule?)

Wind speed and direction – (ATC, FMS)Fuel considerations – (FMS)ETA - (FMS)Upper air Temperature – (FMS)Delay information – (ATC)

EfficiencySafetyPassenger Comfort

Provide passengers with comfortableride

Airmets – (Not specified)Sigmets – (Not specified)Turbulence – (DUATS, ATC, FSS)Topography – (personal knowledge ofregion, out the window)

Passenger ComfortSafetyEfficiency
7.9 Appendix I: Summary Cognitive Demands Tables

I-6

Determining how to avoid wx obstacles(t-storms, hail, lightning)

Clouds tops – (onboard radar, FW)T-storm activity – (onboard radar, ATC)Cloud color – (look out the window)

SafetyEfficiencyPassenger Comfort

Anticipating icing problem Moisture – (look out window, onboard

radar)Temperature – (FMS, ATIS)Icing – (ATC, DUATS, FSS)

SafetyPassenger ComfortEfficiency

Descent/Approach/Landing/Taxi

Dealing with wx obstacles in descent Wx avoidance – (ATC, look out window,

listen to radio frequencies, onboard radar)

SafetyPassenger ComfortEfficiency

Anticipating Wx: Icing, windshear,turbulence, runway conditions,crosswinds, visibility

Icing – (look out the window, ATIS, ATC)Turbulence – (not specified)Cross winds/windshear – (radio chatter,ATC, PIREPS, GPWS)Visibility – (ATC, FBO)

SafetyPassenger ComfortEfficiency

What kind of approach: ILS or visual?Which runway?

Select runway – (ATC, ATIS)Winds/Windshear – (look out the window,ATIS, radio chatter, PIREPS, GPWS,ATC)NOTAMS – (not specified)Visibility – (ATC, FBO)Ceiling – (ATC)TemperatureRunway length restrictions – (ATC)

SafetyPassenger ComfortEfficiency

What kind of landing – hard or gentle? Wind/windshear – (radio chatter, PIREPS,

GPWS, ATC, ATIS, window)

SafetyPassenger Comfort
7.9 Appendix I: Summary Cognitive Demands Tables

I-7I-7

Breaking action reports – (ATC)Runway conditions – (ATC, look out thewindow)

Efficiency

Do I want to hold? Should I divert toalternate destination?

Ceiling – (ATC)Runway conditions – (ATC, look out thewindow)Visibility – (ATC, FBO)T-storms – (ATC, look out the window,listen to radio, onboard radar)Temperature – ( ATIS)Wind – (ATIS, look out the window)

EfficiencySafetyPassenger Comfort

Will this aircraft have quick turnaroundfor next trip?

Wx avoidance – (ATC, look out thewindow, listen to radio, onboard radar)Icing – (look out window, ATIS, ATC)

EfficiencySafetyPassenger Comfort

Anticipate missed approach, go around Runway conditions – (ATC, look out the

window)Windshear – (radio chatter, PIREPS,GPWS, ATC)Select runway – (ATC, ATIS)Visibility – (ATC, FBO)Ceiling – (ATC)T-storm avoidance – (ATC, look outwindow, listen to radio, onboard radar)

SafetyEfficiencyPassenger Comfort
7.9 Appendix I: Summary Cognitive Demands Tables

I-8

Cognitive Demands and Indicators of Expertise

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice EPlanning How much extra fuel do

I need with regard towx?

Winds aloft are difficult topredictThere are trade-offs by carryingextra fuelWx conditions at destination aredifficult to predictWx conditions en route maycause problems (divert, reroute)for fueling considerations

Reallocate “payload”/passengersCarry extra fuel (up to max)Don’t carry too much fuelCheck destination fordelays/holdingRoute selection/flight plandetermines fuel load

May carraltitude aMay not impacts a

What’s the general wxpicture? Will it beaffecting my flying inthe next few days?

Depends on aircraftDepends on reliability offorecast (difficult to predict, wxis dynamic)Is it a multi-day, multi-leg trip?

Watch the wx – TV, internet, etc.

Form a “big picture”“Forecasting” (generateexpectancies)Call FSS for briefing the nightbefore

MisinterpDon’t forlook ahea

Planning (cont) What are the wx

problems for this flight– departure?(What time do I leave,go/no go?)

Have to plan ahead:Schedule departure forpassengers, what are schedulingconstraintsDifficult to predict wx

Change departure time – move upor backChange passengerschedule/itinerary

Feel too schedule“Can do” 

What are the wxproblems for this flight– en route?(What route?)

Limited shelf-life of forecastdataLimited accuracy of forecastdata

Get briefingsLook at internet, TVLook at graphics:-Radar

Misjudge Rely too informati
7.9 Appendix I: Summary Cognitive Demands Tables

I-10 I-9I-9

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

Planning is based on forecast(e.g., fuel, route)Restrictions on possible routesand altitudes (e.g.,

altitude/direction, special useairspace)Given flight plan is notoptimized for wx avoidanceWx is dynamicAircraft limitations (equipment,performance)

-Satellite-SynopsisPlan route behind (upwind) offronts/severe wxTalk to experienced meteorologistat the FSS or personal wxproviderAccount for season and region(based on experience)Consider carrying extra fuel ascontingencyShare SA/brief non-flying pilot

Not plan contingen

What are the wxproblems for this flight– destination?(Is it viable, do Iconsider alternatives,which?)

Limited shelf-life of forecastdataLimited accuracy of forecastdataPlanning is based on forecast(e.g., fuel, route)Destination may have limitedfacilities and equipment (ILS,icing, snow removal, VFR field,uncontrolled, foreign language,etc.) – more uncertaintyDifficult to predict wx – relatedtraffic and delaysWx is dynamicAircraft limitations (equipmentand performance)

Get briefingsLook at internet, TVLook at graphics:-Radar-Satellite-SynopsisTalk to experience meteorologistat FSS or wx providerCall destination (e.g., FBO)Account for seasonal and regionalwx (based on experience)Consider carrying extra fuel ascontingencyShare situation assessment/brief

Misjudge Rely too informatiMay not contingenMore conMay not Interpretaexperiencnot have May not resources
7.9 Appendix I: Summary Cognitive Demands Tables

I-10

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

If too close to minimums

non-flying pilotCheck for wx upwind ofdestination (severity, rate anddirection or movement)Check Airport Facility DirectoryCompare forecast to ownexpectanciesIf too close to minimums, checkATIS more frequently

How much confidencedo I have in the otherpilot?

Don’t know how muchexperience or how capableother pilot is to deal with wxrelated to flying issues (In termsof judgment, decision makingskills, flying skills, andexposure to difficult wxsituations)

Discuss situation between bothpilots – share assessment ofsituation and agree on course ofactionIf in doubt, be conservativeShare duties – “split control”Coordinate duties and control

“Can do” May not /share SA

Can I taxi/takeoff in thiswx? (crosswinds,windshear, runwayconditions, icing) Go orno go?

Wx hazards are difficult topredict and detectWx poses hazards to theflight/takeoffTaking off out of uncontrolledairport – lack of facilities andwx support data

Use radar (on board)Check ATISCheck with ATC (radar andPIREPS)Request a different runwayDelay takeoffExpedite takeoffVisually inspect aircraft andrunway conditions

UncomfoprocedurcrosswinMay not runway ihas anti-iperformaMay damwhile taxFail to gespecific w
7.9 Appendix I: Summary Cognitive Demands Tables

I-11

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

Put on anti-iceUse airport de-icing facilitiesTaxi out with flaps not fullyextended (reduces damage fromslush, snow, and ice)Increase takeoff power setting

Fail to recapabilitiDon’t rechazards

Do I need to consider arejected takeoff?

Wx impacts aircraftperformance, which can changeamount of decision timeavailableNeed to judge aircraftperformance factors that are wxdependentNo direct indication of how wxwill affect the aircraft

Awareness of the wx factorsimpacting aircraft performance

No awareaircraft p

Do I need to delaytakeoff?

Passenger schedules andconstraints add pressure tomake the mission happenJudging when the wx will be aproblem is difficultCosts:-Lose takeoff slot-Need to de-ice again

Brief passengersGather current and forecast wxfrom ATC and ATISUse on-board radar to judgemovement of wx cells (rate anddirection)Use knowledge of local wxconditions & terrain (experience)

Misjudge wxDo not pchanges 

Do I need to considerpost- takeoffalternatives?

Likelihood of engine failure ormalfunction increases at thisstage

Determine if immediate action ispossibleBrief passengers

Fail to coflight pla
7.9 Appendix I: Summary Cognitive Demands Tables

I-12

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

(e.g., engine failure, wxcell blocking route,can’t get back)

Aircraft performance isdegraded by wx conditionsHard to judge severity of wx

Consider alternate course ofactionGet prior ATC clearance forimmediate deviation around wxTurn on continuous ignition toavert engine flame out in case oft-storm/lightning

Climb Anticipating icing

problems

No direct indicators of icingUse of anti/de-ice equipmentimpacts aircraft performance

Turn on anti-icing before takeoffAvoid flying through clouds

Don’t reaicing equperforma

Anticipating windrelated problems(turbulence, windshear,crosswinds)

Winds are very dynamic anddifficult to forecastNeed to provide smooth ride forpassengersNo direct warnings ofturbulence

Avoid t-storms, menacing cloudsTalk to ATC and FSS:-PIREPS-Jet stream updateUse TCAS to determine howother aircraft are maneuveringaround wx

Not comcontrol” Fail to uslocate sto

Determine how to avoidwx obstacles (t-storms,hail, lightning)

Wx is very dynamicDifficult to know the magnitudeor intensity of wx obstacleRequires ATC clearance forquick deviationEmbedded t-storms are difficultto detectHail is difficult to detectWarm air drafts above t-stormsaffect aircraft performance

Manipulate radar to determinecloud heightsRequest immediate reroute fromATCAvoid menacing cloudsTalk to ATC and FSS – PIREPSUse TCAS to determine howother aircraft are maneuveringaround wx

May not enoughFail to uslocate t-sheightsFail to anstorms
7.9 Appendix I: Summary Cognitive Demands Tables

I-13

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

Is my destination stillviable?

Lack of real time dataRequires pilot toproject/anticipate wx from staledataRequires pilot toproject/anticipate wx from radiochatter at destinationAt an uncontrolled airport thereis a lack of information, reliableinformation

Listen for delays/holding atdestinationJudge trends in wx at destination– Getting better? Getting worse?Get ATIS just after the hour(latest reports at destination)Slow down, hold, reroute todestinationReroute/divert to alternate (havingassessed wx at alternative)

Not get wrely on o

How am I doing ingeneral?(Do I need to refuel?Am I on schedule?)

The flight plan is based onforecast wx which may beunreliable (especially windsaloft)Delays at destination aredifficult to predictIt is difficult to assess aircraftperformance due to wx(headwinds, de-icingequipment)

Slow down to avoid holdingChange altitude and/or route forbetter windSlow down to conserve fuel –avoid refueling

Not chec-Winds-Fuel-ETA

Provide passengers withcomfortable ride

Can’t see turbulence, no directindicatorsHard to predict turbulenceShelf life of turbulence reportsis unreliable, also unreliable dueto subjective nature (depends onaircraft and pilot perception)Embedded t-storms are difficult

Listen to radio chatter aboutturbulenceAsk ATC for smoother ride(different altitude)Listen to PIREPS of turbulenceFly around sources of turbulenceSlow down/reduce airspeed =

May not comfortMay not May not reliabilitytheir situ
7.9 Appendix I: Summary Cognitive Demands Tables

I-14

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

to detect on radarMay need to trade-off direct,bumpy ride with an indirect,smooth ride that takes you lowon fuel

more comfortable rideBrief passengers about turbulence

Determining how toavoid wx obstacles (tstorms, hail, lightning)Determining how toavoid wx obstacles (tstorms, hail, lightning)

-Wx is very dynamic-Difficult to know themagnitude or intensity ofobstacle-Requires ATC clearance forquick deviation-Embedded t-storms & hail aredifficult to detect-Warm air drafts above t-stormsaffect aircraft performance

-Manipulate radar to determinecloud heights-Request immediate reroute fromATC-Request different cruisingaltitude-Avoid menacing clouds-Talk to ATC and FSS – PIREPS-Use TCAS to see how otheraircraft maneuver around wx

May not enoughNot use rt-storms heightsFail to anstorms

Anticipating icingproblem

No direct indicator – requiresinference or visual detection(difficult to see at night or lowvisibility)Autopilot corrects for ice buildup

Look out the window, periodicvisual inspection of wings andbootsTurn on de-icing before icingaccumulatesDisconnect autopilot so it doesn’tcompensate for loss of trim

Use de-icnot work Over-reli

Descent/Approach/Landing/Taxi

Dealing with wxobstacles in descent

Less room to maneuver(approach/landing)

Call ATC for good routes throughwx into airportOne pilot on short range radar,one on long range

Fail to lisnavigatin

Anticipating wx: Icing,windshear, turbulence,runway conditions,crosswinds, visibility

Specific to landing /approachSudden changes, rapiddeviations have more impact –fewer options and less space toplay with, especially altitudeSimilar issues as previous

Fly over airfield (“fly by”)Call ATCListen to ATISActivate de-icingHold until wx passes

May not May not needed
7.9 Appendix I: Summary Cognitive Demands Tables

I-15

Phase of Flight Cognitive Demand Why Difficult? Strategies/Actions Novice E

Listen for PIREPS/chatter

What kind of approach?ILS visual? Whichrunway?

No single source of rwy/wxinformationCeiling, visibility, and windhard to report accurately(especially if changeable)

Check airport facilitiesTalk to ATC for ceiling, visibility

Lack conin ability minimum

What kind of landing –hard or gentle?

Need to anticipate wind andrunway conditions

If gusty/windshear or excesswater carry extra speed, land alittle harder, brief passengers, getrunway reports, wind reports

Don’t adj

Do I want to hold?Should I divert toalternate destination?

Pressure to land – fuel,passengers,Conditions may be marginal,info may be unclear,ambiguous, or unavailable

Get updated info from ATC, FSSDiscuss options with pilot-notflyingListen to PIREPS/chatterBrief passengersDo a fly by of runwayKeep alternate airport in mindReschedule ground transportationfor passengers

ConfidenminimumMay not 

Will this aircraft havequick turnaround fornext trip?

Need to keep plane “alive” ->Ensure airport has facilities forquick turnaround

Keep engine warm & aircraftdeicedMay not land if can’t turn around

May not 

Anticipate missedapproach, go around

Requires pilot to think aheadand anticipate how wx mayimpact approach/landing/ missed approach

Go around to avoid t-storms orother obstaclesListen to radio chatter andPIREPS for other approaches

Doesn’t l
7.10 Appendix J: Simulation Interview Summary

J-1

Decision SME 4 SME 5 SME 6 SME 7 SME 8

Planning Phase

Route planning What is the best

route?

Where am I

going? Domestic

or international?

Will icing be a

problem?

Will icing be a

problem?

Should I turn

anti-ice on

during climb and

descent? When

should I initiate

anti-icing?

Aircraft

performance

considerations.

Is there going to

be icing?

Will icing be a

problem?

Should I carry

extra fuel?

Should fuel load

be heavy or

light?

Will I need to

carry extra fuel?

How much fuel

should I carry?

What are the

significant

weather

concerns?

Where are the

wx concerns?

What are major

weather

concerns

(destination,

cruise and

departure)?

Icing?

What are major

wx concerns?

Departure and

destination

Should I call

Flight Service to

speak with

meteorologist?

Which runway

should I use?

Which runway

to takeoff from –

cross winds are

heavy

Should I request

alternate runway

on takeoff –

crosswinds;

Takeoff

concerns

Should I ask

tower about

windshear?

Should I wait for

windshear to

pass? – tends to

move quickly

How do I handle

wx in the

terminal? –

gusty winds,

visibility

Should I takeoff

with anti-ice?

What are my

takeoff

concerns?

Takeoff in

cross winds?

Do I fly into the

wind or battle

the crosswind?

Am I within

crosswinds

limit? Should I

split controls

with co-pilot –

crosswinds.

How am I going

to deal with

gusty winds and

crosswinds?

What are the

cloud tops?

What are the

cloud tops?

What are cloud

tops?

What altitude

should I fly at?

What is the best

altitude to fly at?

Should I request

for higher

altitude to get

above

turbulence?

T-storms How am I going

to avoid t-storm

and turbulence?

How am I going

to avoid tstorms?
7.10 Appendix J: Simulation Interview Summary

J-2

Decision SME 4 SME 5 SME 6 SME 7 SME 8

Turbulence How am I going

to handle

turbulence?

Should I alert

passengers about

turbulence?

How can I make

this flight more

comfortable?

Misc Am I within

personal limits?

How much do I

brief my copilot?

Takeoff

problems

Where do I go if

there is a

problem in

takeoff? Can I

get back?

What do I do if

an engine goes

out?

Talk to ATC? Do I need to call

ATC for flow

control? What

are other planes

doing? Should I

change vector?

Will ATC allow

me to do

deviations I may

want to do

enroute?

Destination Which runway

should I use at

JAX?

Should I make

call ahead to

change meeting

times with

clients? – may

have to vector or

hold

What is going on

in JAX? What

does the

alternate look

like?

Should I call

FSS for

NOTAMS? –

JAX and MCO

Climb/

Cruise

Turbulence How do I handle

turbulence and

convective

activity?

Do I need to

listen to PIREPS

– turbulence

reports

How do I handle

turbulence on

climb out?

Do I need to call

ATC to see what

other aircraft

have been

doing?

How do I handle

clear-air

turbulence?

Altitude Is this a good

altitude?

Should I request

altitude change?

Do I ask ATC

for altitude

change?

What is the best

altitude? Should

I call ATC for

change?

Brief px Do I need to

brief passengers

for rough ride?

Do I brief

passengers/copil

ot – windshear?

Avoid

t-storms

Where are the

thunderclouds?

How to avoid

them?

How should I

deviate around

clouds?

What is the

convective

activity like?

How do I avoid

storms? Are

there embedded

storms?

Can I pick

through

weather?

Contact ATC Do I need to call

ATC to see what

other aircraft

have been

doing?

Should I call

ATC for

rerouting

information?

Should I talk to

ATC for

vector/heading

change? Fly

West?
7.10 Appendix J: Simulation Interview Summary

J-3

Decision SME 4 SME 5 SME 6 SME 7 SME 8

Fuel What is my fuel

state? Do I have

enough for

holding/divert?

How’s my fuel?

Icing Should I turn

anti-ice on early

– no cost in

performance in

cruise and not

much on descent

What are my deicing/anti-icing

capabilities?

Is there icing? Will I need to

anti-ice enroute?

Will there be

icing on

approach?

Speed Should I fly

faster? - Winds

are heavy

Misc When should I

retract the

landing gear –

let water runoff

gear?

Should I turn

continuous

ignition on?

Destination Is the runway

contaminated?

Is JAX plugged

up? What’s

going on for wx

at JAX?

Approach/

Descent

Which runway Which runway

should I choose?

What is runway

length?

What is the best

runway for

landing? Is the

wind direction

aligned with

runway?

Which runway

should I use?

Icing Do I turn anti-ice

on for approach/

descent?

De-ice wings

and engine?

Brief px Should I brief

flight attendant

about

turbulence?

How’s the ride

quality for

passengers?

Should I brief

passengers?

Approach

speed/landing

How do I pad

my approach

speed?

How much

approach speed

should I carry?

How am I going

to handle

landing? – wet

runways/

hydroplaning

How should I

handle the

landing?

What have

other planes

done?

Should I talk to

tower to find out

what other

planes have

done?

What have other

planes done?
7.10 Appendix J: Simulation Interview Summary

J-4

Decision SME 4 SME 5 SME 6 SME 7 SME 8

Visual vs

insturment

Is this an

instrument or

visual approach?

Will I need to

shoot an ILS

approach? Can

field support it?

Check ATIS Should I get

ATIS?

What is ATIS

reporting?

Destination wx What is the wx

like around

destination?

Is my destination

runway

contaminated?

Other Should I hold at

higher altitude to

wait out

turbulence?

Will I need to

change flap

settings?

What’s my

Vref? What’s

my descent rate?

Will I need to

ask for different

approach?
7.11 Appendix K: Critical Decision Incident Tables

K-1

Incident 1 - Subject 2.

SME 2 and his co-pilot were scheduled to fly into Elizabeth Town, NC from

Norfolk, VA on a Cessna 525. They knew before takeoff that Elizabeth

Town was a small, uncontrolled airport (sometimes they can’t even get an

ATIS report from there) with a 500ft minimum. They also knew that the

approach into Elizabeth Town would be non-precision.

The weather out the window was cloudy en route. In fact, at a distance of

about 200 miles out the pilots could not see Elizabeth Town, where on a

clear day Elizabeth Town would have been visible. They listened to the

ATIS reports (a specific radio frequency for every airport) for weather

conditions at Fayetteville, an alternate airport in the Elizabeth Town area,

with the intentions of finding a better place to land. The weather at

Fayetteville reported marginal conditions so they decided to check other

local airports to compare weather reports. Relying on his experience and

understanding of weather and many years of flying, SME 2 also considered

that the weather in Elizabeth Town might not be as bad on the ground as it

looked from the sky, and may even have better weather than the surrounding

airports. The biggest concern for SME 2 was the overcast, cloudy

conditions. Clouds usually have turbulence associated with them and they

wanted to avoid that.

SME 2 continued to listen to ATIS reports from the other surrounding

airports as he approached Elizabeth Town. However, before a decision to

land was made SME 2 performed one more check. He called the FBO on the

ground at Elizabeth Town to get real-time observer perspective. The FBO

reported to SME 2 that he was unable to see across the airport, which

indicated to SME 2 that he needed to find an alternate airport to land at. A

call to the FBO on the ground is preferred over a call to the tower. A call to

the tower usually indicates a strong commitment to land at that specific

airport and if you are just checking the weather, there is no need to call the

tower.

SME 2 called ATC in Fayetteville and arranged to land there as they were

reporting better weather and were the closest alternative. He also arranged

for ground transportation to pick his passengers up at the Fayetteville airport.

Critical cues: small, uncontrolled

airport; non-precision approach; 500

feet minimum ceiling

Cognitive demand: What are the

weather problems at my destination?

Information sources: ATIS, pilot’s

eyes, pilot’s experience with local

weather conditions

Critical cues: cloudy, destination not

visible at 200miles;

Cognitive demands: Is my

destination viable? How will I deal

with potential turbulence during

descent?

Actions: check weather at alternate

destination

Why difficult? Weather is dynamic

and available weather information

often is not current

Actions: continue to monitor ATIS

reports; call FBO at destination

Cues: FBO at destination couldn’t

see across air field; weather at

alternative destination better

Information sources: destination

FBO; ATIS

Actions: arrange to land at alternate

destination; arrange alternate ground

transportation for passengers
7.11 Appendix K: Critical Decision Incident Tables

K-2

Incident 2 - Subject 1.

SME 1 was scheduled to fly into the St. Simons, Georgia airfield, a small

island off the coast of Georgia, out of Norfolk, VA. SME 1 waited until the

day of the mission to make his final decision about flying into St. Simons

because of a hurricane watch in the area. He thought that conditions might

change overnight. He knew that one trip in particular, out of the three he had

to make to St. Simons that day, was going to be difficult because of the

timing and proximity of the hurricane. He used a one-page FAA briefing and

the internet to assimilate his forecasted weather data.

This was the first trip into St. Simons but was confident with his knowledge

of local weather conditions. SME 1 also had experience flying into other

airfields (Hilton Head and Brunswick) that are similar to St. Simons, and in

similar weather conditions. SME 1 knew that the AWOS at St. Simons was

uncertified in handling weather reports. SME 1 also had alternative airports

in mind (Savannah, Brunswick) before leaving.

Before taking off, SME 1 set more restrictive limits on wind speed and

direction than he usually does. If winds surpassed the limits SME 1 set, he

would immediately reroute to an alternative airport. In fact, in some

airplanes SME 1 would not have even attempted to make this trip. He

contacted the FBO at St. Simons to get a real-time assessment of the

situation on the ground, and received near real-time data from Brunswick (15

minutes old).

On final approach, SME 1 prepared himself by using Hilton Head as a

template for landing on St. Simons. The features were virtually the same: a

small island, the climate was similar, and SME 1 knew it had to be a

precision approach. He landed with no problems.

Cognitive demand: What’s the

general weather picture? What are

the weather problems for this flight?

Critical cues: hurricane watch for

destination

Information sources: FAA briefing;

internet

Information sources: pilot’s

experience with local weather

conditions; AWOS report from

destination

Why difficult? AWOS at destination

known to be uncertified; threatening

weather at destination

Cognitive demand: Is my destination

viable?

Actions: consider alternative

destinations

Cognitive demand: How will I deal

with weather obstacles on descent?

Actions: set more restrictive limits

on permissible wind speed and

direction for landing; contact FBO at

airport near destination airport;

listen for weather updates for other

nearby locations

Cues: wind speed and direction at

destination

Information sources: FBO at airport

near destination

Cognitive demand: What kind of

landing do I need to make?

Why difficult: he’d never landed at

this airport before

Information sources: pilots’

knowledge of other similar airports
7.11 Appendix K: Critical Decision Incident Tables

K-3

Incident 3 - Subject 6.

SME 6 was scheduled to fly to London, Ontario out of St. Paul at 5:00pm in

a Gulf Stream 4. Total flight time is 90 minutes, about 450-500 miles. From

about 4:00 – 4:45, SME 6 gathered pre-flight weather information for

departure and destination airports. He primarily uses KAVOURAS, a

weather service based in St. Paul. The service provides verbal briefs,

including the ability to file a flight plan, it provides a radar picture, and

provides weather for departure city and state. The only problem with this

service is that it doesn’t provide the big picture. After collecting initial

weather reports, SME 6 had a discussion with the 1st officer about the trip in

which the 1st officer voiced his concern and thought they shouldn’t go. SME

6 felt otherwise, and asked the 1st officer to call the ATC tower in London.

The conditions at St. Paul were rainy, foggy, below minimums, and

temperatures were about 3 to 4 degrees Celsius. London was reporting

conditions below minimums but they had improved to 100 ft and 1⁄4 mile

visibility. They were also reporting the cloud bottoms were ragged. SME 6

knew that even though conditions might be below minimums when they

arrived in London, he could at least go look ( FAA Part 91 regulations).

Before they left they called the tower in London once again, which reported

conditions were down again, 200 ft and 1⁄2 mile visibility. They decided to

go anyway and briefed the passengers on the situation. They devised several

alternatives if they couldn’t land in London and agreed that the best

alternative would be to turn around and go back to St. Paul. Detroit could

have been an option but required more logistical planning.

Because the reports out of London were “ragged,” this meant conditions

were up and down all day in SME 6’s mind. From his previous flying

experience, SME 6 knew that minimums at night were different than during

the day. At night pilots can see the amount of light shining up through the

clouds from the landing strip. Obviously, day conditions don’t provide this

cue. SME 6 knew he could always go back to St. Paul if the weather was

bad, but he wanted to take a look anyway.

At about 6:00pm, during cruise, SME 6 checked with FW. They had enough

fuel to play with several options: hold for a while, make a pass, or go back to

St. Paul. On Approach/Descent, SME 6 briefed 1st officer (check

frequencies, radios, minimums check, and initiated call-outs). They

contacted ATC. ATC reported that other planes had landed that evening,

which indicated to SME 6 that he would at least try to land. Minimums were

300 – 400 ft, and 1 mile visibility. SME 6 inferred this from the amount of

light shining up through the clouds.

They were able to land without any problems

Cognitive demands: What’s the

general weather picture? What are

the weather problems at destination?

Why difficult? 1st officer disagreed

with pilot about viability of landing

Information sources: private weather

service; ATC at destination tower

Cues: radar picture; temperature,

ceiling, and visibility at destination

and departure airports

Action: discuss destination weather

with 1st officer; call ATC at

destination tower; plan to fly to

destination and then make decision

about landing

Actions: call ATC at destination

tower (again)

Cues: destination ceiling/visibility

Actions: brief passengers; identify

alternate destination which was to

return to departure airport if unable

to land

Cognitive demand: Can I anticipate

and deal with significant weather at

destination?

Why difficult? Ceiling and visibility

conditions at destination kept

changing throughout the day and

evening

Cues: landing lights shining through

low level clouds

Action: fly to destination air field

and look at lights

Cognitive demands: Is my

destination still viable? How am I

doing on fuel?

Action: contact ATC at destination

tower

Cues: ATC reported other planes

had landed recently

Action: attempt landing

Cues: light shining through low

level clouds

Information sources: ATC at

destination tower; pilot’s eyes
7.11 Appendix K: Critical Decision Incident Tables

K-4

Incident 4 - Subject 5.

SME 5 and his co-pilot were scheduled to fly a group of Asian businesspeople from an airfield in Phoenix, AZ (Grand Canyon) to Morristown, NJ,

in a Gulf Stream 4 at 4 pm (6pm EST). SME 5 was planning to stay over

night in NJ, and then fly back to MN in the afternoon. At about 2 pm (4 pm

EST) SME 5 and co-pilot began planning for the trip to the East Coast.

Conditions on the ground in Phoenix were clear and beautiful. He called the

Wx service before they left and they said there were no problems in the NY

area. Current conditions in NJ at around 2pm were reporting marginal with

some gusty winds.

At about 8:30 EST SME 5 started his approach into Morristown. Prior to

this point the ride had gone smoothly and they didn’t expect any problems

ahead. About 100-150 miles out SME 5 picked up the Morristown ATIS,

which reported winds gusting from the NW at 15-20 knots. They were

approaching from the Southwest and he concluded that runway 5, landing to

the Northeast, was the only choice. The Morristown airport sits down in a

valley surrounded by some terraced hills on the Southwest side and a swamp

area to the East. He knew that there would be a significant crosswind. He

was not thinking about alternatives because the weather was not that bad.

He started to descend to about 20,000 ft when he received notice from ATC

that there was turbulence below 10-15,000. At almost 9pm EST, he could

see Morristown 20 miles out. SME 5 was cleared for a visual approach.

When he got down to 6000 ft the winds were worse there than they were

higher up. He hadn’t expected this. He got down to 4000 feet and it was

quite rough. At about six miles out from the airport they started to get

banged around a lot. The passengers were no longer talking. He had the

aircraft on autopilot and auto throttle, but it was not handling the winds well,

so he turned it off around 2000 ft. This caused a warning horn to sound. The

passengers heard this and were very alarmed.

As he was coming over the hills the winds started getting violent, there were

heavy gusts up and down, rolling and yawing. The normal approach speed

was 130-140 knots, but they were flying 160-170 knots. His yoke control

was going from stop to stop (i.e., he was losing control of the aircraft). They

had to abort the approach. He knew he could get the aircraft to the runway

but he didn’t think he could keep the plane on the runway when they landed.

It was very unusual to have these violent gusts. The plane didn’t feel right.

SME 5 now knew that runway 5 wasn’t going to work so he looked at

runway 30. This was too short and he was too heavy to land there. He

decided to abort the landing.

They went back up and circled for 15-20 minutes so he could calm down and

discuss his options with his 1st officer. All the NY airports were gusty, but

they didn’t all have the hills like Morristown. They checked with

LaGuardia, but they were real busy. At around 9:30pm EST they tried to

land again at Morristown. They had enough fuel to try again and if they

couldn’t land it this time they could still go somewhere else.

He did not get any new information on the winds. No one else had landed at

Morristown since he had tried 15-20 minutes ago (but no one had attempted

it either). The turbulence was much less on the second try and they were

able to land.

Cognitive demands: What’s the

general weather picture? Are there

weather problems at destination?

Why difficult? Weather is dynamic

and available information often is

not current

Cues: gusty winds at destination

Information Source: private weather

service provider

Cognitive demands: How do I deal

with significant weather at

destination (gusty winds)? Which

runway should I use?

Cues: winds speed and direction,

runway direction, topography

surrounding destination airport

Information sources: ATIS, pilot’s

familiarity with runways and

topography at destination airport

Cognitive demands: How do I deal

with significant weather at

destination (gusty winds)? What

kind of approach do I need to make?

Information sources: ATIS, pilot’s

visual & kinesthetic experience

Cues: turbulence reported, cleared

for visual approach, experienced

high winds at lower altitude,

turbulence experienced

Actions: turn off autopilot&/throttle.

Cognitive demand: What kind of

approach do I need to make? What

kind of landing do I need to make?

Why difficult? Unexpectedly high

winds made aircraft control difficult

Cues: yoke control handing difficult

Action: Abort the approach

Cognitive demand: should I try a

different runway?

Cues: High winds, runway too short,

plane too heavy

Cognitive demands: Is my

destination still viable? How am I

doing on fuel?

Action: discuss options with copilot

Information sources: checked with

ATC at alternate airport

Cues: alternate airport very busy,

sufficient fuel to try again

Action: try landing at original

destination (again).
7.11 Appendix K: Critical Decision Incident Tables

K-5

Incident 5 - Subject 5.

SME 5 was waiting on the passengers who were late. His copilot was the

most experienced pilot in the organization. The weather was beginning to

turn nasty with a storm moving rapidly towards the field. They followed the

weather on the internet and television in the crew room, and were getting

updates from the tower for over an hour. The storm was moving in at about

30 knots and would hit the field soon, at least delaying their departure

further.

The passengers arrived and boarded the aircraft. The storm was moving in

fast but current conditions at the field were good. No rain, good visibility,

and the wind was not going to be a problem. They knew they would have to

get up quickly and make a quick turn to avoid the storm. There was still a

question of whether they would get off the ground in time, but ATC

confirmed that they looked OK if they expedited their departure. They used

their on-board radar to check the progress of the storm. They swung the

aircraft 360 degrees on the ground to make sure that the weather was clear

around the storm and that they could get out. The radar picture was helpful,

although ground clutter made the picture a little ambiguous. However, the

tower’s radar picture was clearer, and the tower made the same assessment of

the situation. The speed and intensity of the storm had worried the pilot.

They had watched it move rapidly in their direction, and their departure was

in jeopardy. Thirty miles away a tower had been abandoned due to the

storm, so they knew it was severe. Both the pilot and the 1st officer agreed

that they could make it, and their assessment matched the tower’s

assessment. They knew it was heading right for them because they could see

from the weather reports and tracked its movement, and as they prepared for

departure, they could see the growing, ominous clouds. The pilot was

confident in the judgment of his 1st officer, an experienced colleague. Also,

their assessment had been confirmed by the tower. If either of these other

opinions had differed from the pilot’s, he would have postponed the

departure. If the 1st officer had been less experienced, the pilot may not have

been so confident in trying to beat the storm.

They took off, deviated immediately, and made it clear of the storm. Two

minutes later the storm hit the field (according to reports from the tower at

the field).

Cognitive demands: What’s the

general weather picture? What are

the weather problems for departure?

Information sources: internet, TV

Cues: direction and speed of storm

moving towards departure airport

Cognitive demands: do I need to

delay my take-off? How am I going

to avoid weather obstacles during

climb?

Cues: lack of wind and precipitation

at departure airport; visibility at

departure airport, pilot observations

Action: track approaching storm on

on-board radar; talk to tower to get

their assessment of approaching

storm

Information sources: ATC at

departure tower, on-board radar

Cues: radar picture of approaching

storm, speed and intensity of

approaching storm, local tower

abandoned

Action: discuss options with 1st

officer

Cues: experience of 1st officer;

agreement with destination tower

ATC

Action: take-off quickly before

storm arrives

Action: turn aircraft immediately

after take-off to avoid storm
7.11 Appendix K: Critical Decision Incident Tables

K-6

Incident 6 - Subject 4.

SME 4 had to miss an approach into a small, unsupported airfield near

Traverse City, MI. He could not see the airfield even though the ASOS was

telling him that the airfield was above minimums. SME 4 knew that the

temperature and dew-point spread was close, and that the wind was in the

“wrong direction.” But the automated weather observation equipment was

telling him that the airfield was within the limits. He said he “smelled a rat.”

He made the approach, but the visibility was only 2 1⁄2 miles and the ceiling

was under 700 feet. He could not see the runway, so he made a missed

approach. He did not really know how the automated observation equipment

worked, but he sensed that it might be wrong based on his assessment of the

conditions, and his intuition proved him correct. He ended up flying to a

close alternate airfield.

Cognitive demand: Can I deal with

low ceiling and limited visibility at

destination?

Information sources: ASOS at

destination; pilot’s observation of

conditions at destination, pilots’

experience with similar weather

situations

Cues: temperature/dewpoint spread;

wind direction

Actions: attempt approach, observe

ceiling and visibility

Cue: couldn’t see runway

Action: abort approach, fly to

alternate destination
7.12 Appendix L: "Scenarios from Hell" Incidents

L-1

Subject 3 -

• Windshear at takeoff or landing

• Add to workload at landing by changing runways (need to redo approach)

• Low ceilings encountered just after decision height (expert will prepare for missed approach)

• Forecast a close dew pt/temp spread

• Runway clear until rotate speed reached, then socked in

• Start flare on landing and runway in fog

• While in holding pattern, ice accumulates on the bottom (of wings?)

• Ice in clouds

Any slow forming icing conditions (when flying slow, low altitude) [this is particularly difficult when on

autopilot that continues to adjust for it until overloaded]

Subject 5 -

• Strong wind sheer, unexpected at takeoff or landing

• Engine failure on opposite side of cross-wind

• Icing during night flight

• Equipment failures that affect de-icing equipment (asymmetrical malfunctions are hard to control)

• Contaminated runways

• Pitot-static system failure with slow onset (altitude and airspeed indicators are affected)

• Radar clutter or misleading radar image (big cell behind the squall line)

Subject 6 -

• Destination is an uncontrolled airport, maybe no FBO

• Snow or frost on runway—no braking action

• Ceiling and visibility at minimums

• Forecast winds with gust from 10-20 knots

• Landing on runway at 3 degrees

• EICAS gives message for anti-skid failure after takeoff, but then this goes away

• EICAS message comes back on when pilot tries to put wheels down for landing [he should abort landing and look

for alternate with dry runway]

• If fail to gear down, must fly at low altitude to alternate or retract them and do a belly landing

• When aircraft starts to descend the winds are up to 25-30 knots with a 90 degree crosswind

• Runway has blowing snow and anti-skid has failed

• Takeoff from south

• Have thunderstorms in the area

• Running low on fuel

• Runway 4500-5000 ft

Subject 8 -

• Forecast icing conditions (especially for night flights)

• Use a plane with no deicing equipment

• Forecast low visibility at destination

• Co-pilot that hinders (doesn’t call out altitude or descent rate on approach)

• Wake turbulence can be tricky

• Strong wind sheer

• Strong cross winds at takeoff and landing

• Engine failure at takeoff with strong cross wind (makes physical control difficult)

• Bleed air failure (system that affects deicing equipment)

• Pitot-static system icing failure—slowly degrades validity in instruments

Subjects 1,2,4 and 7 did not receive this probe due to lack of time available in the interview.
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-1

CD- Incident 1 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Landing at smalluncontrolledairport

x x Airport directory;

pilot's ownexperience

Cloudy,destination airportnot visible

x x x Visual observation

out the window,usually able to seefrom this distance;ATIS; PIREPS;ATC; FBO onground

Turbulence inclouds

x x x x Sensing the plane

being jerked around

Airport conditionson ground poor,FBO can't seeacross airport

x x FBO

Temps decreasingto freezing level

x x ATIS 

Precipitation onradar

x x x x On-board radar 

Clear conditions atdeparture airport

x x Visual observation;

ATIS, ATC
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-2

Not able to land atdestination

x x Call FBO for weather

on ground atalternate
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-3

CD- Incident 2 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Hurricaneapproachingdestination

x x x FAA

Briefing/Internetforecast

Landing on asmall island

x x x PIREPS, ATC 

DestinationAWOS notcertified forhandling weatherconditions

x Own experience

Wind speed anddirection aconcern forapproach andlanding

x x x FBO on Ground at

destination; ATIS,ATC; PIREPS

Precision approach x x Pilot knowledge of

airport, andexperience landingon similar sizedislands to thisapproach & landing
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-4

CD- Incident 3 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

DepartureConditions rain,fog, below mins

x x x ATIS; Pilot's own out

window view; ATC

Temperaturesdecreasing to 3 to4 degrees C atdeparture anddestination

x x x x x ATIS 

Conditions atdestination belowmins, however,changing andvariable

x x x KAVOURAS

service; Called towerat destination

Ragged clouds atdestination

x x x Called tower at

destination

Co-Pilot andcaptain disagreeon course ofaction. Co-Pilotdoesn't think theyshould go, 1stpilot feelsotherwise

x x x Forecast; ATC at

departure anddestination, ATIS

Flying at night x x Pilot able to see the

amount of reflectionof airport lightsshining throughragged clouds.
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-5

CD- Incident 4 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Destination ariportsituated in a valley

x x Own experience

landing at destination

Only one runwayviable due to winddirection andspeed

x x ATC; Pilot's

knowledge ofairfield; AirportDirectory

Significant crosswind on approach

x x x ATIS

Violent gust andturbulence onapproach, gettingworse and worseas descend

x x x ATIS; plane getting

jerked around a lot;significant feeling ofpitching and yawing

Passengers gettingnervous, tense

x x x 

Surroundingairports reporting

x x Called La Guardia to

check for a slot;
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-6

same winds, andare busy

PIREPS

Pilot losingcontrol of aircraft

x x x 

Severity ofconditions wereunexpected; pilotcalled ahead ondeparture - NYarea reported noproblems

x x x x Called Wx service;

pilot and crew couldfeel the plane beingjerked around

Departure weatherwas beautiful

x xx

Flying from westcoast to east coast

x x Pilot observation;

using ownknowledge andexperience of flyingthis flight path.
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-7

CD- Incident 5 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Severe stormapproaching fieldat 30 knots

x x x On-board radar; ATC

radar; look out thewindow

Conditions at fieldare good despiteapproaching storm

x x x Observation out the

window; ATIS; ATC

Expediteddeparture

x 

Decision ofwhether to take offor wait for stormto pass needed tobe made quickly

x x Used knowledge of

weather and flyingexperience to makeassessment
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-8

CD- Incident 6 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Automatedobservationequipment saysweather is withinlimits

x x ASOS 

Pilot feels likeconditions areworse than whatautomated weatherequipment istelling him

x x x Pilot's observation

out the window

Pilot can't seerunway, visibilityonly 2.5 miles,ceiling < 700 ft

xx x

Small,unsupportedairport

x x Pilot has been there

before, knowslimitations of airport.

Pilot unable toland at destinationairport

x x x 
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-9

SH- Incident 1 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Destination isuncontrolled (mayhave a FBO)

X X Airport directory,

experience

Snow/frost onrunway

X X X FBO report

Ceiling & vis atmins

X X X X FBO report,

Automated reporting

Forecast/actualwinds gusting 10-20kts, crosswind

X X X FBO, Automated

reporting

Winds increase togusting 25-30kts,crosswind

X X FBO, Automated

reporting

EICAS gives antiskid failuremessage, but thengoes off

X X In-cockpit warning

EICAS warningreappears whengear is deployed

X X In-cockpit warning 

Additional:T-storms in thearea

X X X On-board radar,

Sigmets, HIWAS

Low on fuel X X X Fuel indicator Runway short(4500-5000ft)

X X Airport directory,

experience
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-10

SH- Incident 2 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Night X X Time of day, visual Forecast icing X X X Forecast (DUATS) System failures.- no deicingequipment, orbleed air failure;- pitot static failure

X X Bleed air failure

inhibits ability to deice.Pitot-static failure -due toicing/blockage,instruments willcause inaccurateairspeed indications.

Forecast lowvisibility

X X X Forecast; current

ATIS indicators ofceiling and visibility

Unhelpful copilot(doesn't call outaltitudes ordescent rate)

X X X Lack of call-outs by

copilot; copilotdistracted; copilotun-responsive

Strong windshear X X X X X Forecasts, other

PIREPS, radiochatter, windshear
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-11

alert in cockpit, pilotobservations of windbehavior in the air onindicators on theground (treesblowing, gusts acrossbodies of water,windsock)

Wake turbulence X X X X X PIREPS, windshear

alert in cockpit

Strong cross winds X X X See windshear

indicators
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-12

SH- Incident 3 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Strong windshear X X X Forecasts, other

PIREPS, radiochatter, windshearalert in cockpit, pilotobservations of windbehavior in the air onindicators on theground (treesblowing, gusts acrossbodies of water,windsock)

Failure of upwindengine in strongwinds

X X X Yaw in direction of

failed engine; mastercaution/warninglights; other flightdeck indicator lights& system displays

Icing during nightflight

X X X (night) Aircraft has sluggish

response to controlinputs - loss ofairspeed with notchanges in powersetting; may haveicing indicator lights;see outside airtemperature andmoisture in the air(clouds)

Equipment failuresaffecting de-icingequipment

X X Flight deck

indicators for systemmalfunctions (e.g.,electrical power,bleed air system)

Contaminatedrunways and poor

X X X PIREPS; tower

reports; visual on
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-13

braking action airfieldPitot-static failurewith icingconditions, slowonset

X X X See indicators for

icing and for pitotstatic equipmentfailures above

Radar clutter,misleading radarimage

X X On board radar 
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-14

SH- Incident 4 Phases of Flight Features InvolvedChallengingElements:

Takeoff

Climb

Cruise

Descend

Land

All

Weather

System

Environment

Infrastructure

InformationSources

Switch runwayslast minute

X X X ATC request; 

Windshearpotential orcrosswinds on newrunway

X X X See information

sources forwindshear above

Low ceilings onapproach

X X X ATIS, visual,

PIREPS, reportsfrom tower

Forecast a closedew pt/tempspread

X X Forecast, ATIS 

Reduced vishalfway downrunway at decisionpoint (reached

X X Visual cues 
7.13 Appendix M: Scenario Elements and Behavioral Markers.

M-15

rotate speed)Ice accumulationwhile holding forrunway [add adistractor task totake the pilot'sfocus away fromflying the plane]

X X X See indicators for

icing

Icing in clouds X X X See indicators for

icing

Slow iceaccumulation +autopilot + flyinglow and slow

X X X See indicators for

icing
REPORT DOCUMENTATION PAGE Form Approved

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data

sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other

aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and

Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188),

Washington, DC 20503.

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE

July 2001

3. REPORT TYPE AND DATES COVERED

Technical Memorandum

4. TITLE AND SUBTITLE

Cognitive Task Analysis of Business Jet Pilots' Weather Flying Behaviors:

Preliminary Results

5. FUNDING NUMBERS

WU 728-40-10-02

6. AUTHOR(S)

Kara Latorella, Rebecca Pliske, Robert Hutton, and Jason Chrenka

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

NASA Langley Research Center

Hampton, VA 23681-2199

8. PERFORMING ORGANIZATION

REPORT NUMBER

L-18103

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

National Aeronautics and Space Administration

Washington, DC 20546-0001

10. SPONSORING/MONITORING

AGENCY REPORT NUMBER

NASA/TM-2001-211034

11. SUPPLEMENTARY NOTES

Latorella: Langley Research Center, Hampton, VA; Pliske, Hutton, and Chrenka: Klein Associates Inc.,

Fairborn, OH (Pliske is currently at Dominican University, River Forest, Illinois)

12a. DISTRIBUTION/AVAILABILITY STATEMENT

Unclassified-Unlimited

Subject Category 03 Distribution: Nonstandard

Availability: NASA CASI (301) 621-0390

12b. DISTRIBUTION CODE

13. ABSTRACT (Maximum 200 words)

This report presents preliminary findings from a cognitive task analysis (CTA) of business aviation piloting.

Results describe challenging weather-related aviation decisions and the information and cues used to support

these decisions. Further, these results demonstrate the role of expertise in business aviation decision-making in

weather flying, and how weather information is acquired and assessed for reliability. The challenging weather

scenarios and novice errors identified in the results provide the basis for experimental scenarios and dependent

measures to be used in future flight simulation evaluations of candidate aviation weather information systems.

Finally, we analyzed these preliminary results to recommend design and training interventions to improve

business aviation decision-making with weather information. The primary objective of this report is to present

these preliminary findings and to document the extended CTA methodology used to elicit and represent expert

business aviator decision-making with weather information. These preliminary findings will be augmented with

results from additional subjects using this methodology. A summary of the complete results, absent the detailed

treatment of methodology provided in this report, will be documented in a separate publication.

14. SUBJECT TERMS

Cognitive Task Analysis; Business Jet; Weather Information; Human Factors;

15. NUMBER OF PAGES

129

Aviation 16. PRICE CODE

A07

17. SECURITY CLASSIFICATION

OF REPORT

Unclassified

18. SECURITY CLASSIFICATION

OF THIS PAGE

Unclassified

19. SECURITY CLASSIFICATION

OF ABSTRACT

Unclassified

20. LIMITATION

OF ABSTRACT

UL

NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89)

Prescribed by ANSI Std. Z-39-18

298-102