DOT/FAA/AM-97/3

Office ofAviation Medicine

Washington, D.C. 20591

The Use ofWeather

Information in Aeronautical

Decision-Making

Walter E. Driskill

Johnny J. Weissmuller

John Quebe

Darryl K. Hand

Martin J. Dittmar

Metrica, Inc.

San Antonio, Texas

David R. Hunter

Office ofAviation Medicine

Federal Aviation Administration

Washington, DC 20591

February 1997

Final Report

This document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161.

© D?&'

Tf? I*1*

U.S.Department

of Transportation

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents or use thereof.
Technical Report Documentation Page

1. Report No.

DOT/FAA/AM-97/3

2. Government Accession No.

4. Title and Subtitle

The Use of Weather Information in Aeronautical Decision-Making

7. Author(s)

W.E. Driskill, J.J. Weissmuller, J. Quebe, D.K. Hand, M.J. Dittmar,

(Metrica, Inc.), and D.R. Hunter, (FAA).

9. Performing Organization Name and Address

Metrica, Inc.

San Antonio, TX

12. Sponsoring Agency name and Address

Office of Aviation Medicine

Federal Aviation Administration

800 Independence Ave., S.W.

Washington, DC 20591

FAA Office of Aviation Medicine

800 Independence Ave., S.W.

Washington, DC 20591

15. Supplemental Notes

3. Recipient's Catalog No.

5. Report Date

February 1997

6. Performing Organization Code

8. Performing Organization Report No.

10. Work Unit No. (TRAIS)

11. Contract or Grant No.

13. Type of Report and Period Covered

14. Sponsoring Agency Code

16. Abstract

An investigation was conducted ofthe values or worth functions pilots attribute to weather and terrain variables in

making decision about flight in a single-engine aircraft under visual flight rules. The research employed a scenariobased judgment task designed to elicit pilot worth functions for visibility, ceiling, precipitation, and terrain features

(specifically, flight over flat terrain, mountainous terrain, and over water).

Subjects were asked to consider a proposed cross-country flight of approximately one hour duration over each of

the three terrain conditions. They were then given a deck of cards describing various combinations of ceiling,

visibility, and precipitation and asked to first rank order the cards in descending order of comfort with flying under

the described conditions over the specified terrain and then to assign a numerical rating to each card using a scale

of 0 (least comfortable) to 100 (most comfortable). For each subject a linear model was constructed using the

comfort ratings as the dependent variable and the visibility, ceiling, and terrain values as the independent variables.

A separate linear model was constructed for each subject for each ofthe three terrain conditions. The regression

weights associated with each terrain type were then subjected to hierarchical clustering to identify groups of weight

sets differing significantly with respect to worth function emphasis.

Examination ofthe clusters indicated that pilots could be reliably grouped into four groups for the flat and

mountainous terrain scenarios and two groups for the over-water scenario. These clusters are characterized

primarily by the average degree of comfort which the subjects expressed in flying under the conditions described,

by the nature oftheir efficiency of utilization of information (multiplicative models compared to simpler single

variable models), and demographic characteristics such as age and aircraft ownership.

17. Keywords

Pilots, Aircraft, Decision-Making, Aviation Safety,

Aircraft Pilots, Policy-Capturing, Mathematical

Modeling

19. Security Classif. (of this report)

Unclassified

18. Distribution Statement

Document is available to the public through the

National Technical Information Service

Springfield, Virginia 22161

20. Security Classif. (of this page)

Unclassified

Form DOT F 1700.7 (8-72)

21. No. of Pages

56

22. Price

Reproduction of completed page authorized
ACKNOWLEDGMENTS

Collection ofdata from civilian pilots on which this research report is based was made possible by the

cooperation ofnumerousindividuals. We acknowledge their very important contribution. Especially, we

express appreciation to:

Owen Rüssel

San Antonio, Texas

Federal Aviation Administration Office.

Mr. Rüssel not only was responsible for locating and arranging for collection ofdata from the civilian

pilots, but he participated as a proctor during group administration. His assistance made possible the

generation ofthe research report.

Others contributed significantly to the development of data collection instruments and to data

collection. We also express our sincere appreciation to the following persons and organizations with

which they are affiliated:

Marvin G. Ledyard

Alcides Dreumont

William Cervinske

Bruce D. Hoover

Gary J. Northam

L.W. Fournier

Huey McDonald

Lauren Fast

Jim Wright

Randy Smedly

Virginia McDaniel

Thomas E. Taylor

Finally, we are appreciative ofthe time and interest of the 152 anonymous pilots who provided the

raw data employed in the analysis.
ACKNOWLEDGMENTS

Collection ofdata from civilian pilots on which this research report is based was made possible by the

cooperation ofnumerous individuals. We acknowledge their very important contribution. Especially, we

express appreciation to:

Owen Rüssel

San Antonio, Texas

Federal Aviation Administration Office.

Mr. Rüssel not only was responsible for locating and arranging for collection ofdata from the civilian

pilots, but he participated as a proctor during group administration. His assistance made possible the

generation ofthe research report.

Others contributed significantly to the development of data collection instruments and to data

collection. We also express our sincere appreciation to the following persons and organizations with

which they are affiliated:

Marvin G. Ledyard

Alcides Dreumont

William Cervinske

Bruce D. Hoover

Gary J. Northam

L.W. Fournier

Huey McDonald

Lauren Fast

Jim Wright

Randy Smedly

Virginia McDaniel

Thomas E. Taylor

Finally, we are appreciative of the time and interest of the 152 anonymous pilots who provided the

raw data employed in the analysis.
TABLE OF CONTENTS

PAGE

INTRODUCTION j

METHOD 2

General Considerations 2

Design Implications ofBehavioral Decision Research 3

Data Collection Instrument Development /

Pilot-Subjects -,

Data Collection Procedure -,

RESULTS

Identification ofWorth Functions g

Noncompensatory Strategy Detection 13

DISCUSSION AND CONCLUSIONS 13

REFERENCES l6

APPENDIX A-Data collection package A1

APPENDIX B - Pilot information form data B1

APPENDIX C - Expert pilot safety ratings package d

APPENDIX D - Plots of expert pilot safety ratings Y)\

APPENDIX E - Selected pair comparisons among policy groups for each scenario El

APPENDIX F - Comprehensive occupational data analysis program pj

LIST OF TABLES

Table 1. Reliability of the expert pilot safety ratings g

Table 2. Standardized high, medium and low levels ofsafety variables 6

Table 3. Mean experience levels -,

Table 4. Non-mountainous group average weights o

Table 5. Mountainous group average weights q

Table 6. Over water group average weights o

Table 7. Mean demographic variables and comfort levels for non-mountainous scenario groups 10

Table 8. Mean demographic variables and comfort levels for mountainous scenario groups 11

Table 9. Mean demographic variables and comfort levels for over-water scenario groups 12

Table 10. Comparison of compensatory and non-compensatory models 14
EXECUTIVE SUMMARY

The body of this report details how 152 general

aviation pilots were assessed to determine the weights

(worth functions) they assigned to the variables of

terrain, ceiling, visibility and precipitation. The obtained, analysis results were sufficiently clear to allow

the identification (at least tentatively) of distinctly

different worth functions. Specifically, four different

worth functions were identified and characterized for

non-mountainous and mountainous terrain types.

Two different worth functions were identified and

characterized for over-water terrain. Use of weather

information was consistent with expert assessment of

the safety risk associated with VFR flight under varying ceiling, visibility, and precipitation conditions

over non-mountainous, mountainous, and water terrain. However, while use is consistent, pilots vary in

their expressed degree of comfort in flying over the

different terrain under different weather conditions.

These differences in comfort level may occur, in part

at least, because of:

•Differences of understanding of the risks associated with flying under differing above-minimum

levels of visibility, ceiling, and precipitation and

how terrain type impacts use ofsuch information.

•Differences in their self-assessment and perception oftheir ability and/or skill in flying under the

varying conditions.

In summary, this study found:

• It is possible to capture reliably pilots' "comfort"

policies in terms ofemphasis upon terrain, ceiling,

visibility, and precipitation using a linear modeling approach.

•Pilots use compensatory judgment rules with sophisticated interactions.

•Pilots generally agree on how to prioritize the

safety implications ofterrain/weather conditions.

•Pilots' policies tend to fall cleanly into stable

policy groups although the individual pilots who

hold those views switch groups according to terrain.

• Pilots' policies vary based upon:

1. General comfort level

2. A mixture of age and general experience

3. Number of hours flown in last 90 days

4. Reasons for flying (employed as pilot versus

pleasure flying)

IMPLICATIONS

• Pilots' use of weather information is consistent

with expert assessment ofthe safety risk associated

with VFR flight under varying ceiling, visibility,

and precipitation conditions over non-mountainous, mountainous, and water terrain.

•While use is consistent, pilots vary in their expressed degree ofcomfort in flying over the different terrain under different weather conditions.

•Training or other interventions addressing risk

assessment and self-perception would appear to be

appropriate. Present use of the Five Hazardous

Thought Patterns is one method of addressing self

perception. Training directed at improving pilot

understanding of the effects of terrain-type on

interaction of meteorological conditions is another potential source. Increased emphasis on risk

assessment and self perception training and exercises in initial training and in subsequent seminars

(FAA-sponsored and otherwise) would seem to be

warranted.

• Although the compensatory decison model is used

by expert and by many other pilots as well, its use

in all situations may not be advisable. Some care

and consideration must be observed in training

inexperienced pilots to use the decison models of

expert pilots, as their injudicious use may be a

threat to aviation safety.
THE USE OF WEATHER INFORMATION IN AERONAUTICAL DECISION-MAKING

INTRODUCTION

A National Transportation Safety Board report

(NTSB, 1989) reviewed 361 general aviation accidents that occurred between 1983 and 1987 in which

Visual Flight Rules (VFR) into instrument meteorological conditions (IMC) was listed as a probable

cause or a related factor. (Most accidents have multiple causes/factors assigned as probable.)

Ninety-seven percent of the probable causes were

attributed to the flight crew, and 42% of these cited

the manner in which weather information was obtained (or not obtained), assimilated, and used. There

were 276 fatal accidents, resulting in 583 fatalities.

An earlier review of aircraft accident data (Jensen,

1982) reported similar data and concluded that 80 to

85% of aircraft accidents can be assigned broadly to

pilot error. The review also indicated that 51.6% of

the fatal and 35.1% ofthe non-fatal accidents resulted

from faulty decision-making. Because of the prominence ofpilot error and flight from VFR into IMC as

causes of general aviation accidents, the Federal Aviation Administration (FAA) sponsors training interventions on pilot decision-making which address both

factors. The interventions focus on both cognitive

and affective (or motivational) components of pilot

decision-making (Brecke, 1981;Jensen, 1982;Jensen,

Adrion, & Lawton, 1987). These two components

differentiate among processes pilots employ in using

and evaluating information available to them and

making choices about actions to take. The cognitive

component deals with the processes by which pilots

establish and evaluate the alternatives in a decisionmaking situation. Jensen (1982) proposed that this

component relates to pilot ability to search for and

establish the relevance of all available information

regarding a situation, to specify alternative courses of

actions, and to determine expected outcomes from

each alternative. This component invokes the intellectual, discriminative abilities which depend upon

human capabilities to sense, store, retrieve, and integrate information.

From a cognitive perspective, according to Jensen

(1982), pilots should discriminate among situational

dimensions. He asserts that for discriminative judgment, these important questions must be addressed:

1) Do pilots consider all situational dimensions and

do they assign proper weights to the data elements for

each of these dimensions? 2) Do pilots efficiently

integrate the relevant information from the data elements?

Cognitive training provides information designed

to increase pilot awareness of flight safety under

adverse conditions and to assist pilots in assigning and

integrating proper weights to the data elements. Training consists of exercises demonstrating the risks associated with flying activities and the underlying

behavioral causes of typical accidents. The effects of

stress on decision-making and the management of

stress are also important training elements. Informational materials include data about such factors as

causes of accidents by phase of operation and pilot

experience, the aeronautical decision-making processes, and weather forecast accuracy and use. Information about weights pilot attribute to flight data

elements, however, is not available for use in training

or other interventions.

According to Jensen et al. (1987), use of aeronautical decision-making interventions for instrument

pilots produces substantial reductions in pilot error

rates. Reduction in decisional mistakes attributed to

the training ranges from 10 to 50%. Similar training

exists for student, private, commercial, and multicrew aircraft pilots.

A very limited amount of research on the cognitive

processes ofpilot decision-making has been reported.

This research, however, has two important findings.

First, the feasibility of investigating the intellectual

processes in decision-making through assessment of

pilot worth or values pilots attribute to flight data

elements has been demonstrated. Second, pilots appear to weight flight data elements differentially,
because of such factors as experience and training.

Flathers, Giffin, & Rockwell (1982) presented pilots

with a diversion-decision scenario requiring an inflight decision to change destinations during a crosscountry flight, because of equipment malfunction.

Subjects were presented alternative landing sites varying according to weather conditions, air traffic control facilities, approach, and flight time to reach the

sites. Two levels of each variable were represented.

Subjects rank-ordered the sites according to their

preference ofdiversion landing site. Conjoint analysis

was employed to derive worth functions for each of

the four variables. Comparisons of the worth functions among pilots with differing experience and

training (e.g., flight hours, source of certification)

revealed that pilots differentially weighted the variables. Interactions among the four variables were not

found. Flathers etal. also referenced a study by Curry

(1976) that reported similar results from a study of

approach-to-landing decision making. These results

suggest that pilot error in decision making, one ofthe

two most frequently cited probable causes of general

aviation accidents, may, in part, occur because pilots

do not assign proper weights to the flight data elements that are available to them.

The present study sought to extend this area of

research through an investigation of the values or

worth functions pilots attribute to weather and terrain

variables in making decisions. The specific objectives

were as follows:

•Identify individual weights or worth functions

pilots attribute to ceiling, visibility, precipitation,

and terrain flight data elements in three representative cross-country flights in a small aircraft.

• Assess interaction effects on assignment ofweights

that may affect the worth functions of the data

elements.

•Assess whether pilots differentially weight these

data elements, because ofsuch personal factors as

age, flying hours, and source of certification.

METHOD

General Considerations

The research reported here employed a scenariobased judgment task designed to elicit pilot worth

functions for visibility, ceiling, precipitation, and

terrain variables. Behavioral decision research shows

that task design is crucial for making consistent judgments. Depending upon the characteristics of a judgment task, individuals sometimes use a compensatory

strategy and other times employ noncompensatory

strategies in making decisions. A compensatory strategy is one which processes all relevant information

and trades off the good and bad aspects of each

alternative. Noncompensatory strategies avoid tradeoffs among values by reducing information processing demands by ignoring potentially relevant problem

information, such as selecting a single attribute and

sorting on that alone.

A major concern was to design a task that would

avoid use of noncompensatory cognitive strategies by

the subjects when they were making their judgments.

Subjects who employ cognitive decision strategies

that are artifacts of the problem task, such as restructuring, application of simple heuristics (representativeness, availability, anchoring) and other more

complex noncompensatory processes (e.g., elimination-by aspects, lexicographic) usually are trying to

reduce cognitive effort. As a result, subjects frequently

neglect information relevant to the decision problem

(Payne, Bettman, &Johnson, 1988; Payne, Bettman,

& Johnson, 1992).

The task was intended to facilitate consistent, compensatory decision strategy utilization by the respondents to make, using Jensen's (1982) terminology,

rational judgments. Rational judgment implies that

pilots should use weighted-additive strategy processes.

In the weighted-additive process pilots should be

expected to consider the values of each element or

attribute; weight the values of each element by its

importance for a safe flight; and sum the weighted

values as a basis for overall evaluation of each flight

scenario. In practice, however, some pilots may be

expected to have developed mixed cognitive strategies

(from their experience or training) for considering
data elements. While some may have weightedadditive strategies which they consistently apply to

judgments, other pilots may have developed rules

which they apply to some oftheir judgments that give

the appearance ofsome non-compensatory strategy or

heuristic, such as elimination-by-aspects. For example,

some pilots may have developed a simple heuristic

thattheywill not make any flight over water regardless

of weather conditions; however, weather conditions

may have important values when flying over other

kind ofterrain. The possibility of multiple strategies

was addressed in the analytic methodology (Appendix

F) employed by providing for distinguishing between

"real" multiple policies and those that may have been

artifacts of the task itself.

Design Implications of Behavioral Decision

Research

Design ofthe decision task was based on the results

of behavioral decision research which indicates that

differences in decision strategies within a problem

task occur, according to Payne (1982), because "...information processing in decision making, as in other

areas of cognition, is highly contingent upon the

demands of the task" (p. 382). Different decision

strategies can be thought of as different rules for

conducting the search among the attributes and alternatives (Payne, Bettman, & Johnson, 1988). To

reduce the likelihood of subjects in this study using

strategy variations which are induced by the nature of

task itself (versus using their own decision sets),

careful attention was given to response mode and task

and context characteristics (Einhorn & Hogarth 1981;

Payne, Bettman, & Johnson, 1988; Payne, Bettman,

& Johnson, 1992).

Judgment versus Choice Response Mode. Judgment

and choice tasks require different cognitive processes.

In a judgment task, subjects are provided successive

presentations ofalternatives to each ofwhich a subject

assigns a value reflecting its psychological worth.

Sometimes the values are in terms of a rating scale;

sometimes the psychological worth might be expressed

in terms of the amount of money the subject would

pay for the alternative. In a choice task, subjects are

presented with two or more alternatives and asked to

select the alternative most preferred or the order in

which the alternatives are preferred.

Some ofthe clearest examples of decision reversals

have been noted between judgment and choice tasks.

Subjects reversed their decision in tasks containing

identical information as a function of whether they

were required to make judgments among the dimensions or make choices among the dimensions (Payne,

1982). Because ofthese findings, a judgment task was

used in this research. In judgment tasks, subjects seem

to select an attribute ofan alternative as an anchor and

then adjust that anchor based on information associated with other attributes. Such processing is more

holistic. The choice mode seems to involve primarily

dimensional processing by which dimensions of one

alternative are compared with parallel dimensions of

other alternatives.

Task and Context Task Factors. Task and context

factors cause different aspects of the problem to be

salient and evoke different processes for combining

information (Payne, Bettman, & Johnson, 1992, p.

90; Payne, 1982; Einhorn & Hogarth, 1981). Task

factors are general characteristics ofa decision problem

which do not depend upon the particular values ofthe

attributes or alternatives, such as the number of

alternatives, number ofattributes, information display,

and time limits. Context factors, such as similarity of

alternatives, are associated with the particular values

ofthe alternatives. The following task factors guided

development of the experimental task:

•In general, cognitive effort in responding to a

decision task seems to increase as the number of

alternatives presented for each choice problem

increases. Subjects tend to employ compensatory

types of decision strategy when choices are made

between only two alternatives. As alternatives are

added, subjects tend to use choice strategies, such

as the elimination-by-aspects rule. In the task

employed in this research, judgments among the

weather-terrain scenarios was not required. Subjects considered each weather-terrain scenario independently and judged their comfort level about

flight under the conditions described.
• In general, increases in the number ofattributes or

dimensions (e.g., the amount of information) of

alternatives increases the variability ofstrategy and

decreases quality ofjudgments. Subjects are likely

to use some decision rule (e.g., elimination by

aspects) when faced with complex choices. The

design of the problem task in this research limited

each scenario to four attributes or dimensions.

•Time pressure seems to affect task complexity.

Measures of information search showed that subjects tend to spend more time observing negative

information under time pressure. Although no

time limit was imposed, most subjects in the study

reported here completed the task in less than 50

minutes (including time for administrative instructions).

•Information display also may affect judgmentchoice behavior. Use of attribute information was

reported as increased when this information was

displayed in an organized manner and that information acquisition (about attributes) tends to

proceed in a way consistent with the display format. Studies have demonstrated that displays designed to encourage alternative-based processing

tended to produce more alternative-based information processing. Also, completeness of the information displayed is important. Subjects may

tend to discount alternatives when there is missing

information about one or more attributes. The

judgment task presented dimension information

consistently across all scenarios. Each scenario

contained the same dimensions. Although values

varied, dimensions appeared in the same order and

format. Other data elements than those represented in the dimensions which could be important in making judgments about the scenarios

(e.g., equipment status, time of day, pilot physical

and psychological condition) were held constant

across all scenarios for two reasons. First, the

appearance of missing information was avoided.

Second, variations in these elements would have

introduced uncontrolled variables into the design.

• The only context variable that could have implications for the proposed research relates to the similarity ofdimensions or attributes. Itseems clear that

in choice tasks where choices among alternatives is

the task, similarity of attributes among alternatives

increases the cognitive effort required to make choices.

The effort required to make comparisons among

alternatives essential to choice-making is increased,

thus inducingsome non-compensatory strategy, such

as elimination-by-aspects (Payne, 1982). Since

judgment of each scenario was made independently, the effects of similarity should have been

eliminated or minimized.

Data Collection Instrument Development

Data collection instruments consisted of a) a test

booklet (used to obtain pilot background and experience information), b) a set oflaminated colored maps

(one for each of three terrain types: water, nonmountainous, and mountainous), and c) three sets of

en route weather scenarios (each set corresponding to

a terrain type). Appendix A is a set of these data

collection instruments.

Test Booklets. Booklets consisted of three separate

parts. Part one contained instructions for assigning

"comfort" ratings to the weather scenarios. Part two

elicited demographic information such as certification

source, age, total flying hours, total hours flown in the

last 90 days. This demographic information paralleled

that cited by the National Transportation Safety Board

(NTSB, 1989) as being related to aircraft accidents

the NTSB studied from 1983 to 1987. Part three of

the booklet was designed to obtain the pilot-subject's

degree ofexperience flying over each ofthe three types

of terrain, and their experience flying into various

levels ofprecipitation (light rain, moderate rain, heavy

snow), visibility (1 NM, 4 NM, 8 NM), and ceiling

(800 ft, 1800 ft, 4000 ft). A 5-point scale (0=1 have

never flown into these or highly similar conditions;

4=1 have flown into these conditions numerous times

(11+)) was used for these ratings.

Terrain Maps/Flight Routes. The terrain mapsselected

were for the Great Lakes (overwater), North Texas (nonmountainous) and New Mexico (mountainous) areas
and were clearly identifiable with respect to these

terrain types. In the selection ofthe over-water terrain

map, southern coastal areas were removed from

consideration because oflimitations imposed on the

development ofrealistic weather scenarios (e.g., snow

would not be a condition normally encountered over

southern coastal areas). Itwas feltthatlocal knowledge

of terrain, if possessed by some subjects and not

others, would act as an uncontrolled independent

variable. The importance of minimizing this local

knowledge was emphasized by the Shanteau and Nagy

(1979) research which indicated thatsubjects' previous

knowledge about a task to be performed influenced

their choices in task performance. Since pilots were to

be drawn from the San Antonio, Texas area, the route

map selected (Great Lakes, North Texas, and New

Mexico) minimized the potential impact of local

knowledge.

Routes to be flown (City County to Delta County,

Michigan for over water, Hereford to Dalhart, Texas

for non-mountainous, and Las Vegas to San Juan

Pueblo, New Mexico for mountainous) were laid out

to minimize impact ofrestricted, special-use airspace,

and low-level military training routes. These flight

routes were clearly marked on the corresponding color

terrain map. In addition to flight routes, maps were

also annotated with local and destination weather and

a description ofthe type and condition ofthe aircraft

being flown (e.g., Cessna 172 without transponder

and navigation radio out).

Scenario Development. En route weather scenario

content was developed from the NTSB (1989) study

of weather-related general aviation accidents from

1983 to 1987. In addition to a probable cause ofVFR

flight into IMC conditions, which was the criterion

used by the NTSB to select accidents for analysis,

89% ofthese accidents were attributed to the following

four factors:

Terrain conditions 225(18.6%)

Low ceiling 216 (17.9%)

Poor visibility 448 (37.2%)

Precipitation 180 (15.3%)

As a result ofthese findings, scenarios were generated (using a multi-stage process) around variations in

terrain, ceiling, visibility, and precipitation. Specifically, 81 scenarios were produced to accommodate

three levels of each of these four independent attributes. Each scenario was printed on a 5 by 4 inch

card, color coded with respect to terrain type, and

described a combination of en route weather conditions (ceiling, visibility, and precipitation) encountered during the cruise phase offlight over each ofthe

three terrain types.

The first stage ofscenario development focused on

the identification and validation of the three levels

(low, medium, and high) of ceiling, visibility, and

precipitation variables to be used in scenario development. Specifically, an Expert Pilot Safety Ratings

Form (Appendix C) was developed and administered

to 22 pilot-subjects. This group of pilots had an

average of 3,700 hours of total flight time and had an

average age of 42 years. Of the 22 pilots, 16 were

certified flight instructors and 11 were employed as

pilots. These 22 pilots used a 0-to-100 point scale

(0=Absolutely unsafe/100=totally safe) to rate the

relative safety of the three routes (over water, nonmountainous, and mountainous) and of levels of

ceiling, visibility, and precipitation with respect to

VFR flying by a pilot with 500 hours of flying experience.

These 22 pilot-subjects first provided safety ratings

for 8 levels ofprecipitation (ranging from no precipitation to freezing rain), 12 levels ofvisibility (ranging

from Vi NM to more than 8 NM), and 16 levels of

ceiling (ranging from 600 feet to over 5000 feet).

Interrater agreements for the obtained ratings (standardized to a mean of 5.0 and a standard deviation of

1.0) were computed using the GRPREL function of

the Comprehensive Occupational Data Analysis Programs (CODAP; Christal, 1974; Staley and

Weissmuller, 1981). (Abrief description of CODAP

and its components is provided in Appendix F.) The

interrater agreements, shown in Table 1, were extraordinarily high. The interrater agreement (Ru

) indicates the degree to which pilots within the policy

group agreed on how to sort and rate the 81 weather

scenarios with respect to comfortlevel. Values of 0.20

and above are considered "good agreement" (Staley
Table 1. Reliability of the Expert Pilot

Safety Ratings.

Table 2. Standardized High, Medium and

Low Levels of Safety Variables.

Variable Levels N

Raters

Rii Rkk

Routes 22 .67 .97

Precipitation Levels 22 .88 .99

Visibility Levels 22 .92 .99

Ceiling Levels 22 .92 .99

All Variable Levels 22 .82 .99

and Weissmuller, 1981). The policy stability measure

(Rkk) indicates the expected correlation ofthis policy

with the policy, which would be found in a re-survey

of "k" subjects. Over all variables and levels, Ru was

.82andRkkwas .99.

To identify values associated with the three levels

of precipitation, visibility, and ceiling, means and

standard deviations were computed, standardized to a

mean of 5.0 and a standard deviation of 1.0 for each

of the three attributes, and then plotted. Sets of

"High," "Medium," and "Low" values were then

identified (by inspection) for each level. The results,

shown in Table 2, represented a reasonable spread

across the range of values. Appendix D contains the

associated plots used.

Three sets of27 weather scenarios were then generated (one set for each terrain type). Specifically, scenario sets were initially generated such that each

scenario within a set was unique with respect to

precipitation, visibility, and ceiling level combination (e.g., High/High/High, High/High/Medium,

etc.). Within level, however, selection of the specific

value was randomized i.e., selection of 5000 ft, 4000

ft, or 3500 ft within the high level of the ceiling

variable was randomized. Each set of scenarios was

then randomly numbered (600-to-626 for non-mountainous, 700-to-726 for mountainous, and 800-to826 for water).

Prior to finalizing the three sets of scenarios, each

set was evaluated with respect to plausibility. Subjectmatter expert judgment was used for this purpose.

Specifically, 12 weather experts (meteorologists and

Variable Benchmark

value/Level

Precipitation:

none 6.745/High

light rain 6.000/High

light snow 5.514/Medium

moderate rain 5.205/Medium

moderate snow 4.687/Medium

heavy rain 4.287/Low

heavy snow 3.975/Low

freezing rain 3.589/Low

Visibility:

more than 8 NM 6.598/High

8NM 6.410/High

7NM 6.165/High

5NM 5.446/Medium

4NM 4.939/Medium

3NM 4.639/Medium

1 1/2 NM 3.960/Low

1 NM 3.896/Low

1/2 NM 3.822/Low

Ceiling:

5000 ft 6.419/High

4000 ft 6.244/High

3500 ft 6.081/High

2000 ft 5.393/Medium

1800 ft 5.010/Medium

1600 ft 4.702/Medium

900 ft 3.928/Low

800 ft 3.818/Low

600 ft 3.664/Low

weather observers from the National Weather Service

Forecast Office in San Antonio, Texas, and USAF

Weather Detachments at Reese AFB, Texas, and

Randolph AFB, Texas) were asked to rate the degree

to which scenario weather combinations (precipitation, visibility, and ceiling values) could reasonably

be expected to occur together (e.g., heavy snow, 5NM

visibility, and a ceiling of 3500 feet would not be

expected to occur simultaneously). Subjects rated

each scenario on a 0-to-100 scale (0=Totally improbable/100=Very probable). Obtained interrater agreement was high — Ru was .63 and Rkk was .95.
Analysis of the resulting mean ratings led to the

replacement of 12 of 81 draft scenarios. Scenarios

with mean ratings (standardized to a mean of 5.0 and

a standard deviation of 1.0) more than 1.5 standard

deviations below the mean rating were considered

improbable and replaced. Four scenarios from the

non-mountainous set, three from the mountainous

set, and five from the over water set were replaced

because of low mean ratings. The replacement scenario within a given set was drawn from one of the

other two sets. The scenario with the lowest acceptable mean value (ofthose scenarios not already used as

replacements) was always selected as the replacement

scenario. This process ensured that (to the extent

possible) an equivalentlevel weather combination was

substituted for the unrealistic weather combination.

Pilot-Subjects

A total of 152 pilots were tested from January to

March 1994. Complete data were not available for

two pilots who were subsequently dropped from the

study. The remaining pilots (131 males and 19 females) ranged in age from 18 to 79 with a mean age of

42. Total flying hours ranged from 6 to 26,500 with

a sample mean of 1,694 hours. Within this sample, 31

were certified flight instructors (CFI/CFII), 87 held

private pilot certificates, and 9 were students. The

remaining subjects were certified as either commercial

or airline transport pilots. Table 3 contains the pilotsubjects self-reported mean experience levels (0=Never

flown into condition/4=Flown into condition numerous times).

Data Collection Procedure

Introductory information was designed to be neutral. No reference was made to research into pilot error

and pilot-subjects were allowed to respond anonymously. Subjects were motivated to give their personaljudgments and to avoid "book" responses. It was

emphasized that the only right answers were the responses a subject chose to make.

Data collection took place in group settings, with

up to 25 pilots in a group. Pilots first read the

instructions contained in part one ofthe test booklet,

and then completed parts two (pilot information

form) and three (attribute experience ratings). Pilots

Table 3. Mean Experience Levels.

Attribute Mean Standard

Deviation

Terrain

Water 1.0 .95

Non-mountainous 3.6 .95

Mountainous 1.4 1.56

Precipitation

Light rain 2.4 1.18

Moderate rain 1.6 1.35

Heavy snow .4 1.00

Visibility

1 NM 1.7 1.44

4NM 2.7 1.29

8NM 3.6 .84

Ceiling

800 ft 1.7 1.55

1800 ft 2.9 1.23

4000 ft 3.6 .80

were then given the first of three sets of 27 weather

scenarios and (using the corresponding terrain map)

asked to sort the stack from least comfortable to most

comfortable about completing the flight. Next, the

pilots were asked to assign a 0-to-100 "comfort"

rating (0=least comfortable about completing the

flight/100=most comfortable about completing the

flight) to each scenario in the set. Pilots wrote their

rating on each scenario card in a clearly marked box.

When pilots completed the ratings for a set ofweather

scenarios, they were given the next set to complete.

The order of presentation of scenario sets to pilotsubjects was randomized.

RESULTS

Data analysis was designed to address two primary

issues. The first focused on the identification and

description of weights (worth functions) that pilots

attribute to ceiling, visibility, and precipitation with

respect to terrain type. This analysis included an

assessment of the relationship between pilots' worth

functions and associated personal factors such as age,

total flying hours, source of certification, etc.. The
second and more narrow issue involved data analysis

designed to detect the extent to which noncompensatory strategies were used by pilot-subjects. Analysis

methodology and analysis results associated with the

first issue (identification of worth functions) will be

described first.

Identification ofWorth Functions

Separate sets of analyses were conducted for each

terrain type. Specifically, each pilot-subject's worth

function (i.e., the relative emphasis given to ceiling,

visibility, and precipitation with respect to the assigned "comfort" ratings) was computed using the

following regression equation:

CF = A + B + C + AB + AC + BC + ABC

where CF was a vector of a pilot's 27 weather scenario

"comfort" ratings for a given type ofterrain; and A, B,

and C (and their interactions) were vectors of

benchmark values (of the corresponding scenarios)

for ceiling, visibility, and precipitation, respectively.

A total of 150 sets of regression weights (one set for

each pilot-subject) were computed with respect to

each terrain type. These individual sets of regression

weights were considered to be a pilot-subject's worth

function with respect to the given terrain type.

R

2

values associated with these weight sets were

relatively high. For mountainous terrain, 78% ofthe

150 equations had R

2

values equal to or greater than

.70. For the non-mountainous and over water terrain,

60% ofthe equations had R

2

values equal to or greater

than .70. These R

2

values indicate the degree to which

pilots' comfort ratings are explained by their worth

functions and scenario benchmark values.

The 150 sets of regression weights associated with

each terrain type were then subjected to hierarchical

clustering (HIER-GRP) to identify groups of weight

sets differing significantly with respect to worth function emphasis. Within both mountainous and nonmountainous terrain types, foursignificantly different

policy groups were detected. Within the over water

terrain type, two significantly different policy groups

were detected. Tables 4, 5, and 6 display, for the nonmountainous, mountainous, and over water scenarios,

respectively, the worth functions forsignificant policy

groups by showing the average standard regression

weights which were greater than or equal to .25. The

average R

2

, inter-rater agreement (Ru), policy stability (Rkk), and average comfort level (CF) are also

reported for each group and terrain.

Tables 7, 8, and 9 contain the background data for

pilots in each subject identified policy group for the

non-mountainous, mountainous, and overwater scenarios, respectively. In addition, Appendix E lists the

results of a series of pair comparisons of demographic

and comfort level values for policy groups in each of

the three scenarios. Since the purpose of these pair

comparisons was to assist in developing an understanding ofthe critical differences among these groups,

only those comparisons that were found to be statistically significant are listed.

Table 4. Non-mountainous Group Average Weights.

GP1 GP2 GP3 GP4

A (Ceiling) — — — —

B (Visibility) — — — —

C (Precipitation) — — 0.51 —

AB (Ceiling/Visibililty) 1.5 1.5 — 0.34 —

AC (Ceiling/Precipitation) 0.8

BC (Visibility/Precipitation) —

ABC (Ceiling/Visibility/Precipitation) — 2.1

0.25

0.37

Average R

Rater Agreement, R

Policy Stability, R

11

Average Comfort tevel

Number of subjects

0.79 0.77 0.66 0.58

0.57 0.53 0.40 0.35

0.98 0.98 0.95 0.95

32.0 28.0 50.0 17.0

34.0 52.0 27.0 37.0
Table 5. Mountainous Group Average Weights.

GP1 GP2 GP3 GP4

A (Ceiling) — — — —

B (Visibility) — — — —

C (Precipitation) — — — —

AB (CeilingA/isibililty) — — 0.51 —

AC (Ceiling/Precipitation) — — 1.1 0.29

BC (Visibility/Precipitation) 0.91 — — —

ABC (CeilingA/isibility/Precipitation) 0.53 2.3 — 0.58

Average R 0.83 0.80 0.75 0.60

Rater Agreement, R 0.49 0.53 0.44 0.35

Policy Stability, R

11 0.93 0.99 0.96 0.95

Average Comfort tevel 36.0 29.0 46.0 17.0

Number of subjects 14.0 69.0 31.0 36.0

Table 6. Over Water Group Average Weights.

GP-A GP-B

A (Ceiling) — —

B (Visibility) — —

C (Precipitation) — —

AB (Ceiling/Visibility) — —

AC (Ceiling/Precipitation) — 0.98

BC (Visibility/Precipitation) — 0.45

ABC(CeilingA/isibility/Precipitation) 1.9 —

Average R 0.69 0.67

Rater Agreement, R 0.43 0.35

Policy Stability, R

11 0.99 0.95

Average Comfort tevel 27.0 40.0

Number of subjects 112.0 38.0
Table 7. Mean Demographic Variables and Comfort Levels for NonMountainous Scenario Groups.

Background Variable Non-mountainous Groups

GP1

S-N

GP2

J-N

GP3

J-B

GP4

S-C

Number of Subjects 34 52 27 37

Flying History

Total Hours

Last 90 Days

1942

51

1516

37

1869

47

1679

25

Average Age 44 41 38 46

Percent Instrument Rated 53% 38% 48% 42%

Percent Owning Aircraft 32% 44% 19% 41%

Percent Female 12% 17% 4% 14%

Mean Terrain Experience 3.50 3.65 3.41 3.49

Mean Precipitation Experience

Light Rain

Moderate Rain

Heavy Snow

2.62

1.79

.38

2.50

1.71

.33

2.41

1.63

.67

2.11

1.27

.38

Mean Visibility Experience

1 NM

4NM

8NM

1.62

2.74

3.38

1.79

2.92

3.77

1.85

3.00

3.70

1.38

2.24

3.30

Mean Ceiling Experience

800 ft

1800 ft

4000 ft

1.65

2.88

3.56

1.75

3.13

3.81

1.78

2.93

3.67

1.62

2.46

3.38

Percent Using Weather Source

FAA/FSS

TV/Radio

Other

None

74%

33%

0%

0%

66%

32%

12%

4%

56%

35%

7%

3%

68%

26%

13%

1%

Percent Filing Flight Plan* 58% 55% 45% 57%

Percent Certificated Held

PP

COM/ATP

CFI/CFII

53%

32%

21%

58%

37%

21%

56%

22%

26%

70%

24%

16%

Percent Flying

Job-employed as Pilot

Business/Work Related

Pleasure

21%

11%

61%

17%

7%

74%

23%

11%

60%

3%

6%

85%

Average Comfort Level 32 28 50 17

Average R

2

.79 .77 .66 .58

Letter choices were converted to a 0-to-100 range: a=100, b=75, c=50, d=25, e=0.

10
Table 8. Mean Demographic Variables and Comfort Levels for

Mountainous Scenario Groups.

Background Variable Mountainous Groups

GP1

S-N

GP2

J-N

GP3

J-B

GP4

S-C

Number of Subjects 14 69 31 36

Flying History

Total Hours

Last 90 Days

1969

47

1963

49

561

34

2140

21

Average Age 51 41 34 48

Percent Instrument Rated 51% 36% 50% 39%

Percent Owning Aircraft 43% 39% 16% 44%

Percent Female 7% 10% 6% 25%

Mean Terrain Experience 1.57 1.58 1.10 1.36

Mean Precipitation Experience

Light Rain

Moderate Rain

Heavy Snow

2.50

2.00

.36

2.59

1.81

.43

2.10

1.19

.19

2.31

1.42

.58

Mean Visibility Experience

1 NM

4NM

8NM

1.86

3.14

3.57

1.90

2.94

3.72

1.35

2.61

3.55

1.39

2.25

3.28

Mean Ceiling Experience

800 ft

1800 ft

4000 ft

2.43

3.50

3.86

1.84

3.10

3.77

1.19

2.52

3.58

1.58

2.50

3.28

Percent Using Weather Source

FAA/FSS

TV/Radio

Other

None

58%

24%

7%

2%

68%

32%

11%

1%

69%

28%

5%

3%

64%

35%

17%

3%

Percent Filing Flight Plan* 44% 55% 54% 56%

Percent Certificates Held

PP

COM/ATP

CFI/CFII

36%

36%

29%

59%

33%

22%

68%

16%

16%

61%

33%

19%

Percent Flying

Job-employed as Pilot

Business/Work Related

Pleasure

18%

12%

66%

22%

8%

62%

12%

9%

76%

5%

7%

86%

Average Comfort Level 36 29 46 17

Average R

2

.83 .80 .75 .60

* Letter choices were converted to a 0-to-100 range: a=100, b=75, c=50, d=25, e=0.

11
Table 9. Mean Demographic Variables and Comfort Levels

for over-Water Scenario Groups.

Background Variable Over-Water Groups

GP-A

T-N

GP-B

T-B

Number of Subjects 112 38

Flying History

Total Hours

Last 90 Days

1896

39

1187

39

Average Age 43 40

Percent Instrument Rated 46% 39%

Percent Owning Aircraft 39% 26%

Percent Female 12% 16%

Mean Terrain Experience 1.04 .89

Mean Precipitation Experience

Light Rain

Moderate Rain

Heavy Snow

2.49

1.71

.43

2.18

1.29

.37

Mean Visibility Experience

1 NM

4NM

8NM

1.74

2.80

3.62

1.42

2.50

3.37

Mean Ceiling Experience

800 ft

1800 ft

4000 ft

1.77

2.80

3.65

1.50

2.68

3.53

Percent Using Weather Source

FAA/FSS

TV/Radio

Other

None

67%

33%

12%

1%

65%

24%

7%

5%

Percent Filing Flight Plan* 56% 50%

Percent Certificates Held

PP

COM/ATP

CFI/CFII

58%

33%

21%

63%

21%

18%

Percent Flying

Job-employed as Pilot

Business/Work Related

Pleasure

14%

9%

72%

19%

6%

67%

Average Comfort Level 27 40

Average R

2

.69 .67

Letter choices were converted to a 0-to-100 range: a=100, b=75, c=50, d=25, e=0.

12
With respect to non-mountainous terrain (Table

4), four worth functions could be interpreted. The

most obvious ofthese were those associated with GP1

and GP2, which we have chosen to designate as the

Senior-Nominal Group and Junior-Nominal Group,

respectively, based upon an inspection ofthe significant differences among demographic characteristics

and comfort levels ofthe groups, as shown in Table 7

and Appendix E. The worth function associated with

GP1 seemed to be related to pilots who emphasized

the interactive relationship of ceiling and visibility in

their assessment of flight completion comfort. The

function associated with GP2 appeared to represent

pilots who interactively related ceiling, visibility, and

precipitation. The remaining functions for GP3 (Junior-Bold) and GP4 (Senior-Cautious) were less distinct (i.e., did not have standard weights greater or

equal to 1.0). All mountainous terrain worth functions (Table 5) appeared to involve the interrelationship of pairs ofweather attributes or all three weather

attributes. With respect to over-water terrain (Table

6), there were two relatively distinct functions. The

function for GP-A (Total-Nominal) was similar to the

previously cited GP2 (Junior-Nominal) functions.

Significant differences in background variable values

among non-mountainous, mountainous, and overwater groups are listed in Appendix E.

Within the compensatory model set, the additive

model summed the three benchmark factors, and

implied equal weights for the three factors. The multiplicative models (both two and three factor models)

cross-multiplied benchmark values, implied a policy

in which one or more factors may introduce a dampening effect which can overpower the level ofthe other

factors. The worst-factor cut-off models used the

lowest of the three benchmark values, and implied a

policy in which the poorest ofthe three factors became

the only focus. Within the noncompensatory set,

models representing policies associated with continuous single factors and single-factor cutoffs (ceiling,

visibility, or precipitation) were used.

Tablel 0 lists the compensatory and noncompensatory models used, and their respective correlations

with pilot-subject mean comfort ratings. It may be

seen that in almost all instances, the highest correlations (across terrain types) were always associated with

the compensatory models. Of these compensatory

models, the three-factor additive and multiplicative

models consistently showed the strongest relationships with assigned comfort ratings. Overall results

tend to support the contention that compensatory

strategies were widely used by pilot subjects.

DISCUSSION AND CONCLUSIONS

Noncompensatory Strategy Detection

To determine the extent to which noncompensatory

strategies were used by the subject-pilots in assigning

weatherscenario comfort ratings, the relationships (correlations) between comfort ratings and sets of compensatory and noncompensatorymodels were examined and

compared. Specifically, the standardized scenario benchmark ratings (which had been standardized to a mean of

5.0 and a standard deviation of 1.0 for all scenarios) were

firsttransformed (using sets oftransformations designed

to simulate compensatory and noncompensatory policies), and then correlated with pilot-subjects' scenario

mean comfortratingsfor each terrain type. The compensatory model set consisted of additive, multiplicative,

and worst-factor cut-offmodels. The noncompensatory

set consisted ofsets ofsingle-factor, continuous, and cutoffmodels (one set for each factor — ceiling, visibility,

and precipitation).

First, let us note that the results ofthis study must

be considered tentative in that the worth functions

were derived from a subject-pilot sample primarily

experienced in flying over non-mountainous terrain.

Because ofthe terrains used and the experience base of

the pilots sampled, this study may produce unstable

weights for the "water" and "mountain" terrains as

experience levels were very low. In particular, low

experience levels over water may account for the fact

that only two policies were detected. Additional studies (in other parts of the country using subjects with

higher levels of mountainous and over-water flying

experience) will be required to assess the stability of

the obtained worth functions. These studies will also

allow us to evaluate the degree to which the compensatory and noncompensatory models are used by pilot

in these settings.

13
Table 10. Comparison of Compensatory and Noncompensatory Models.

Models Used Obtained Correlation (r)

Non-mountainous Mountainous Water

Compensatory Models

A + B + C (Additive) .94 .93 .83

A * B * C (Multiplicative) .96 .96 .89

A * B (Multiplicative) .86 .84 .65

A * C (Multiplicative) .76 .82 .79

B * C (Multiplicative) .79 .86 .73

Worst Factor Models

Worst Factor (X> 5.0) .69 .71 .54

Worst Factor (X> 4.5) .80 .82 .78

Worst Factor (X> 4.0) .76 .79 .74

Noncompensatory Models

A .48 .53 .46

A(X>5.0) .44 .55 .46

A(X>4.5) .38 .42 .35

A(X>4.0) .38 .42 .35

B .67 .62 .48

B (X> 5.0) .56 .53 .41

B(X>4.5) .62 .46 .46

B(X>4.0) .62 .46 .46

C .60 .73 .62

C(X>5.0) .60 .67 .57

C(X>4.5) .53 .56 .50

C(X>4.0) .35 .55 .41

A = Ceiling

B = Visibility

C = Precipitation

14
Subject to that caveat, the obtained analysis results

were sufficiently clear to allow the identification (at

least tentatively) of distinctly different worth functions. Specifically, four different worth functions

were identified and characterized for non-mountainous and mountainous terrain types. Two different

worth functions were identified and characterized for

over-water terrain. Pilot use of weather information

was consistent with expert assessment ofthe safety risk

associated with VFR flight under varying ceiling,

visibility, and precipitation conditions over nonmountainous, mountainous, and water terrain. However, while use is consistent, pilots vary in their

expressed degree of comfort in flying over the different terrain under different weather conditions. These

differences in comfortlevel may occur, in part atleast,

because of:

• Differences ofunderstanding ofthe risks associated

with flying under differing above-minimum levels

of visibility, ceiling, and precipitation and how

terrain type impacts use ofsuch information.

• Differences in their self-assessment and perception

of their ability and/or skill in flying under the

varying conditions.

Training or other interventions addressing risk

assessment and self-perception would appear to be

appropriate. The use of training directed at the Five

Hazardous Thought Patterns is one method of addressing selfperception. Training directed at improving pilot understanding of the effects of terrain-type

on interaction ofmeteorological conditions is another

potential source. Increased emphasis on risk assessment and self perception training and exercises in

initial training and in subsequent seminars (FAAsponsored and otherwise) would seem to be warranted. Careful consideration must be given, however,

to the desired outcome of this training. The present

study has shown that many pilots use a compensatory

model for evaluating weather information, however

no data have been collected to demonstrate that the

use of a compensatory model is appropriate for all or

even the majority of pilots.

Consider the implications ofacompensatorymodel

in which a high value for one variable (say, visibility)

can compensate for a low value on another variable

(say, ceiling) in making the determination of the

suitability ofthe weather conditions for flight. From

the standpoint ofinformation processing, this maybe

an efficient use of information, however it may also

lead pilots to make poor decisions under some circumstances. It may be argued that in many situations

(for example, in mountain flying) a noncompensatory

model should be used. One such model might set a

minimum value for ceiling values (sufficient to clear

all mountain peaks, for example) which must be met,

regardless of the visibility. That is, in such a model,

having a very high visibility does not compensate for

having a low ceiling. Under those circumstances,

training inexperienced pilots to use the compensatory

model used by experts might be inappropriate and

could, in fact, lead to a decrease in safety. Indeed, the

dominance ofthe compensatorymodels held by centralTexas pilots might partiallyaccountfor accidents involving these pilots when they fly into mountainous areas.

Thus, the use ofacompensatorymodel, even though

it is the favored mode, may be hazardous under some

circumstances. Training might therefore be developed that would allow pilots to assess their personal

decision making model, and which would train them

in the use of non-compensatory models where such

models might reduce risk.

In summary, this study found:

•It is possible to capture reliably pilots' "comfort"

policies in terms ofemphasis upon terrain, ceiling,

visibility, and precipitation using a linear modeling approach.

• Pilots use compensatory judgment rules with sophisticated interactions.

•Pilots generally agree on how to prioritize the

safety implications ofterrain/weather conditions.

•Pilots' policies tend to fall cleanly into stable

policy groups although the individual pilots who

hold those views switch groups according to terrain.

• Pilots' policies vary based upon:

1. General comfort level

2. A mixture of age and general experience

3. Number of hours flown in last 90 days

4. Reasons for flying (employed as pilot versus

pleasure flying)

15
REFERENCES

Brecke, P. (1981). Instructional design for aircrew judgment training. Proceedings of the First Aviation

Psychology Symposium. Columbus, OH: Ohio State

University, 145-160.

Christal, R.E. (1974). The United StatesAir Force occupationalresearchproject. (AFHRL-TR-73-75, AD774 574). Lackland AFB, TX: Occupational Research Division, Air Force Human Resources Laboratory

Curry, R. (1976). Worth assessment of approach to

landing. Proceedings ofthe annual conference on

manual control. NASA report TX-73. University

ofIllinois, Urbana, IL: 585-591.

Einhorn, H.J. & Hogarth, R.J. (1981). Behavioral decision theory: processes ofjudgment and choice. In

Rosenzweig, M.R. & Porter, L.W. (Eds), Annual

review of psychology, Palo Alto, CA: 32, 53-88.

Fiathers, G.W., Giffin, W.C., & Rockwell, T. (1982) A

study of decision-making behavior of aircraft pilots deviating from a planned flight. Aviation,

space and environmental medicine, 53, 958-963.

Jensen, R.S (1982). Pilotjudgment: Training and evaluation. Humanfactors, 24, 61-73.

Jensen, R.S., Adrion, J., & Lawton, R. S. (1987), Aeronauticaldecision making forinstrumentpilots. DOT/

FAA/PM-86/43. U.S. Department ofTransportation, Federal Aviation Administration, Washington, D.C.

National Transportation Safety Board Safety Report,

(1989). General aviation accidents involving visual

flightrulesflightinto instrument meteorological conditions. NTSB/SR-89/01, National Transportation Safety Board, Bureau of Safety Programs:

Washington, D. C.

Payne, J.W. (1982). Contingent decision behavior. Psychological Bulletin, 92, 382-402.

Payne, J.W., Bettman, J.R., & Johnson, E.J. (1988).

Adaptive strategy selection in decision making.

Journal ofExperimental Psychology, 14, 534-552.

Payne, J.W., Bettman, J.R., & Johnson, E.J. (1992).

Behavioral decision research: A constructive processing perspective. In Rosenzweig, M.R. & Porter, L.W. (Eds), Annual review of psychology, Palo

Alto, CA: 43, 55-86.

Shanteau,J.,&Nagy, G.F. (1979). Probability ofacceptance in dating choice. Journal ofPersonality and

Social Psychology, 37, 522-533.

Staley, M.R., & Weissmuller, J.J (1981). InterraterReliability: The development ofan automated analysis

tool. (AFHRL-TP-81-42). BrooksAFB, TX: Technical Services Division, Air Force Human Resources Laboratory

16
APPENDIX A

DATA COLLECTION PACKAGE

Al
Rate your comfort level about proceding given the following weather

update (100 = The MOST comfortable; 0 = the LEAST comfortable):

ENROUTE: A weak cold front has stalled along a line from Sturgeon

Bay to Manistique. Ceilings are reported at 5000 ft with visibility of 1V* NM

and lightrain.

ROUTE 1

Card Number 801

Rate your comfort level about proceding given the following weather

update (100 = The MOST comfortable; 0 = the LEAST comfortable):

ENROUTE: A weak cold front has stalled along a line running west from

Amarillo into New Mexico.. Ceilings are reported at 4000 ft with visibility of 1

NM and light rain.

ROUTE 2

Card Number 601

Rate your comfort level about proceding given the following weather

update (100 = The MOST comfortable; 0 = the LEAST comfortable):

ENROUTE: A weak cold front has stalled along a running from Santa

Fe northeast into Colorado. Ceilings are reported at 600 ft with visibility of 5

NM and freezing rain.

ROUTE 3

Card Number 701

A3
Weather Combinations: Route 1— Delta County to City County.

Scenario Ceiling Visibility Precipitation

800 4000 7 moderate rain

801 5000 1.5 light rain

802 4000 4 none

803 5000 8+ none

804 2000 3 none

805 1800 3 heavy rain

806 1800 7 light snow

807 3500 1 heavy rain

808 900 1.5 moderate rain

809 1800 7 moderate rain

810 800 1.5 none

811 1600 8 none

812 1600 3 heavy snow

813 4000 5 light snow

814 900 0.5 heavy snow

815 1600 1.5 heavy rain

816 5000 0.5 moderate rain

817 600 8 moderate rain

818 600 7 none

819 1600 1.5 moderate snow

820 5000 8+ freezing rain

821 900 5 none

822 800 4 moderate rain

823 1600 4 moderate snow

824 1800 5 moderate rain

825 1800 0.5 none

826 900 4 heavy rain

A4
Weather Combinations: Route 2— Dalhart Airport to

Hereford Airport.

Scenario Ceiling Visibility Precipitation

600 5000 8 light snow

601 4000 1 light rain

602 1600 3 heavy snow

603 4000 3 light rain

604 2000 7 light snow

605 4000 8+ light rain

606 3500 0.5 heavy rain

607 1800 0.5 moderate snow

608 600 0.5 light snow

609 1800 4 none

610 800 8+ light snow

611 800 8 none

612 1800 5 moderate rain

613 600 8 moderate rain

614 900 4 heavy rain

615 5000 1.5 moderate rain

616 1800 7 moderate rain

617 900 5 light snow

618 900 0.5 heavy rain

619 2000 1.5 none

620 1800 3 heavy rain

621 2000 0.5 freezing rain

622 1600 4 moderate snow

623 800 3 none

624 900 1.5 light rain

625 2000 8+ light rain

626 4000 5 light snow

A5
Weather Combinations: Route 3— San Juan Pueblo Airport

to Las Vegas Airport.

Scenario Ceiling Visibility Precipitation

700 5000 5 none

701 600 5 freezing rain

702 3500 8 moderate rain

703 900 1 heavy snow

704 2000 0.5 moderate rain

705 900 4 heavy rain

706 4000 8+ none

707 600 3 moderate snow

708 1800 7 moderate rain

709 800 1 none

710 4000 0.5 heavy snow

711 600 1 moderate snow

712 4000 4 light snow

713 3500 1 light snow

714 1600 3 heavy rain

715 2000 8+ light rain

716 2000 1.5 none

717 1600 1 freezing rain

718 600 8 moderate rain

719 3500 3 freezing rain

720 2000 5 light snow

721 900 5 light rain

722 900 8 light rain

723 1600 3 heavy snow

724 2000 4 none

725 5000 1.5 light rain

726 1800 5 moderate rain

A6
A7
-fr i.

IUOWTMOA

#1

M I '' ^r I i i 1

4 735

HIO)

> ' * -Sr

]^jgfii%1

CMlwo J^-wjJ^nwW-K

51

AF-»*^

OAIMAHT^V il

5 4299t

i, P7>/

.1

f

i t\\ i ifi i I ' ' if* ' ' Vj

HF

4400

i*5**'

HAUT(DHI

,, '•%

"•^^-^

/'1

s^*2. *y CAMCO «ANCH

4999 - MI

.~>

US?"1«

J Hortl«y

• L • yt vil°

;

! v'TV'

i/c^--1' ^ ,<;

^

11* *

1MB»

gWW i i i

ROUTE 2

!f?

—^fe«IAC5i

Pl|»^^ „-TV ^JH ^

fi^^tR<Är? ^}.jrj- ^-Vy _fc (> »>P ^ .1» ;'

_ v... ..^"-i'Tt'll-^

M113

ptl)

. "4ese-.

MAG HEADING

DISTANCE

TIME (NO WIND1

343

70NM

44Min

' riurr

S i^l:u«T»_

2

4S7S

i? i^^Effi You are solo enroute to the Dalhart Airport (DHT) southwest ofDalhart, Texas

fH^Sjsi and plan to fly VFRfollowing the route shown on this sectional labeled

; 3«ji|| ROUTE 2. You have made a fuel stop atthe Hereford Airport (HRX) northeast of

*»BW|| Hereford, Texas. Your Cessna 172 is working perfectly exceptthatthe navigation

radio is broken and you do not have a transponder. It is 11:00 AM local time, you are

topped offwith avgas, are carrying one 8 pound suitcase, and receive the following

weather report:

+'

HEREFORDWEATHER: Clear, visibilty 10+NM, wind 240/10, temperature 48,

dewpoint 40, altimetersetting 29.90

ENROUTE WEATHER: See ENROUTE weatherscenario list.

DALHARTCOUNTYWEATHER: 9,000 overcast, visibility 5 NM, wind 350/10,

temperature 40, dewpoint 32, altimetersetting 30.01

-M^-

no*^SS^

A^

<?«

.; 4X01

.1

A8
MAG HEADING

15NM

11 Min

FBgBT

A9
APPENDIX B

PILOT INFORMATION FORM

Bl
Table B-1. Pilot Demographic Information. Table B-2. Flying Categories.

Count Percent

CERTIFICATES HELD:

Private Pilot 89 59%

Commercial Pilot 35 23%

Airline Transport Pilot 12 8%

Certified Flight Instructor 13 9 %

Certified Flight Instructor

Instrument 24 16%

INSTRUMENT RATED:

Yes 67 45%

No 83 55%

STATE OF RESIDENCE:

Colorado 1 1%

Maryland 1 1%

New Mexico 1 1%

Texas 144 96%

Wisconsin 1 1 %

Other 2 1%

AGE1:

16-20 — 10 7%

21-40 59 39%

41-50 28 19%

51 . 60 30 20%

61-70 18 12%

71 + 5 3%

SEX:

Female 19 13%

Male 130 87%

Blank 1 1%

ENGINE RATING:

Single 108 72%

Multi 37 25%

Other 5 3%

MEDICAL CERTIFICATE:

Class 1 -30 20%

Class 2 49 33%

Class 3 69 46%

EVER BEEN A MILITARY PILOT?

No 135 90%

Yes 15 10%

Count Percent

PERCENT OF FLYING AS:

PRIMARY JOB-EMPLOYED

AS A PILOT

0o/0 120 -

1 - 20% 2

21 - 40% 1

41 - 60% 3

61 - 80% 4

81 - 100% 20

-80%

— 1%

— 1%

— 2%

— 3%

-13%

PERCENT OF FLYING AS:

BUSINESS SUPPORTWORK RELATED

0o/0 ii6 770/0

1 - 20% 15 10%

21 - 40% 3 2%

41 - 60% 9 6%

61 - 80% 3 2%

81 - 100% 4 3%

PERCENT OF FLYING AS:

PLEASURE

0o/0 1 5 IOO/O

1 - 20% 20 13%

21 - 40% 5 3%

41 - 60% 7 - 5%

61 - 80% 6 4%

81 - 100% 9 7 65 %

Note 1 : Age Mean = 42.64; S.D. = 15.44

B3
Table B-3. Flight Plan Use. Table B-5. Flying History - Total Hours.

HOW OFTEN DO YOU FILE A

FLIGHT PLAN WITH THE FAA?

Count Percent

All of the time 22 15%

Most of the time 44 29%

Some of the time 43 29%

Seldom 23 15%

Never 9 6%

(Invalid/No response)— 9 6%

Table B-4. Weather information sources

PERCENT USE OF: FAA/FSS

Count Percent

0o/0 14 9<>/0

1 - 20% 16 11 %

21 - 30% 9 6%

41 - 60% 19 13%

61 - 80% 15 10%

81 - 100% 77 51%

PERCENT USE OF: TV/RADIO

0 % 4 6 31 %

1 - 20% 38 25%

21 - 40% 12 8%

41 . 60% 22 15%

61 - 80% 11 7%

81 - 100% 21 14%

PERCENT USE OF: OTHER

0o/0 HO 73%

1 - 20% 17 11 %

21 - 40% 6 4%

41 - 60% 5 3%

61 - 80% 6 4%

81 - 100% 6 4%

PERCENT USE OF: NONE

0o/0 . 139 930/0

1 - 20% 7 5%

21 - 40% 1 1%

41 - 60% 1 1%

61 - 80% 0 0%

81 - 100% 2 1%

TOTAL FLYING HOURS

Count

0 - 99 20 —

100 - 299 48 -

300 - 599 24 -

600 - 899 13 -

900 - 1199 7 -

1200 - 1499 5 —

1500 - 1799 4 -

Percent

— 13%

— 32%

— 16%

— 9%

— 5%

—- 3%

—- 3%

1800 - 2099 4 3%

2100 - 2399 1 —-- —- 1%

2400 - 2699 1 — 1 %

2700 - 2999 2 1 %

3000 - 30000 21 -14%

Mean: 1716.89

S.D. 3742.17

DAY HOURS

0 - 99 29 19%

100 - 299 44 29%

300 - 599 25 17%

600 - 899 9 6%

900 - 1199 7 5%

1200 - 1499 7 5%

1500 - 1799 4 3%

1800 - 2099 1 1%

2100 - 2399 1 1%

2400 - 2699 2 1 %

2700 - 2999 2 1 %

3000 - 30000 19 13%

Mean: 1418.14

S.D. 3036.35

NIGHT HOURS

0 4

1 . 19 53

20 - 49 31

50 - 99 16

- 3%

-35%

-21%

-11%

100 - 299 24 16%

300 - 599 7 5%

600 - 899 3 2%

900 - 1199 3 2%

1200 - 1499 2 1%

1500 - 15000 7 5%

Mean: 293.01

S.D. 1054.43

B4
Table B-5. (Continued)

ACTUAL INSTRUMENT HOURS

0 47 31%

1 . 19 49 330/0

20 - 49 12 8%

50 - 99 14 9%

100 - 299 12 8%

300 - 599 4 3%

600 - 899 2 1 %

900 - 1199 3 2%

1200 - 1499 2 1%

1500 - 15000 5 3%

Mean: 220.09

S.D. 995.52

Table B-6. Flying Experience in the Last

90 Days.

ACTUAL PLUS HOOD HOURS

0 -24

1 . 19 38

20 - 49 22

50 -99 31

16 %

25%

15o/0

21%

100 - 299 16 11%

300 - 599 7 5%

600 - 899 2 1%

900 - 1199 0 0%

1200 - 1499 2 1%

1500 - 15000 8 5%

Mean: 276.26

S.D. 1181.99

NIGHT INSTRUMENT HOURS

0 72

1_19 6

20 - 49 13

50 - 99 3

100 - 299 10

300 - 599 2

600 - 899 1

900 - 1199 1

1200 - 1499 0

-48%

-31%

-- 9%

-- 2%

- 7%

-- 1%

- 1%

-- 1%

-- 0%

1500 - 15000 2 1%

Mean: 105.28

S.D. 767.16

TOTAL INSTRUMENT APPROACHES FLOWN

0 -60 -40%

1 . 24 16 11%

25 - 49 16 11%

50 - 99 16 11 %

100 - 299 22 15%

300 - 599 6 4%

600 - 899 — 1 1 %

900 - 9999 13 9%

Mean: 291.67

S.D. 964.28

TOTAL HOURS

Count

0 1 2 -

1 . 24 70 —

25 - 49 36 -

50 - 99 13 ...

100 - 299 18 -

300 - 9999 1 -

Mean: 39.45

S.D. 54.69

DAY HOURS

0 15

1 .24 74

25 - 49 34

50 - 99 12

100 - 299 14

300 - 9999 1

Mean: 36.95

S.D. 54.52

NIGHT HOURS

0 69

1 . 24 79

25 - 49 1

50 - 99 0

100 - 299 1

300 - 9999 0

Mean: 3.99

S.D. 9.41

ACTUAL INSTRUMENT HOURS

0 87

1 . 24 58

25 - 49 3

50 - 99 2

100 - 9999 0

Mean: 3.55

S.D. 9.39

Percent

—- 8%

— 47%

— 24%

— 9%

— 12%

— 1%

-10%

-49%

-23%

- 8%

- 9%

- 1%

-46%

-53%

- 1%

-- 0%

-- 1%

-- 0%

-58%

-39%

-- 2%

- 1%

-- 0%

B5
Table B-6. (Continued)

TOTAL INSTRUMENT HOURS

0 „81

1 . 24 62 —

25 - 49 6

50 - 99 1

100 - 9999 0

Mean: 4.94

S.D. 8.98

ACTUAL NIGHT INSTRUMENT HOURS

-54%

-41%

- 4%

~ 1%

- 0%

118

- 31

- 0

-- 1

- 0

-79%

-21%

- 0%

- 1%

- 0%

0

1 . 24

25 . 49

50 - 99

100 - 9999

Mean: 1.06

S.D. 6.61

NUMBER OF INSTRUMENT APPROACHES

FLOWN

0 81

1 . 24 54

25 - 49 11

50 - 99 3

100 - 299 1

300 - 9999 0

Mean: 7.02

S.D. 13.93

-54%

-36%

-- 7%

-- 2%

- 1%

- 0%

Table B-7. Characteristics of Aircraft

Most Frequently Flown.

NUMBER OF ENGINES

Count Percent

One Engine 136 91%

Two Engines 8 5%

(Other/No Response) —- 6 4%

IFR CAPABLE?

Yes 119 790/0

No—- 30 20%

(Other/No Response) —- 1 1 %

AUTO-PILOT EQUIPPED?

Yes 40 27 %

No 108 72%

(Other/No Response) — 2 1 %

WEATHER RADAR EQUIPPED?

Yes 14 9%

No 134 8 9 %

(Other/No Response) — 2 1 %

TRANSPONDER EQUIPPED?

Yes 145 97%

No 3 2%

(Other/No Response) -- 2 1 %

HOURS OF CRUISE CAPABLE

3 Hours 16 11 %

4 Hours 55 37%

5 Hours 48 32%

6 Hours 15 10%

7 Hours 8 5%

(Other/No Response)— 8 5%

DO YOU OWN THIS AIRCRAFT?

Yes 5 4 3 6 %

No 9 4 63 %

(Other/No Response) — 2 1 %

B6
Table B-8. Attribute Experience - Terrain.

FLOWN OVER LARGE BODY OF WATER

Count Percent

I have NEVER flown into these or similar conditions- 88 59%

I have flown into these conditions ONCE or TWICE 19 13%

I have flown into these conditions SEVERAL times (3-5) 15 10%

I have flown into these conditions MANY times (6-10) 11 7%

I have flown into these conditions NUMEROUS times (11+) 17 11%

FLOWN OVER NON-MOUNTAINOUS TERRAIN

I have NEVER flown into these or similar conditions 7 5 %

I have flown into these conditions ONCE or TWICE 3 2%

I have flown into these conditions SEVERAL times (3-5) 11 7%

I have flown into these conditions MANY times (6-10) 11 7%

I have flown into these conditions NUMEROUS times (11+) 118 79%

FLOWN OVER MOUNTAINOUS TERRAIN

I have NEVER flown into these or similar conditions 67 45%

I have flown into these conditions ONCE or TWICE 21 14%

I have flown into these conditions SEVERAL times (3-5) 22 15%

I have flown into these conditions MANY times (6-10) 11 7%

I have flown into these conditions NUMEROUS times (11+) 29 19%

Table B-9. Attribute experience - Precipitation

LIGHT RAIN

Count Percent

I have NEVER flown into these or similar conditions 3 2%

I have flown into these conditions ONCE or TWICE 39 26%

I have flown into these conditions SEVERAL times (3-5) 42 28%

I have flown into these conditions MANY times (6-10) 25 17%

I have flown into these conditions NUMEROUS times (11+) 41 27%

MODERATE RAIN

I have NEVER flown into these or similar conditions 34 23%

I have flown into these conditions ONCE or TWICE 51 34%

I have flown into these conditions SEVERAL times (3-5) 31 21%

I have flown into these conditions MANY times (6-10) 8 5%

I have flown into these conditions NUMEROUS times (11+) 26 17%

HEAVY SNOW

I have NEVER flown into these or similar conditions 122 81%

I have flown into these conditions ONCE or TWICE 11 7%

I have flown into these conditions SEVERAL times (3-5) 6 4%

I have flown into these conditions MANY times (6-10) 5 3%

I have flown into these conditions NUMEROUS times (11+) 6 4%

B7
Table B-10. Attribute Experience -Visibility.

ONE NAUTICAL MILE

Count Percent

I have NEVER flown into these or similar conditions 42 28%

I have flown into these conditions ONCE or TWICE 36 24%

I have flown into these conditions SEVERAL times (3-5) 32 21 %

I have flown into these conditions MANY times (6-10) 11 7%

I have flown into these conditions NUMEROUS times (11+) 29 19%

FOUR NAUTICAL MILES

I have NEVER flown into these or similar conditions— 12 8%

I have flown into these conditions ONCE or TWICE 19 13 %

I have flown into these conditions SEVERAL times (3-5) 24 16%

I have flown into these conditions MANY times (6-10) 38 25%

I have flown into these conditions NUMEROUS times (11+) 57 38%

EIGHT NAUTICAL MILES

I have NEVER flown into these or similar conditions 4 3%

I have flown into these conditions ONCE or TWICE 4 3 %

I have flown into these conditions SEVERAL times (3-5) 11 7%

I have flown into these conditions MANY times (6-10) 17 11%

I have flown into these conditions NUMEROUS times (11+) 114 76%

Table B-11. Attribute experience - Ceiling

800 FEET

Count Percent

I have NEVER flown into these or similar conditions 45 30%

I have flown into these conditions ONCE or TWICE 38 —25%

I have flown into these conditions SEVERAL times (3-5) 20 - 13%

I have flown into these conditions MANY times (6-10) 1 1 7%

I have flown into these conditions NUMEROUS times (11+) 36 24%

1800 FEET

I have NEVER flown into these or similar conditions 8 5%

I have flown into these conditions ONCE or TWICE 17 11 %

I have flown into these conditions SEVERAL times (3-5) 27 18%

I have flown into these conditions MANY times (6-10) 32 21%

I have flown into these conditions NUMEROUS times (11+) 66 44%

4000 FEET

I have NEVER flown into these or similar conditions 3 2%

I have flown into these conditions ONCE or TWICE 3 2%

I have flown into these conditions SEVERAL times (3-5) 8 5%

I have flown into these conditions MANY times (6-10) 20 13%

I have flown into these conditions NUMEROUS times (11+) 116 77%

B8
APPENDIX C

EXPERT PILOT SAFETY RATINGS

Cl
EXPERT PILOT SAFETY RATINGS

You have been selected as a Subject Matter Expert (SME) in the field of aviation

safety due to your personal flying experience and/or your FAA job experience.

Place yourself in this scenario: Kelly (a new pilot VFR-only qualified with around

500 flying hours) asks you, "How safe would it be for me to fly this route VFR?"

Use this scale from 0 to 100 to indicate your expert safety assessment where:

0 = Absolutely unsafe for Kelly to fly VFR.

100 = Totally safe for Kelly to fly VFR.

Note 1: Use your total experience (VFR and IMC) to evaluate the conditions on

the following pages. NONE of the choices shown may deserve a "0" for "Absolutely

unsafe" or a "100" for "Totally safe."

Use any numbers in the range from 0 to 100 to indicate your expert opinion, but

you MAY NOT use the same number more than once on one page. If you feel that two

choices are "Absolutely unsafe," rate one as "0" and the other as "1".

You and we know that evaluating conditions for determining the safety of any

flight involves interaction between all weather, aircraft, pilot and terrain variables. Yet

each of us has developed benchmarks for certain values of each variable on which we

make flight planning decisions. We ask you to consider each variable (i.e., TERRAIN,

PRECIPITATION, VISIBILITY and CEILING) separately and give us your opinion on the

relative safety of each value for VFR flying. Treat each page as a separate and

unrelated rating assignment.

Read each of the following pages in turn, and give us your opinion on the safety

of each value using the scale at the top of the page and using each number only once

on each page.

Note 2: Values are not necessarily presented in order of "worst-to-best" or "bestto-worst." The order is simply to help us ensure that all target values are covered.

Remember, you are evaluating each of the variables against the safety for a VFR

qualified pilot with around 500 hours of flying time to fly in a reliable single engine VFR

only aircraft.

Turn the page and begin.

C3
TERRAIN

Rate each route on how safe you think it would be for a VFR only pilot

with around 500 flying hours experience to encounter. Use the following scale

and each number no more than once:

0 = Absolutely unsafe.

100 = Totally safe.

ID Rating Destination

a. Route "1" (City County to Delta County, Ml)

b. Route "2" (Hereford to Dalhart, TX)

c. Route "3" (Las Vegas to San Juan Pueblo, NM)

C4
PRECIPITATION

Rate each item on how safe you think it would be for a VFR only pilot with

around 500 flying hours experience to encounter. Use the following scale and

each number no more than once:

0

100

ID

a.

= Absolutely unsafe.

= Totally safe.

Rating Description

No Precipitation

b. Light Rain

c. Moderate Rain

d. Heavv Rain

e. Light Snow

f. Moderate Snow

g- Heavy Snow

h. Freezing Rain

C5
VISIBILITY

Rate each item on how safe you think it would be for a VFR only pilot with

around 500 flying hours experience to encounter. Use the following scale and

each number no more than once:

0 = Absolutely unsafe.

100 = Totally safe.

ID Rating Description

a. 1/2 NM

b. 1 NM

c. 1 1/2 NM

d. 2NM

e. 2 1/2 NM

f. 3NM

a. 4NM

h. 5NM

I. 6NM

i- 7NM

k. 8NM

I. More than 8 NM

C6
CEILING

Rate each item on how safe you think it would be for a VFR only pilot with

around 500 flying hours experience to encounter. Use the following scale and

each number no more than once:

0 = Absolutely unsafe.

100 = Totally safe.

ID Rating Description

a. 600 ft.

b. 700 ft.

c. 800 ft.

d. 900 ft.

e. 1000 ft.

f. 1200 ft.

P. 1400 ft.

h. 1600 ft.

i. 1800 ft.

i- 2000 ft.

k. 2500 ft.

I. 3000 ft.

m. 3500 ft.

n. 4000 ft.

0. 5000 ft.

P. over 5000 ft

C7
Personal Experience Profile

Thank you for your expert ratings. Your data will be combined with data

from other experts to form the foundation for studying safety judgment of

General Aviation (GA) pilots.

Now, please fill out the PILOT INFORMATION and four EXPERIENCE

Forms which ask for your personal flying history. These forms are intended only

for use in this study of GA pilots. Notice neither your name nor any personal

identifying information is requested.

Please feel free to write suggestions or comments on these sheets

regarding problems we may expect within the GA community with these forms.

The PILOT INFORMATION sheet requests personal flying history

information which may be important in explaining the decision making process

used by different subgroups within the GA pilot community.

The ATTRIBUTE EXPERIENCE forms appear similar to the four rating

sheets you just completed. Instead of rating "HOW SAFE?" you are asked to

indicate how much experience you have had under specific conditions. Because

these forms are designed for the GA pilot community, you, as a senior expert,

may well be responding level "5" to all these items - and that is OK.

Here is the scale used on all four Attribute Experience forms:

0 = I have NEVER flown into these or highly similar conditions.

1 = I have flow into these conditions ONCE or TWICE

2 = (same as above)

3 = I have flown into these conditions SEVERAL times.

4 = (same as above)

5 = I have flown into these conditions MANY times.

COMMENTS and/or SUGGESTIONS

C8
EXPERIENCE: TERRAIN

How much personal expeience (VFR and IMC) do you have with these

conditions:

0 = I have NEVER flown into these or highly similar conditions.

1 = I have flow into these conditions ONCE or TWICE

2 = (same as above)

3 = I have flown into these conditions SEVERAL times.

4 = (same as above)

5 = I have flown into these conditions MANY times.

ID Exper. Destination

a. Route "1" (City County to Delta County, Ml)

b. Route "2" (Hereford to Dalhart, TX)

c Route "3" (Las Vegas to San Juan Pueblo, NM)

C9
EXPERIENCE: PRECIPITATION

How much personal expeience (VFR and IMC) do you have with these

conditions:

0 = I have NEVER flown into these or highly similar conditions.

1 = I have flow into these conditions ONCE or TWICE

2 = (same as above)

3 = I have flown into these conditions SEVERAL times.

4 = (same as above)

5 = I have flown into these conditions MANY times.

ID Exper.

a.

Description

No Precipitation

b. Light Rain

c. Moderate Rain

d. Heavy Rain

e. Light Snow

f. Moderate Snow

q. Heavy Snow

h. Freezing Rain

CIO
EXPERIENCE: VISIBILITY

How much personal expeience (VFR and IMC) do you have with these

conditions:

0 = I have NEVER flown into these or highly similar conditions.

1 = I have flow into these conditions ONCE or TWICE

2 = (same as above)

3 = I have flown into these conditions SEVERAL times.

4 = (same as above)

5 = I have flown into these conditions MANY times.

ID Exper. Description

a. 1/2 NM

b. 1 NM

c. 1 1/2 NM

d. 2NM

e. 2 1/2 NM

f. 3NM

Q. 4NM

h. SIMM

I. 6NM

i. 7NM

k. 8NM

I. More than 8 NM

Cll
EXPERIENCE: CEILING

How much personal expeience (VFR and IMC) do you have with these

conditions:

0 = I have NEVER flown into these or highly similar conditions.

1 = I have flow into these conditions ONCE or TWICE

2 = (same as above)

3 = I have flown into these conditions SEVERAL times.

4 = (same as above)

5 = I have flown into these conditions MANY times.

ID Exper. Description

a. 600 ft.

b. 700 ft.

c. 800 ft.

d. 900 ft.

e. 1000 ft.

f. 1200 ft.

g. 1400 ft.

h. 1600 ft.

i. 1800 ft.

i- 2000 ft.

k. 2500 ft.

I. 3000 ft.

m. 3500 ft.

n. 4000 ft.

0. 5000 ft.

P- over 5000 ft

C12
APPENDIX D

PLOTS OF EXPERT PILOT SAFETY RATINGS

Dl
Safety Ratings of Precipitation by Expert Pilots

None

Light Rain

Light Snow

Moderate Rain

Moderate Snow

Heavy Rain

Heavy Snow

Freezing Rain Wi3.58J

3r§?5-

-*ee=i

4,l%&-

5.206

h5t4-

6TüÖÖ-

-frf*.5

3.5 4 4.5 5 5.5 6 6.5 7

Safety Rating

Safety Ratings of Terrain by Expert Pilots

Flat Terrain

Over Water

Mountainous

^^^

/

/

AX »61

1

3.98

3.6 4 4.4 4.8 5.2 5.6 6

Safety Rating

U.358

6.4

D3
Safety Ratings of Visibility by Expert Pilots

>8NM ^ 6.

8NM /"^6.41

7NM ^V65

6NM ^ ^5. 806

5NM ^M^\ 446

4NM ^ ^939

3NM ^ "4 .639

2 1/2 NM ^ 253

2NM K4.O6 7

1 1/2 NM f 396

1 NM + 3.896

1/2 NM *3 82?

3.6

598

4.4 4.8 5.2 5.6 6

Safety Rating

6.4 6.8

Safety Ratings of Ceiling by Expert Pilots

>5000 < Rfi

5000 - R41P

4000 ^fi?44

3500 <* 6 DP 1

3000 /* 5 999

2500 /i^5R3

2000 ^-*^5 3Q3

1800 .^ ""fi m

1600 ^ 4 709

1400 Jf 4 5 19

1200 JT 4 9ft

1000 ^ 4 1 19

900 w 3 99ft

800 * a R1ft

700 * V'R3

600 * afifi 4

3.6 4.4 4.8 5.2 5.6 6

Safety Rating

For all ratings, Mean = 5.00; Standard Deviation = 1.00

D4

6.4 6.8
APPENDIX E

SELECTED PAIR COMPARISONS AMONG

POLICY GROUPS FOR EACH SCENARIO

(All mean differences statistically significant for p < .05)

El
Highlights of Characteristics for Mountainous Terrain Groups:

GP1 S-N:

S-N J-N

51 41

36% 59%

S-N J-B

51 34

2.43 1.19

3.50 2.52

36% 68%

2.00 1.19

43% 16%

1969 561

S-N S-C

.83 .60

Al 21

3.50 2.50

36 17

3.14 2.25

3.86 3.28

66% 86%

36% 61%

2.43 1.58

GP2 J-N:

J-N S-N

41 51

59% 36%

J-N J-B

29 46

41 34

39% 16%

3.10 2.52

10% 6%

1963 561

1.81 1.19

1.84 1.19

2.59 2.10

1.90 1.35

33% 16%

.80 .75

Senior-Nominal Policy Group: (Mountainous)

Compared to the Junior-Nominal group, this Senior-Nominal group has:

Higher average Age

Lower percentage report PP as highest certification

Compared to the Junior-Bold group, this Senior-Nominal group has:

Higher average Age

Higher experience with Ceiling 800 FT

Higher experience with Ceiling 1800 FT

Lower percentage report PP as highest certification

Higher experience with Precip Moderate Rain

Higher percentage own aircraft

Higher average Total Hours Flown

Compared to the Senior-Cautious group, this Senior-Nominal group has:

Higher average R-Squared (Use of Expert Info)

Higher average Hours Flown Last 90 days

Higher experience with Ceiling 1800 FT

Higher average Comfort Level

Higher experience with Visibility 4 NM

Higher experience with Ceiling 4000 FT

Lower percentage fly for Pleasure

Lower percentage report PP as highest certification

Higher experience with Ceiling 800 FT

Junior-Nominal Policy Group: (Mountainous)

Compared to the Senior-Nominal group, this Junior-Nominal group has:

Lower average Age

Higher percentage report PP as highest certification

Compared to the Junior-Bold group, this Junior-Nominal group has:

Lower average Comfort Level

Higher average Age

Higher percentage own aircraft

Higher experience with Ceiling 1800 FT

Higher percentage Female

Higher average Total Hours Flown

Higher experience with Precip Moderate Rain

Higher experience with Ceiling 800 FT

Higher experience with Precip Light Rain

Higher experience with Visibility 1 NM

Higher percentage report COM/ATP as highest cert.

Higher average R-Squared (Use of Expert Info)

E3
J-N S-C

29 17

.80 .60

62% 86%

3.72 3.28

3.77 3.28

22% 5%

49 21

2.94 2.25

3.10 2.50

41 48

1.90 1.39

GP3 J-B:

J-B S-N

34 47

1.19 2.43

2.52 3.50

16% 36%

68% 36%

1.19 2.00

16% 43%

561 1969

J-B J-N

46 29

16% 39%

34 41

2.52 3.10

561 1963

1.19 1.81

1.19 1.84

2.10 2.59

1.35 1.90

16% 33%

.75 .80

J-B S-C

46 17

34 48

16% 44%

.75 .60

6% 25%

5% 17%

16% 33%

561 2140

Compared to the Senior-Cautious group, this Junior-Nominal group has:

Higher average Comfort Level

Higher average R-Squared (Use of Expert Info)

Lower percentage flying for Pleasure

Higher experience with Visibility 8 NM

Higher experience with Ceiling 4000 FT

Higher percentage Primary Job as Pilot

Higher average Hours Flown Last 90 days

Higher experience with Visibility 4 NM

Higher experience with Ceiling 1800 FT

Lower average Age

Higher experience with Visibility 1 NM

Junior-Bold Policy Group: (Mountainous)

Compared to the Senior-Nominal group, this Junior-Bold group has:

Lower average Age

Lower experience with Ceiling 800 FT

Lower experience with Ceiling 1800 FT

Lower percentage report COM/ATP as highest cert.

Higher percentage report PP as highest certification

Lower experience with Precip Moderate Rain

Lower percentage Own Aircraft

Lower average Total Flying Hours

Compared to the Junior-Nominal group, this Junior-Bold group has:

Higher average Comfort Level

Lower percentage Own Aircraft

Lower average Age

Lower experience with Ceiling 1800 FT

Lower average Total Flying Hours

Lower experience with Precip Moderate Rain

Lower experience with Ceiling 800 FT

Lower experience with Precip Light Rain

Lower experience with Visibility 1 NM

Lower percentage report COM/ATP as highest cert.

Lower average R-Squared (Use of Expert Info)

Compared to the Senior-Cautious group, this Junior-Bold group has:

Higher average Comfort Level

Lower average Age

Lower percentage Own Aircraft

Higher average R-Squared (Use of Expert Info)

Lower percentage Female

Lower percentage use Weather Source: Other

Lower percentage report COM/ATP as highest cert.

Lower average Total Flying Hours

E4
Senior-Cautious Policy Group: (Mountainous)

Compared to the Senior-Nominal group, this Senior-Cautious group has:

Lower average R-Squared (Use of Expert Info)

Lower average Hours Flown Last 90 days

Lower experience with Ceiling 1800 FT

Lower average Comfort Level

Lower experience with Visibility 4 NM

Lower experience with Ceiling 4000 FT

Higher percentage flying for Pleasure

Higher percentage report PP as highest certification

Lower experience with Ceiling 800 FT

Compared to the Junior-Nominal group, this Senior-Cautious group has:

Lower average Comfort Level

Lower average R-Squared (Use of Expert Info)

Higher percentage flying for Pleasure

Lower experience with Visibility 8 NM

Lower experience with Ceiling 4000 FT

Lower percentage Primary Job as Pilot

Lower average Hours Flown Last 90 days

Lower experience with Visibility 4 NM

Lower experience with Ceiling 1800 FT

Higher percentage Female

Higher average Age

Lower experience with Visibility 1 NM

Compared to the Junior-Bold group, this Senior-Cautious group has:

Lower average Comfort Level

Higher average Age

Higher percentage Own Aircraft

Lower average R-Squared (Use of Expert Info)

Higher percentage Female

Higher percentage use Weather Source: Other

Higher percentage report COM/ATP as highest cert.

Higher average Total Flying Hours

Highlights of Characteristics for Over-Water Terrain Groups:

GP-A T-N: Total-Nominal Policy Group: (Over Water)

T-N T-B Compared to the T-B group, this T-N group has:

27 40 Lower average Comfort Level

1% 5% Lower percentage use Weather Source: None

1.71 1.29 Higher experience with Precip Moderate Rain

GP-B T-B: Total-Bold Policy Group: (Over Water)

T-B T-N Compared to the T-N group, this T-B group has:

40 27 Higher average Comfort Level

5% 1% Higher percentage use Weather Source: None

1.29 1.71 Lower experience with Precip Moderate Rain

E5

GP4 S-C:

S-C S-N

.60 .83

21 47

2.50 3.50

17 36

2.25 3.14

3.28 3.86

86% 66%

61% 36%

1.58 2.43

S-C J-N

17 29

.60 .80

86% 62%

3.28 3.72

3.28 3.77

5% 22%

21 49

2.25 2.94

2.50 3.10

25% 10%

48 41

1.39 1.90

S-C J-B

17 46

48 34

44% 16%

.60 .75

25% 6%

17% 5%

33% 16%

2140 561
APPENDIX F

COMPREHENSIVE OCCUPATIONAL

DATA ANALYSIS PROGRAM (CODAP)

Fl
Comprehensive Occupational Data Analysis Program (CODAP)

CODAP is a suite ofsoftware products developed by the US Air Force for the analysis oflarge bodies

ofoccupational data, commonly obtained from job analyses and otherstudies ofelements ofthe Air Force

personnel system. In addition to its use by theAir Force, CODAP has also been adopted for use by foreign

and domestic governmental units, including the Australian government and the state ofMaryland. One

use ofCODAP which is particularly relevant to the currentstudy is its application in studies ofpromotion

boards. The linear modeling capabilities of CODAP components allow for the identification of the

worth functions used by promotion boards in the evaluation ofindividuals for promotion. This process,

which is often termed "policy-capturing" allows for the explication ofthe relative weighting which the

board and the constituent members have applied to the available demographic, training, performance,

and other data which are provided for each individual eligible for promotion in arriving at their global

promote versus do-not-promote decision. In some ways this is highly similar to the decision process

undertaken by pilots in evaluating the many elements ofweather information prior to making their go

versus no-go decision. A similar technique (conjoint analysis) has been used in previous studies of aviator

decision making (Flathers, Giffin, and Rockwell; 1982).

GRPREL: (pronounced "Group" "Rel") A standard interrater reliability program in the CODAP

system. For a given list of items rated by a set of Subject Matter Experts (SMEs), this program reports

two measures ofinterrater agreement (Rn and RJ. The Rn value indicates the reliabilitly ofthe observed

set ofratings—while 0.10 is considered a minimum for usable rater agreement, a value of0.20 or greater

is desired. The Revalue is driven by the number ofraters acutally used. Although an R kk of0.90 is usually

desired, it may not be practical in a particular study because ofa small number ofraters (SMEs) that may

be subdivided even further into smaller groups based on policy differences. The GRPREL program also

computes means and standard deviations for each item in the list. Item-level reports are printed in three

orders: original sequence, ordered descending y mean value, and ordered descending on standard

deviation. GRPREL computes each rater's correlation with the full-group mean vector and uses a

probability evaluation to recommend the removal ofdeviant (non-cooperative or reversed scale) raters.

The program can automatically iterate and remove flagged raters until either a sufficient level of

agreement (Rkk = 0.90) is reached or no raters can be found with a probability (of deviant rating) above

0.95.

An overview ofthe CODAP system is provided by Christal (1974), and a complete description ofthe

interrater reliability components is given in Staley and Weissmuller (1981). The interested reader is

directed to those sources for a more complete technical description of these software analytic tools.

F3 £u.S. GOVERNMENT PRINTING OFFICE: l»7 . SS»47»/4M«I