VII. CONCLUSIONS AND DISCUSSION

This study sought to examine the decision-making
skills of typical general aviation pilots by comparing
their preferred solutions to a number of scenarios with
the solutions recommended by a panel of experts. It
was found that, overall, general aviation and expert
pilots agreed in their judgments of the appropriate
course of action in situations critical to flight safety.
That is, there was a high correlation between the
vector of mean recommended solutions provided by
the expert and the vector of mean solutions chosen by
the general aviation pilots.

Notwithstanding the high overall agreement be-
tween general aviation and expert pilots, the degree of
agreement by individual general aviation pilots with
the recommended solutions varied widely. This was
evidenced by the range of scores (6 to 44) obtained
when the scenario items were analyzed using a simple
right-wrong scoring system. That analysis showed
that, on average, the general aviation pilots selected
the recommended alternative for only about half of
the scenarios.

Two demographic factors seem to influence the
general aviation pilots’ judgments: age and quantity
of flying experience. Younger pilots tend to make
judgments involving higher risk than older pilots.
The same is true for more experienced pilots in terms
of total and recent flying experience. Because the
correlations are small and not all statistically signifi-
cant, we must be cautious in interpreting these results,
as the trend we see may prove, on further investiga-
tion, to simply be a spurious statistical result. How-
ever, because the implications are intriguing, and with
the preceeding caveat in mind, we will explore them
briefly. Recall that the data presented earlier (Table 4)
suggest that the expert panel, in their rank ordering of
the alternatives, generally assigned the #1 rank to the
safest (that is, least risky) of the four alternatives
given. Higher SDI scores are therefore indicative of
subjects who endorse the least-risky (most conserva-
tive) solution for each scenario.

The data suggest, however tentatively, that while
age is positively associated with increased conserva-
tism (less risky), flying experience is negatively associ-
ated with conservatism. That is, older pilots are more
conservative than younger pilots, but pilots with more
experience are less conservative than pilots with little
experience. Therefore, the pilots who are the least

risk-averse would be those who are both young and
who have higher levels of total and recent experience.
This interpretation is consistent with the findings of
two studies of weather information utilization
(Driskill, Weissmuller, Quebe, Hand, Dittmar, etal.,
1997; Driskill, Weissmuller, Quebe, Hand, et al.,
1997) in which pilots rated a number of combinations
of weather conditions in terms of their comfort level in
flying under such conditions. Hierarchical cluster analy-
ses indicated that the groups with the highest comfort
level (that is, least risk-averse) were the youngest.

Similar results were also obtained from a study of
risk factors for aircraft accidents in New Zealand
(O'Hare, Chalmers, & Bagnall, 1996), in which a
group of “high flyer” young, active pilots was over-
represented in accidents. However, Stokes, Kemper,
& Marsh (1992, pg 34) investigated decision-making
in simulated flights in the MIDIS and found “...no
evidence that better decision making automatically
comes with advancing years or the accumulation of
flight hours.” Since the methods and criteria used
among these studies and the many others (cf., Barnett,
1989; Jensen, 1995; Wickens et al., 1987) that have
evaluated the impact of age and experience on pilot
performance differ considerably from those used in
the present study, it is difficult to assess the compara-
bility of their results. Further, we must again note that
the correlations obtained here were quite small indeed
and, for the most part, represent only trends that may
not prove to be statistically reliable. Thus, this tenta-
tive finding must be treated with considerable caution
but may be worth exploring in future studies.

The main purpose of the study seems to have been
accomplished. That is, a written instrument was de-
veloped that may be used to evaluate the decision-
making skills of general aviation pilots. The SDI
derived from the instrument has adequate reliability
as measured by coefficient alpha, and an approxi-
mately normal distribution, centered around a mean
performance level ofapproximately 50% correct. These
are necessary (though certainly not sufficient) condi-
tions for an instrument that might be used to assess
general aviation pilot decision-making. Additional
studies will be required to evaluate the relationship of
the SDI to external criteria such as accident involve-
ment or involvement in potentially hazardous events
(for example, running low on fuel), along with addi-
tional psychometric properties of the SDI and other
potential indices derived from the instrument.


42. You are going to spend two hours in the traffic pattern. You plan to:

Rank Order

RHONK
wewu

beae

Alternative

a. Stay in the pattern at your home airport.

b. Go work where the traffic is least.

c. Go to a strange field.

d. Go toa challenging field (short, grass, etc.).

43. You arrive at the rental airplane where the attendant (whom you know is a commercial pilot) says it is gassed to the top,
preflighted and ready to start. He will stay and help until you leave the parking ramp. You decide to:

Rank Order

123
123

123
123

4
4

4
4

Alternative

a. Thank him, check the fuel tanks, oil , then climb in and start.

b. Scan the airplane for any obvious errors, climb in the plane, and check the fuel gages. If they show full,
begin your preflight at engine start.

. Take out your checklist and do a complete preflight.

d. Do a fast walk around especially checking the fuel tanks and caps, oil stick and all doors closed.

44. You are cruising at 2500 feet on a beautiful clear day 10 miles out enroute to the Planters County Airport with your best friend
then he/she asks “What do you do if the engine quits?” You decide to:

Rank Order Alternative

1234 a. Pull the mixture and show how the engine can be restarted.

1234 b. Pull on the carb heat, bring the throttle to idle and demonstrate a forced landing to a low approach.

1234 . Tell your friend about what you would do.

1234 d. Wait until you are over the uncontrolled airfield and demo a forced landing to a full stop.

‘Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance
Runway _| Avaitable

Planters County 3200x75 No. No. Yes Yes 0700-1800

45, You are planning a cross country which will require a fuel stop. In what order would you consider the following factors in

selecting the airport at which to stop?

Rank Order — Alternative

1234 a. The amenities (restrooms, food service, loaner car, etc.)

1234 'b. The pilot support facilities (FSS access, weather station, etc.)

1234 ¢. The size of the airport and its congestion ( those factors that make for slow fuel stops).
1234 d. The cost of the fuel or the method for payment.

B17



26, The weather is stuck in the summertime high mode with clear mornings, hazy afternoons, puffy clouds scattered at 5500 feet
AGL with visibility at 7 miles or more. When you go cross country in these weather conditions you usually decide to:

Rank Order Alternative

a. Don't file but fly airways.

b, File VER and stay off airways.

c, Don’t file and don’t use airways.

4. File VER on airways as much as possible,

NNNN

3
3
3
3

beer
pees

27. Take-off and en route weather are VFR with a dry line scheduled through your destination about your ETA. It may push some
thunderstorms ahead of it so your weather briefing ends with “VFR flight is not recommended.” There are several good alternate
airfields along the route of flight and beyond your destination. You decide to:

Rank Order Alternative

1234 a. Go without filing a flight plan.

1234 b. File VFR to an airport short of your destination, land and let any weather pass over.

1234 c. Delay your departure until the “VFR flight is not recommended” statement is removed from the forecast.

1234 d. File VFR to your destination.

Airport Runway 24hr Tower ARSA Lighted Telephone Maintenance
Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 hs

7753x150

28. You have been cruising at 8500 feet for four hours and are preparing to do a fairly steep en route descent directly onto short
final at the Regional Airport. Weather en route was a ~30 degrees but the warm sun made the flight most comfortable. Regional
weather is 54 degrees, dew point of 52 degrees and calm winds. You decide to:

Rank Order = Alternative

1234 a. Expedite the descent to give as much time on final approach as possible.
1234 b. Make sure the carb heat and window defrosters are on and set at high.
1234 c. Slow the descent rate to allow the airframe to warm before landing.
1234 d. Find a rag and keep it handy in case you have to clean windows.

29, The weather for departure and the first half of your four hour cross country was slightly better than marginal VFR. You made it
off and have leveled at cruise altitude in VFR conditions and are preparing to check in with Flight Watch. You suspect they will ask
you for a PIREP to check their forecast. You decide to:

Rank Order Alternative

1234 a. Calculate your drift, determine the winds, make note of the cloud cover and types, and note the OAT to be

teady when they ask.

b. Beg off - telling them you have your hands full and can’t take the time.

1234 ¢. Expect to give at least your position, cloud bases and tops, visibility and relate any deviations between what
you saw and what was forecast, .

1234 d. Prepare to either confirm the accuracy of their forecast, or tell of the observable differences.

N
o
-

B13


Flying Hours

TOTAL HOURS PERCENTAGE (%)
1-25 36
251-5 30
501-75: 10

751-1000
1001-1250
1251-1500
1501-1750
1751-2000
2001-2250
2251-2500
2501-2700
2701-3000
3001-3250
3251-3500

HOOPeY ANNUN

TOTAL WEATHER HOURS
0-49

os

ONKEHYUROO

HOURS OF INSTRUCTION
0-9 ..
10-19
20-29
30-39
40-49
50-59
60-69
70-79

150-15
160-169...
170-179..
180-189...

NE OUNNRANKRANWOROHON

C2

TOTAL NIGHT FLYING HOURS

100-124
125-149.
150-174.
175-199.
200-224.
Over 225

FLYING HOURS LAST 60 DAYS

NNNHE WRN DO

HORAK


obtained from 31 pilots. This group was comprised of
FAA staff personnel, including the present and former
managers of the Flight Standards District Office
(FSDO) in San Antonio; flight instructor personnel at
Embry-Riddle Aeronautical University, University of
North Dakota, Daniel Webster College, and Ohio
State University; and pilots attending a one-day FAA
sponsored safety seminar in San Antonio, Texas.

Demographics and experience of the pilots com-
prising the safety expert panel compare favorably with
the criteria set for panel membership and are reported
in Table 6 and compared with the sample of general
aviation pilots.

Expert Interrater Agreement and Policy Stability

The rank order, percentage of safety risk, and
alternative plausibility judgments of 31 experts were
entered for GRPREL analysis.

Rank Order, Interrater agreement (R,,) and stabil-
ity (R,,) of the expert rankings of alternative courses
of action are .44 and .93, respectively. Both values
exceed the criteria set for agreement and stability.
There was no evidence of a secondary policy.

Safety Risk, Interrater agreement (R,,) and stabil-
ity (R,,) of the expert rankings of alternative courses
of action are .45 and .94, respectively. These values
are almost identical to those for rank order.

The distribution of mean ratings is shown in Table
2. Percentage of safety risk means for the alternatives
are distributed from 2.08 (lowest risk) to 86.95 (high-
est risk). The grand mean of the rankings is 36.52,
with a standard deviation of 19.19.

Alternative Plausibility. In making their plausibil-
ity ratings, the experts were asked to estimate the
percentage of 500-hour pilots who might choose the
course of action represented in each alternative. The

Table 2. Distribution of Safety Risk
Ratings Across the 204 Alternatives

Frequency N Frequency N
91-100 0 41-50 34
81-90 2 31-40 34
71-80 6 21-300 31
61-70 18 11-20 39
51-60 26 01-10 14

Grand Mean = 36.52; Standard
Deviation = 19.19

R,, and R,, for these ratings were .22 and .84, respec-
tively, based on 30 raters. The grand mean of the
plausibility ratings for the 204 alternatives was 37.87
with a range from 9.09 (lowest plausibility) to 76.11.
The distribution of ratings is shown in Table 3.
Overall, the ratings appeared to be reasonable and
plausible.

Effect of “Why Fly” on Expert Ranking and
Rating

The “Why Fly” statement in each scenario was
intended to represent the motivational component of
aeronautical decision-making. If the statement had its
intended effect, first-choice alternatives for scenarios
should be expected to have higher risk ratings than
alternative courses of action — that is, the first-
ranked alternative would not be the alternative with
the lowest-rated safety risk. A Pearson product mo-
ment correlation of alternatives rankings and ratings,
however, was .808, indicating that the “Why Fly”
statements had little overall effect on the rankings and
ratings. Given this high correlation, any “Why Fly”
effect is most likely to be specific to a given flight
scenario.

The relationship between rankings and ratings is
depicted in Table 4. This table shows the number of
times the first-ranked alternative for a scenario corre-
sponded with the alternative with the lowest safety risk
rating. Note that the risk ratings for each alternative were
converted to rank order to make the relationships more
easily perceived. Several factors should be noted in the
Table 4 data. First, for 42 of the scenarios, the first-
ranked choice also had the lowest risk. Only nine sce-
narios had a higher risk alternative than the first-ranked
alternative. Of these nine, the first-ranked choice had the
second lowest risk percentage for four scenarios. The

Table 3. Distribution of Plausibility
Ratings Across the 204 Alternatives

Frequency N_ Frequency N
91-100 0 41-50 56
81-90 0 31-40 48
71-80 2 21-30 42
61-70 8 11-20 24
51-60 23 01-10 1

Grand Mean = 37.87; Standard
Deviation = 21.85






46. Three of your closest friends have bought you a choice ticket and are paying for you to rent this airplane and fly the four of you
the 180 miles up to the university in the morning for the “BIG” early afternoon football game, then back in the early evening.
Another friend will meet you at the college airport and drive all of you to the game and back. Departure weather was overcast 3000
ft ceiling with 5 miles and light haze with temperatures in the 60s. Pilots flying the same route reported enroute weather as
occasional 1500 ft ceilings with 3 miles visibility and scattered showers, The College Airport is clear with bright sunshine. Forty-
five miles from the College Airport you have descended to 1000 feet staying just below the ceilings and encounter rain dropping
visibility to under 3 miles. The terrain is flat farmland with no published obstacles above 250 ft tall. You decide to:

Rank Order Alternative

1234 a. Remain under the clouds, keep visual contact with the ground and scoot through.

1234 b. Doa 180 and return home.

1234 c. Divert to the Madison County Airport located at 7 o’clock 50 NM and wait for the worst weather to pass.

1234 d. Put it toa vote.

Airport Ruoway Tower ARSA Lighted Telephone Maintenance

Runway | Available

College Airport ‘5000x100 24 His No Yes Yes 24 hrs
4099x100

Madison County Airport 3800x75, None No Yes Yes None

47. You are balfway in a two hour late evening flight from the Regional Airport cruising at 4500 feet over a route with an MEA of
1500 feet. The weather has been clear as forecast when without any warning you find yourself in a cloud. You decide to:

Rank Order Alternative

a. Continue straight ahead for a while and see what happens.
b. Make a 180 degree level tum and get out.

c. Start a wings level shallow descent to get under it.

d. Start a wings level climb to get on top.

ven
wow
aan

48. You are packing your flight kit to goon a VER cross country trip home for the Christmas Holidays. In addition to the
sectional and flight plan, you usually include current editions of:

Rank Order Alternative

a. Take a full set of IFR charts and terminal plates for the section of the country in which you fly.
b. Take only the VFR sectional and flight plan.

¢. Plot what IFR information you think will be helpful on the sectional.

d. Always carry a full set of IFR charts and plates on a cross country.

eee
Newn
wows
bees

49, ‘When you get your weather briefing for a cross country flight requiring at least one fuel stop, which part of the forecast do you
consider the most critical:

Rank Order Alternative
a. The weather at the departure point.
b. En route weather to the fuel stop.

¢. The weather at the fue! stop.
d. Weather at the final destination.

RYNN
wee
aeee

B18


ACKNOWLEDGMENTS

An expression of appreciation is extended to Tom Hennessee and Owen Russel, present and former
Federal Aviation Administration Flight Safety Program Manager for the San Antonio Flight Safety
District Office; to Cynthia Gass of Airways Aviation; and to Stanley Finch of Berry Aviation. They gave
freely of their time to provide intensive review of the scenarios used in this research. Their advice

materially improved the quality of this study and is deeply appreciated.

iii


17. While en route you want to find out what is going on along the weather pattern you observe ahead. You decide to:

Rank Order

NNNN
ww wu
phase

Alternative

a. Call an airport tower below and ask.

b. Call FSS and ask.

c. Find the ATC frequency, call and ask them.

d. Identify an airplane ahead and ask for a PIREP.

18. You have been away for five days and are returning to the Justin County Airport to return the 172 to the friend who loaned it to
you and pick up your car. The weather is clear and cold as forecast and a white blanket covers the ground. When you near the
Justin County Airport, you notice the runway has not been cleared. You cannot tell how deep the snow is, but the county road is
fairly clear except for a small strip of snow down the middle. You decide to:

Rank Order = Alternative

1234 a. Divert to the Regional Airport and return the plane another day.

1234 b. Land, but hold the airplane off the runway until is in a full stall, and keep the nose wheel off the ground as
tong as possible.

1234 ¢. Make a normal landing, but don’t touch the brakes unless absolutely necessary.

1234 d. First, do a touch and go to see how deep the snow is keeping your airspeed up and the nose wheel off the
ground. If control is no problem, land.

Airport Runway 24br Tower | ARSA Lighted | Telephone | Maintenance

Runway Available
Regional Airport 8800x150 Yes ‘Yes Yes Yes 24 hrs
7753x150
Justin County Airport 3200x50 No No Yes Yes 0700-1800

19. You just checked in with approach on 124.9 after a long solo cross country before entering ARSA. Listening to traffic being
vectored, it becomes apparent the FedEx flights are all returning just ahead of you, and it could be 20 minutes before you land at the
Regional Airport where you rented this airplane. The problem is you have to urinate and can’t wait the 20 minutes plus taxi time.
Your trusty relief bottle is in the pouch behind the front passenger seat. You decide to:

Rank Order Alternative

1234 a. Continue to follow vectors, get out the bottle and use it.

1234 b. Tell approach of your problem and request landing priority.

1234 cs. Get clearance outside ARSA, find a safe area to loiter and use the bottle.

1234 d. Divert to the Justin County Airport which you overflew 16 NM back and land.

‘Airport ‘Runway 24br Tower | ARSA Lighted | Telephone | Maintenance

Runway Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 brs
7753x150

Justin County Airport 3200x50 No No Yes Yes 0700-1800

20. You have announced on CTAF and upon starting your turn to base you see another aircraft on a straight-in which will conflict.

‘You decide to:

Rank Order Alternative

1234 a. Continue on, flash your landing lights.

b. Doa level left 360 degree tum for spacing.
c. Turn right, exit the pattern and re-enter.
d. Extend your downwind to take spacing behind the straight-in.

Bll


Finally, we must note that although general avia-
tion pilots may demonstrate on paper that they have
the knowledge and perspective for deciding upon and
taking the safest course of action, there is no assurance
that in real-time situations, under the pressures and
motivations of the moment, that they will in fact
apply this knowledge appropriately. Indeed, accident
statistics (e.g., NTSB, 1989) suggest that they often
do not make the correct choice in these critical situa-
tions. An instrument capable of reliably detecting
individual differences in decision-making skills could
therefore have a substantial impact on aviation safety
by, for example, identifying those individuals most at
tisk for a decision-related accident or incident and
therefore most in need of remedial decision-making
training. Whether the current instrument has that
capability remains unknown, but this will be ad-
dressed in future research.

VH. REFERENCES

Albert, W.G., Phalen, W.J., Selander, D.M., Dittmar,
MJ., Tucker, D.L., Hand, D.K., Weissmuller,
JJ., & Rouse, I.F. (1994). Large-scale laboratory
test of occupational survey software and scaling
procedures. Proceedings of the 36th Annual Confer-
ence of the International Military Testing Associa-
tion (pp 241). Rotterdam, The Netherlands.

Barnett, B. J. (1989). Modeling information processing
components and structural knowledge representa-
tions in pilot judgment. Unpublished doctoral
dissertation, University of Illinois, Urbana-
Champaign.

Christal, R. E & Weissmuller, J. J. (1988) The job-task
inventory. In S. Gael (Ed.), The job analysis hand-
book for business, industry, and government (Vol 2).
New York: John Wiley.

Christal, R. E. & Weissmuller, J. J. (1976). New
CODAP programs for analyzing task factor infor-
mation. (AFHRL-TR-76-3, AD-A025 121).
Lackland AFB, TX: Occupational and Man-
power Research Division, Air Force Human
Resources Laboratory.

13

Driskill, W. E., Weissmuller, J. J., Quebe, J., Hand, D,
Dittmar, M., & Hunter, D. R. (1997). The use of
weather information in aeronautical decision-mak-
ing. DOT/FAA/AM-97/3. Washington, DC: Fed-
eral Aviation Administration, Office of Aviation
Medicine. Available from the National Technical
Information Center, Springfield, VA 22161. Or-
der #ADA323543.

Driskill, W. E., Weissmuller, J. J., Quebe, J., Hand, D,
& Hunter, D. R. (1997). The use of weather infor-
mation in aeronautical decision-making: 11 DOT/
FAA/AM-97/23. Washington, DC: Federal Avia-
tion Administration, Office of Aviation Medicine.

Goody, K. (1976). Comprehensive occupational data analy-
sis programs (CODAP): Use of REXALL to identify
divergent raters. (AFHRL-TR-76-82, AD-A025
847). Lackland AFB, TX: Occupation and Man-
power Research Division, Air Force Human Re-
sources Laboratory

Hunter, D. R. (1995). Airman research questionnaire: Meth-
odology and overall results. DOT/FAA/AM-95/27.
‘Washington, D.C: Federal Aviation Administration,
Office of Aviation Medicine. Available ftom the
National Technical Information Center, Springfield,
VA 22161. Order #ADA300583.

Jensen, R. S. (1982). Pilot judgment: Training and
evaluation. Human factors, 24, 61-73.

Jensen, R. S. (1995). Pilot judgment and crew resource
management. Aldershot, England: Ashgate Press.

National Transportation Safety Board Safety Report
(1989). General aviation accidents involving visual
Slight rules flight into instrument meteorological con-
ditions. NTSB/SR-89/01. Washington, DC: Na-
tional Transportation Safety Board, Bureau of
Safety Programs.

O’Hare, D., Chalmers, D., & Bagnall, P. (1996). A
preliminary study of risk factors for fatal and non-
fatal injuries in New Zealand aircraft accidents:
Final report to the Civil Aviation Authority of
New Zealand. Dunedin, New Zealand: Depart-
ment of Psychology and Injury Prevention Re-
search Unit, University of Otago.


GENERAL AVIATION PILOT STUDY

This data collection is covered under OMB Approval Number 2120-0587. The public reporting burden for this collection of information
‘atimated to average ope hour per response. Ifyou wish to tom the accuracy of the eatimate or make suggestions for reducing this
2:

‘ransportation
Federal Aviation Administration

B3


APPENDIX A

EXPERT PILOT PANEL DATA FORM

3. In the evening after an exhausting three day business meeting at a downtown hotel, you have loaded your rental airplane at the
Downtown Airport and prepare to file your VFR flight plan for the two hour flight home when you discover you left your only pair of
reading glasses in the meeting room back at the hotel. You have no problem seeing the panel gages, or distance vision, but can’t read a
map or chart. Weather is solid VFR and if you depart within the next 20 minutes you will arrive at your home airport before dark. You
decide to:

Rank Order Alternative ‘%Plausible %Risk
1234 a. Depart and fly home.
1234 b. Call the hotel, if they have your glasses go get them and fly home fate this
evening.
1234 ¢. Call the hotel, if they do not have your glasses, buy a cheap pair from an all

night drug store and fly home tonight.

1234 d.. Call the hotel, if they have your glasses, go get them, spend the night and fly
home in the morning.
‘Airport Runway 2br Tower | ARSA Lighted] Telephone | Maintenance
Runway _| Available
Regional Airport 8800x150 Yes Yes Yes Yes 24 brs
7753x150

4 ‘You are cruising at 4500 feet on top of a thin haze layer with the outside air temperature at 65 degrees. It has been twenty-five hours
since the engine was overhauled and the run-up check was well within limits. The engine slowly loses RPM with no indications of oil or
fuel problems. You suspect carburetor icing and pull on the carb heat. ‘The engine backfires, vibrates and loses RPM fast. You decide to:

Rank Order Alternative %Plausible %Risk

1234 & Pull out the mixture, stop the engine and check the fuel selector valve, mag
‘switch settings and declare an emergency.

1234 b. Push in the carb heat, keep the engine running and divert to the closest airfield

1234 ¢. Keep the carb heat on and see what happens.

1234 d. Push in the carb heat, keep the engine at idle, declare an emergency and ask for
advice.

Al


21, You are on short final at an uncontrolled airfield with one other airplane in the pattern and have not completed your checklist.
‘You decide to:

Rank Order Alternative

34 a. Check the flap setting and land.

34 b. Go around.

34 c. Check the mixture and land.

34 d. Keep your head out of the cockpit and land.

22, The early afternoon ramp temperature at the Regional Airport is already 94 degrees and the inside of the airplane is like an
oven. You are flying your mother up to your sister’s to be with her during surgery this evening. Your mother is afraid the hot
airplane will make her airsick, so would you please spend as little time on the ground in the heat as possible. You are parked on the
Aircraft Rental and Leasing ramp and see 10 aircraft lining up on the south taxiway for a runway 09 takeoff. Winds are 060/12.
You decide to:

Rank Order Alternative

1234 a. Start and follow the traffic to runway 09.

1234 b. Start and ask for a runway 35 takeoff.

1234 ¢. Start and request an intersection takeoff on runway 09.

1234 4. Delay going to the airplane until traffic has cleared.

Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance

Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 hrs

7753x150

23. You are at a rancher friend’s private airstrip and he asks you to fly him in his 172 to check on his cattle (his foot is in a cast),
‘Weather appears to be around 3000 feet overcast with widely scattered showers within eyesight. You decide to:

Rank Order Alternative

1234 a. Go with what you see.

1234 b. Telephone the FSS for a briefing.
1234 ¢. Call to the local radio station for the forecast.
1234 d. Ask the rancher what you should do.

24. Your BFR is coming due and you feel a little rusty. You decide to:

Rank Order Alternative

a. Look for an instructor who can both get you current and administer the BFR.
b. Hire an instructor to get ready but have a different instructor give the BFR.
c. Practice solo and then take the BFR.

d. Have a pilot friend work with you to get ready for the BFR.

REND

3
3
3
3

bhee

25. You have stopped for gas at a small airstrip and are loaded with cargo. You can only fuel to 30 gallons in the tanks ard keep
under the airplane's max gross weight. A 30 gallon load will just enable you to make it home with the required reserve without
another fuel stop. You have no calibrated dip stick and have a new attendant to pump the gas for you. You decide to:

Rank Order Alternative

a. Fill it using the gages to read 3/4 full.

bo, Fill it full then have the attendant drain off the difference between the tanks capacity and 30 gallons.
¢. Leave the problem entirely to the attendant.

. Use a calibrated stick the attendant has in the office that is from an earlier model 172

nen
we ae
Rhee

B12


Table 8. Scenario Keyed Responses

Scenario Keyed Scenario
No. Response No.
1 b 18
2 b 19
3 d 20
4 c 21
5 d 22
6 b 23
7 c 24
8 a 25
9 d 26
10 d 27
1 c 28
12 c 29
13 d 30
14 b 31
15 c 32
16 a 33
17 b 34

Note: The keyed response is that which was rank-ordered #1 by the expert panel.

Table 9. Item Statistics

Item P D tp Item P
No. No.

OCONODAON=
o
o
iy
nm
iS}
Ls)
o
s

D= Discrimination Index
Tpp = point-biserial correlation for keyed response

Keyed

Response

a

emvraMmovoacToToTaTAQn

Scenario
No.

Item
No.

Keyed
Response

39
08
37
44
26
51

97
43
28
02

27
85
39

aaovarctec00ccvacocAaS


compared with expert pilot safety risk judgments,
some pilots select courses of action having greater
safety risk than other actions that could be taken.

A flight mission or “Why Fly” statement was used
to establish the motivational dimension in each sce-
nario. These statements were based on the following
situational factors that could affect the way pilots
judged the courses of action available to them:

weather phenomena

mechanical malfunctions

biological crises (e. g., sick pilot or passenger)
social influences (e. g., passenger requests impacting
flight safety)

5. organizational (e. g., employer or Air Traffic Con-
trol requests for pilots to perform some action)

BEN

The remainder of this report presents the guidelines
and methods for developing realistic flight descriptions
and details about capturing expert pilot policy, collect-
ing data from the sample of general aviation pilots,
computing and comparing pilot safety risk indices, and
analyzing how pilot demographic data and motivating
conditions relate to pilot safety indices.

II. SCENARIO DEVELOPMENT

Because of the efficacy of scenarios for describing
flights simulating situations requiring pilot judgments
(Driskill et al., 1997), this study employed a scenario-
based approach. Each scenario would consist of a
succinct description of the situation at hand and four
plausible alternatives. Although some studies
(Wickens, et al, 1987; Stokes, Kemper, & Marsh,
1992) have suggested that the use ofa multiple-choice
format attenuates the differences between novices and
experts, the overall design of the present study which
relied upon self-administration of the data collection
instrument dictated its use. How scenarios were de-
veloped is described below.

Guidelines
Eight principal guidelines directed scenario devel-
opment:

1. Content of each scenario should be based on situa-
tions that have actually occurred in preparing for or
making a flight.

2. Descriptions should clearly require a decision criti-

cal to flight safety.

3. Each scenario should contain four plausible and
realistic alternative decision choices that pilots could
reasonably be expected to make. No choice should
be totally unsafe; nor should any be obvious or a
“book solution.”

4. Circumstances or events representing the motiva-
tional conditions (i. e., weather, mechanical, bio-
logical, sociological, and organizational) should be
either explicit or stated in such a way that they are
self-evident from the situation presented.

5. Plausibility and realism of situational descriptions
and associated decision choices should be verified
by expert pilots.

6. The four alternatives should contain flight safety
risk elements differing with respect to amount or
degree of risk.

7. Scenario content should eliminate effects of local
knowledge on the part of respondents.

8. Because the data collection instrument was to be
administered by mail, the amount of time required
for completion should not exceed one hour.

Sources of Scenario Development

The task scientist, who has more than 30 years
experience as a pilot, based development of the sce-
narios on four sources.

Previous FAA Research. Specification of initial
scenario topics was based on the results of a large
survey of American pilots (Hunter, 1995). A nation-
wide sample of pilots provided their own “hangar
flying stories” about dangerous situations, courses of
action, judgments made, and actions taken. Each
anecdote was accompanied by lessons learned. The
survey produced 135 anecdotal descriptions of lessons
learned, e. g., “Don’t let others do your preflight for
you;” and “Do not ever let your passenger talk you
into doing something your better judgment says don’t
do.” Frequently, there was more than one lesson
learned. In these cases, the primary and secondary
factors were identified as potential targets for scenario
development.

National Transportation Safety Board Records. A
review of NTSB summaries of accident and incident
reports was used to narrow the list of lessons learned.
The review identified accidents or incidents in which
the details matched items on the lessons learned list as
well as being representative of the five motivational
conditions. Later, details in the summaries were used
to develop scenario content.


The next four scenarios (11 - 14) are related and occur on the same flight:

11, You are at the College Airport to pick-up three passengers and their baggage and return them to the Regional Airport. Before
refueling you add up the weights and find with full fuel (40 Gallons) your load will be 40 pounds over the book's max gross weight.
Weather for the 3:00 PM return trip is forecast at 6000" scattered, visibility 10+ and the winds at 5000 feet cruising altitude will net
a 10 knot tailwind. Using the Operator's Manual fuel consumption rate and the tailwind you correctly calculate it will take 34
gallons of 100LL to land at Regional with exactly 30 minutes reserve. You will overfly the Justin County Airport and could land for
fuel as a backup. You decide to:

Rank Order Alternative

1234 a. Load 34 gallons and file a flight plan direct to the Regional Airport.

1234 ‘b. Load 40 gallons and file direct to the Regional Airport.

1234 cc. Load only 34 gallons and file to the Regional Airport with a stopover at Justin County Airport.

1234 d. Load 34 gallons, do not file and sec if the fuel consumption and tailwind hold and decide later what to do.

Airport Runway ‘24hr Tower ARSA Lighted Telephone Maintenance

Runway Available

Regional Airport 8800x150 Yes Yes Yes ‘Yes 24 hrs
7753x150

Justin County Airport 3200x50 No No Yes Yes 0700-1800

College Airport 5000x100 Yes No Yes Yes 24 hrs
4099x100.

12. You have taken-off from the College Airport and an en route weather check has a late afternoon thunderstorm approaching the
Regional Airport from the opposite side of town. It is slow moving and is expected to cross the Regional Airport shortly after your
ETA. You check and the fuel consumption and tailwind are holding. You have arrival fuel with a 30 minute reserve. You decide
to:

Rank Order Alternative

1234 a. Continue to the Regional Airport and speed up a bit.

1234 b. Land at the Justin County Airport, add fuel and continue to the Regional Airport circling northeast around
the thunderstorm.

1234 c. Land at the Justin County Airport and wait until the weather passes.

1234 d. Land at the Justin County Airport, add fuel and continue to the Regional Airport circling southwest around
the thunderstorm.

13, Your friends persuaded you to Jand at the Justin County Airport. You plan to fill each tank half full to keep the weight in the
utility category. The thunderstorm remains slow moving, is over the Regional Airport on a path to the Justin County Airport and is
growing in size and intensity. It is 6:00 PM, getting dark, the storm can be seen approaching and the attendant is leaving but will
give everyone a lift into Driskill City. You decide to:

Rank Order Alternative

a. Takeoff for the Regional Airport circling around the thunderstorm and coming in behind it.

b. Wait with the airplane until the weather passes, then fly into the Regional Airport.

¢. Leave the passengers and baggage and fly the airplane anywhere away from the path of the storm.

d. Leave the airplane and either get a room in Driskill City or call and have someone drive out from the Big
City and pick-up all of you.

eee
RENN
wowe
a eee

‘Airport Runway 24br Tower | ARSA Lighted | Telephone | Maintenance
Runway _| Available
Regional Airport 8800x150 Yes Yes Yes Yes hes
7753x150
Tustin County Airport 3200x50 No No Yes Yes (0700-1800

BO


1. You are flying an “Angel Flight” with a nurse and non-critical child patient to meet an ambulance at a downtown regional
airport. You filed VFR, it is 11:00 P.M. on a clear night when at 60 NM out you notice the ammeter reading zero and correctly
deduce the alternator has failed. Your best guess is that you have from 15 to 30 minutes of battery power remaining. You decide to:

Rank Order Alternative

1234 a. Declare an emergency, turn off all electrical systems except for 1 NAVCOM and transponder and continue
to the Regional Airport as planned.
1234 v, Declare an emergency and divert to the Planter’s County Airport which is clearly visible at 2 o'clock, 7
NM.
1234 ¢. Declare an emergency, turn off all electrical systems except for 1 NAVCOM, instrument panel lights,
intercom and transponder and divert to the Southside Business Airport which is 40 NM straight ahead.
1234 @ Declare an emergency, turn off all electrical systems except for 1 NAVCOM, instrument panel lights,
intercom and transponder and divert to Draper Air Force Base which is 10 o'clock at 32 NM.
‘Airport Runway ahr Tower | ARSA Lighted | Telephone | Maintenance
Runway Available
Regional 8800x150 Yes Yes Yes ‘Yes 24 hrs
7753x150
Planters County 3200x75 No No. Yes Yes 0700-1800
Southside Business 4835x100 Yes Yes Yes Yes 0700-1800
4129x100
‘Draper AFB 11500x300 Yes No Yes Yes ‘None

2. You are solo om a late night cross country cruising VFR at 9000 fect with two hours left to your destination when you become
very drowsy. You decide to:

Rank Order Alternative

a. Direct the cold air vent onto your face, sing, keep moving about, anything you can to keep awake.
b. Land at an airfield 8 miles ahead, get a motel room and call it a night.

¢. Descend and continue flying at a lower altitude.

d. Land at the airstrip ahead, walk around, then takeoff and continue.

RRR
Bweuwe
bbe

3. In the evening after an exhausting three day business meeting at a downtown hotel, you have loaded your rental airplane at the
Downtown Airport and prepare to file your VFR flight plan for the two hour flight home when you discover you left your only pair of
Teading glasses in the meeting room back at the hotel. You have no problem seeing the panel gages, or distance vision, but can’t
read a map or chart. Weather is solid VFR and if you depart within the next 20 minutes you will arrive at the Regional Airport
before dark. You decide to:

Rank Order Alternative

1234 a. Depart and fly home.

1234 b. Call the hotel, if they have your glasses go get them and fly home late this evening.

1234 ¢. Call the hotel, if they do not have your glasses, spend the night, have a pair expressed to you and fly home

tomorrow.

1234 d. Call the hotel, if they have your glasses, go get them, spend the night and fly home in the morning,

Airport Ruoway 24hr Tower ARSA Lighted Telephone Maintenance
Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 brs

7753x150.

B6


Il.

Ill.

IV.

VII.
Vu.

TABLE OF CONTENTS

Page
BACKGROUND

SCENARIO DEVELOPMENT

Guidelines... ssveeanesseee
Sources of Scenario Development.
Scenario Content

Scenario Structur

CAPTURING EXPERT PILOT POLICY

Formatting and Instructions for Expert Pilot Judgment Tasks
Safety Expert Criteria ..
Collection of Expert Data
Expert Interrater Agreement and Policy Stability
Effect of “Why Fly” on Expert Ranking and Ratin,

ADNUUAA RRNHKN

COLLECTING GENERAL AVIATION PILOT JUDGMENTS .....cccssssssessseessessseceseessseesseesssee 7
Data Collection Instrument ssssanecesnveseesseesessnses seessesenneee 7
Data Collection and General Aviation Sample Demographics and Experience 8
ANALYSIS OF SCENARIO ITEM RESPONSES ........... 8
Item Analysis and Score Generation .... 9

Analysis of sample demographic variables .
CONCLUSIONS AND DISCUSSION .oeccssssssssssessssssessssseesssssesssnesecssnneessneesssneessnecesssnessssecsssee 12.
REFERENCES ....ssssesssssssssssstessnscssnsssvecnssscnsscesssssssecsnscssvecsasesaseesseessuessnvessuscsaseesusessessaneesssessneessseesees 13

LIST OF FIGURES

Figure 1. Distribution of safety deviation index scores

LIST OF TABLES

Table 1. Mean ratings of reasons for taking greater flight risks......
Table 2. Distribution of safety risk ratings across the 204 alternatives
Table 3. Distribution of plausibility ratings across the 204 alternatives
Table 4. Comparison of expert first choices with risk ratings for 51 scenarios .
Table 5. Summary of expert rankings and safety risk ratings
Table 6. General aviation sample and expert panel demographics and flying experience comparison .
Table 7. Mean rank and percentage of safety risk differences between experts and sample pilots.
Table 8. Scenario keyed responses.
Table 9. Item statistics.....
Table 10. Safety deviation index statistics
Table 11. Correlations of the SDI and demographic variables .





Table 10. Safety Deviation Table 11. Correlations of the SDI and

Index statistics demographic variables

N of Items 51 Variable r p
N of Examinees 246 Age 1044 105
Mean 27.2 Total flying hours -.0796 218
Variance 36.0 Total weather flying hours -0991 151
S.D. 6.0 Total instrument flying hours -.1198  .071
Median 27 Total night flying hours -.0306 .642
Alpha 0.753 Flying hours - last 60 days -1121 086
Mean P 0.534 Weather flying hours - last 60 days 1417 033
Mean Biserial 0.371 Instrument flying hours -last60 days -.0628 = .345
Minimum Score 6 Night flying hours - last 60 days -.1509 .022
Maximum Score 44

Number of Pilots
20

15

10 F

6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44

SDI Score
Figure 1. Distribution of Safety Deviation Index scores.

ll


EVALUATING THE DECISION-MAKING SKILLS OF GENERAL AVIATION PILoTs

This report describes research comparing judg-
ments ofa sample of general aviation pilots (predomi-
nately Private Pilot certificate holders) and an expert
panel of pilots with respect to decisions critical to
flight safety. The objective was to develop a method-
ology to assess the decision-making skills of general
aviation pilots by using a written format. Current
techniques for assessing decision-making in pilots
typically rely upon the observation of behavior in
either a flight simulator or an actual aircraft — both
of which are both time consuming and expensive.
Hence, a methodology that could be used outside
those settings would have significant advantages both
for research and for training.

I. BACKGROUND

Based ona review of 361 general aviation accidents,
aNational Transportation Safety Board report (NTSB,
1989) concluded that 97% of the probable causes
were attributable to flight crew. An earlier review by
Jensen (1982) attributed as many as 85% to pilot
error, and in 51.6% of the fatal accidents, faulty
decision-making was cited as a probable cause.

Studies at the University of Illinois (cf., Barnett,
1989; Stokes, Kemper, & Marsh, 1992; Wickens et
al., 1987) have evaluated pilot decision making in a
simulated environment using MIDIS (Microcom-
puter based Decision Simulator). These studies have
generally demonstrated that there is little relationship
between pilot demographic characteristics (such as
experience levels) and performance on decision-mak-
ing tasks.

Two earlier studies (Driskill, Weissmuller, Quebe,
Hand, Dittmar, & Hunter, 1997; Driskill,
Weissmuller, Quebe, Hand, & Hunter, 1997) exam-
ined pilot decision-making through linear modeling
of pilot worth functions. Pilots were presented with
scenarios in which weather conditions (ceiling, vis-
ibility, and amount and type of precipitation) varied.
Pilots were required to assign “comfort levels” to each
scenario that indicated their relative risk assessments
of the scenarios. These studies indicated that, while

pilots tended to order scenarios involving flight under
varying favorable and unfavorable weather conditions
(e. g., unlimited ceiling and visibility and no precipi-
tation to very low ceiling and visibility and freezing
rain) in the same way, there were wide differences in
the comfort levels they assigned to flight under the
varying conditions. These results imply that some
pilots either do not know the safety risk implications
of flight, or some are more willing to incur safety risks
when they fly.

The studies also showed that expert pilots (sea-
soned instructors, for example) had consistent ratings
of weather conditions. This indicated that they shared
a common concept of what constituted “good” and
“bad” weather conditions. In the weather modeling
studies, this common concept was used to establish
baseline indices for the generation and selection of
scenarios. This suggests that these experienced pilots
may also share common concepts of “good” and “bad”
decisions in other aviation settings, and that those
common models could form a baseline against which
the decision-making of other pilots might be compared.

The present study, therefore, sought to examine the
degree to which the decisions of less experienced
pilots agreed with the decisions recommended by
expert pilots for a set of potentially hazardous in-fight
events presented in a written format. Two questions
were posed: (1) Do the judgments of less experienced
pilots (about 500 hours total flying hours) differ from
those of an expert panel in making decisions critical to
flight safety? (2) To what extent are demographic
factors associated with the decisions made? Although
previous studies (for example, Stokes, Kemper, 8
Marsh, 1992) have addressed these issues to some
extent, the present study is unique in the use of a
written format for the assessment of decision-making.

In the research described below, three assumptions
were made: (1) The affective or motivational compo-
nent of aeronautical decision-making (a “Why Fly”
factor) affects pilots’ willingness to accept higher
safety risk. (2) This willingness is reflected in how
they judge alternative courses of actions in flights
requiring decisions critical to flight safety. (3) When


APPENDIX B

GENERAL AVIATION PILOT STUDY MATERIALS

Bl


DOT/FAA/AM-98/7

Office of Aviation Medicine
Washington, D.C. 20591

Evaluating the Decision-
Making Skills of General
Aviation Pilots

Walter E. Driskill
Johnny J. Weissmuller
John C. Quebe
Darryl K. Hand

Metrica, Inc.
San Antonio, TX

David R. Hunter

Office of Aviation Medicine
Federal Aviation Administration

Washington, DC 20591

February 1998
Final Report

This document is available to the public
through the National Technical Information
Service, Springfield, Virginia 22161.

Q

U.S. Department
of Transportation

Federal Aviation
Administration




are easily categorized as high, midrange, and low.
Three reasons are in the high rating category, with
ratings ranging from 74.97 to 87.73; 10 have midrange
ratings from 12.10 to 47.10; and 12 have low ratings
of 0.30 to 2.31.

Subject Matter Expert Interviews. When draft
scenarios and alternatives based on the preceding
sources were developed, senior pilots completed a
field review of candidate scenarios. These senior pilots
provided an intensive, critical review of the scenarios
and alternatives. Participating in the field review were
present and former FAA Flight Safety Program Man-
agers and senior flight instructors in local aviation
schools. On the basis of their critical reviews, 51
scenarios were selected for use in the study.

Scenario Content

Aircraft, Since the baseline for expert pilot risk
assessments was a pilot with approximately 500 flying
hours, the most commonly flown aircraft used by
these pilots appeared to be the Cessna 172. Flight in
a rented aircraft of this type was the basis of each
scenario. The choice, however, restricted the me-
chanical and performance scenarios to fairly basic
malfunctions, when compared with some of the air-
craft incidents reviewed. To make clear to all respon-
dents (experts and national sample), a flier from an
airplane rental agency was included in the survey. The
flier described the airplane and onboard equipment.

Virtual Airspace. During the development stage, a
decision was made to recast scenarios from the FAA
survey and NTSB summary records into a virtual
airspace designed around six fictitious airports con-
taining the characteristics necessary to support sce-
nario alternatives. Several factors influenced this
decision. First, since the sources of the scenarios
involved incidents or accidents occurring across the
United States and several foreign nations, mainte-
nance of each scenario in its original setting would
have required extensive descriptions and possible sup-
porting charts and airfield diagrams. Second, the
effects of “local knowledge” should be eliminated. If
actual airfields were used, it is possible that some
respondents might be familiar with them and would
bring some level of terrain or airspace “local knowl-
edge” into the decision-making process.

Airport diagrams, patterned upon actual airport
directories, were constructed to support the scenarios.
The latitudes and longitudes were fictionalized and
set in the middle of the United States to minimize
references to similar airports. Included along with the
airfield diagrams was information about runway length
and lighting, indicating whether the tower operated
on a 24-hour schedule, availability of airport surveil-
lance radar, and listing telephone and maintenance
availability. Because such information is not normally
available to private pilots, a diagram of the fictitious
Air Force base was not included. To discourage re-
spondents from forming alternatives not in the four
provided for each scenario, a comprehensive map
showing all of the airfields was not included.

Scenario Format. Each scenario contained a stem
describing the circumstances of the flight and clearly
showing the need fora decision critical to flight safety.
The stem for biological, sociological, and organiza-
tional scenarios contained an explicit statement of the
mission of the flight. Results of the “Why Fly” survey
were used to balance motivating conditions used in
the stem. Motivation levels ranged from high (>75) to
very low (<12) risk acceptance.

The need for a critical decision was self-evident in
mechanical and weather scenarios, since these kinds
of scenarios do not require a direct statement. Pilots
at cruising altitude when their engine quits, for ex-
ample, do not have to consider the purpose of their
flight in order to initiate their decision-making pro-
cesses to resolve their problem.

Each scenario stem was followed by four alternative
decision choices. Intensive review by expert pilots
assured that each alternative was plausible — that is,
each offered a decision a 500-hour pilot might reason-
ably be expected to make — and differed with respect
to the degree of safety involved.

Scenario Structure

The final set of scenarios used for data collection
consisted of 51 scenarios with all supporting informa-
tion (e.g., airport diagrams). By situational category,
the distribution was as follows: mechanical 8, weather
14, biological 8, sociological 10, and organizational
11. Motivational levels, based on the “Why Fly”
study, were distributed as follows: high or strong, 2;
middle range, 14; and low range or weak, 35.


“Why Fly” Survey. Scenarios, except those involving
weather phenomena and mechanical malfunctions, re-
quired a clear statement of flight mission or motivation.
To determine the relative value of motivational state-
ments for biological, organizational, and sociological
scenarios, 30 pilots provided ratings of 25 of the most
common reasons for flying. In the group of pilots, 10
were flight instructors and 16 were airline transport
pilots. Seventeen of them were employed as pilots. They
rated each reason for flying based on their willingness to
accept risks in order to complete a flight, using a scale of
0 (only willing to accept the least risk) to 100 (willingness
to accept greatest risk). They also provided the number

of times they had flown for each of the reasons rated.

Measures (intraclass correlations) of interrater agree-
mentand stability of responses about reasons for accept-
ing greater risk and for number of times flown for each
reason were computed using GRPREL from the Com-
prehensive Occupational Data Analysis Programs
(CODAP). This analysis package is briefly described in
Appendix E. Interrater agreement (r,,) was .65 and .49,
respectively. Stability (1,,) of responses was .98 and .96.
Stability measures above .90 indicate that a survey of
another comparable sample of subjects would yield
highly similar results.

Means for each reason, also computed by GRPREL,
are shown in Table 1. Review of the means indicates that
the reasons pilots might be willing to accept greater risk

Table 1. Mean Ratings of Reasons for Taking Greater

Flight Risks

Reason

Home-Holidays
Search-Rescue
Angel Flight

Deliver Spare Parts

Hunting Trip

Instruct Students

Sightseeing

Proficiency Flight

Fly Boss to Meeting

Resort-Vacation

Photograph-Show Clients
Property

Fly Co-Workers to Meeting

Fly Home (After 3 Days)

Fly Home (After 7 Days)

Visit Friends

Receive Instruction-Flight
Check

Fly Home (After 1 Day)

Friends to Ball Game

Make Airline Connection

Home-Funeral

Fly Self to Meeting

Administer Flight Checks

Deliver Serum

Ferry-Deliver Airplane

Ferry VIPs

Mean Rating

High Rated
87.73*
79.70
74.97

Midrange Rated
45.57
41.37
38.63
36.87
36.45
35.43
33.77
31.55

26.17
12.10

Low Rated
2.31
1.77
1.72

1.47
1.44
1.40
1.10

*Maximum Rating is 100.00; minimum rating is 0.00.

3




‘8. You as the pilot-in-command (PIC) are going to fly your old instructor pilot to the Planters County Airport so he can pick-up an
airplane coming out of maintenance and give it a functional check flight. Both of you arrive at the airport later than you planned
and he offers to do part of the preflight. You decide to:

Rank Order Alternative

a. Do the planning, filing and preflight together.

>. Have him get the weather NOTAMS and file the flight plan while you preflight the airplane.
¢. Have him preflight the airplane while you get the weather, NOTAMS, and file.

d. Ask him who should do what.

NNR
wee
pees

‘Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance
Runway __| Available
Regional Airport 8800x150 Yes Yes Yes Yes 24 hrs
7753x150

9. You are at a small airport with minimal facilities and at the end of your walk around preflight the flaps refuse to retract from 30
degrees. It was a planned three hour flight back home to the Regional Airport. The attendant says he has seen this problem before
and it is the limit switch sticking. There is no A&P here but there is an A&P at an airport 35 miles up the road. The attendant says
he knows where a switch for this exact model 172 can be quickly picked-up and he could install it. He says he also could reach up

through the inspection port and free the switch enough to raise the flaps, but cani.ot guarantee they will work when airborne. You

call the rental agency and get their answering machine - you are on your own. You decide to:

Rank Order Alternative

1234 a. Leave the flaps down and fly to the nearby (35 miles) airport and have an A&P fix the problem.

1234 b. Have the attendant reset the switch, get the flaps up and fly back to Regional.

1234 c. Have the attendant change the switch, check it out then fly home and have the rental agency inspect the
work.

1234 d Wait until the rental agency can fly an A&P in and change the switch.

10. You are planning a night cross country down to Florida and winds and weather favor cruising around 8000 feet MSL. The
forecast winds and visibility will enable you to make your destination (solid VFR weather) with a 60 minute fuel reserve in one hop.
‘You decide to:

Rank Order Alternative

a. Let down early and cruise in at a much lower altitude if fuel permits.

b. Stay at altitude as long as possible before performing an en route descent.
¢. Make sure there is a working oxygen system on board in case you need it.
d. Plan to use oxygen for this flight.

NNN
wow
heae

B8


APPENDIX D

SUMMARY STATISTICS BY SCENARIO
1 . .

Scenario # — Expert Rank Risk -2 Risk -3 Risk -4
1 bdca 12.61 38.77 40.97 54.64
2 bdca 9.90 40.61 54.19 60.97
3 dbca 5.68 39.19 44.84 60.48
4 cbhda 30.71 45.00 61.93 84.33
5 dbca 10.55 15.06 17.73 51.68
6 bead 17.13 17.83 20.70 52.83
7 cabd 40.33 50.83 53.67 56.67
8 adcb 16.28 23.59 24.90 27.14
9 dbca 4.97 36.21 59.48 64.48
10 dcba 16.22 23.89 36.96 47.50
‘Bt cadb 12.00 41.67 43.90 61.83
12 cbhda 9.45, 44.68 47.74 63.87
13 dbca 7.26 23.39 39.52 63.23
14 bdac 29.84 45.97 59.03 70.32
15 cdab 8.33 15.20 34,87 56.33
16 adbc 22.63 50.26 54.74 59.74
17 beda 12.06 18.23 39.12 40.88
18 abcd 6.05 48.58 54.47 67.89
19 dbca 14.72 20.00 25.83 47.22
20 dcba 11.05 25.79 48.42 76.32
21 bacd 13.06 36.94 40.83 41.84
22 dabe 8.06 24.72 41.67 54.72
23 bacd 17.78 46.94 52.17 62.35
24 bade 11.95 13.68 38.95 47.37
25 bdac 15.53 50.83 61.05 74.05
26 dbac 20.00 26.39 55.28 61.11
27 cbda 14,72, 33.06 46.94 66.67
28 beda 27.63 32.37 42.89 56.58
29 cdab 13.44 14.22 24.78 28.23
30 abed 21.67 25.00 49.17 61.39
31 dcab 13.64 33.06 38.06 53.06
32 bdac 5.79 26.58 67.10 86.95
33 adbc 15.28 30.00 37.50 57.50
34 dcba 7.83 16.25 21.42 43.50
35 beda 2.25 21.67 45.83 64.17
36 dcba 16.42 30.00 34.58 39.83
37 bdca 4.58 17.08 47.67 52.50
38 bdca 17.92 18.33 42.08 58.33
39 bacd 25.45 26.36 30.00 38.64
40 dacb 22.92 30.75 36.25 40.50
Al bade 9.83 23.33 31.25 36.67
42 bacd 17.50 18.33 48.75 63.33
43 cdab 10.83 38.33 65.00 75.83
44 cdba 2.08 38.33 51.25 TINT
45 bade 13.27 23.64 23.64 23.73
46 bead 17.50 33.75 76.67 79.54
47 dbea 32.92 50.17 50.42 58.33
48 bead 16.45 28.18 30.45 33.18
49 cbad 32.73 33.64 35.00 42.54
50 dbca 18.64 44.54 51.82 64.09
51 dbca 13.18 39.54 40.00 42.73

Alternatives are listed in the order given by the expert panel — the first alternative listed was ranked #1 by the
expert panel, and so on.

Risk values are those assigned by the expert panel to the four alternatives. The Risk-1 value is that assigned to
the alternative ranked #1 by the expert panel, and so on.

DI


4. You are cruising at 4500 feet on top of a thin haze layer with the outside air temperature at 65 degrees. It has been twenty-five
hours since the engine was overhauled and the run-up check was well within limits. The engine slowly loses RPM with no

indications of oil or fuel problems. You suspect carburetor icing and pull on the carb heat. ‘The engine backfires, vibrates and loses
RPM fast. You decide to:

Rank Order _— Alternative

1234 ‘& Pull out the mixture, stop the engine and check the fuel selector valve, mag switch settings and declare an
emergency.

b. Push in the carb heat, keep the engine running and divert to the closest airfield.

c. Keep the carb heat on and sce what happens.
d. Push in the carb heat, keep the engine at idle, declare an emergency and ask for advice.

eee
yen
wow
bee

5. You are preparing to enter the VFR traffic pattern at the Regional Airport and hear the tower report winds from 280 at 15 knots,
and they are vectoring traffic to the primary 8800 ft runways 35. A Piper Cherokee asks to usc the 7753 x 150 runway 27. The
Cherokee is told the runway is not active, but to you it looks OK. You decide to:

Rank Order Alternative

1234 a. Accept clearance to runway 35 and follow the traffic.

1234 b. Ask to use runway 27.

1234 ¢. Insist on using runway 27 stating that the crosswinds are unsafe for you to use runway 35.

1234 a. Divert to the Southside Business Airport where the runway is almost directly aligned with the wind.

‘Airport Runway 24br Tower | ARSA Lighted | Telephone | Maintenance

Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 hs
7753x150

Southside Business 4835x100 Yes Yes Yes Yes 0700-1800
4129x100

6. Tt is a cool clear summer afternoon with no wind when you arrive in ARSA going to the Regional Airport. ‘You realize you are
going to be spaced 4 miles behind a commercial 727 on final to runway 17. You decide to:

Rank Order Alternative

1234 a. Stay high on the glide slope and land past where you saw the 727 touchdown.

1234 b. Ask for a 360 tum to increase the spacing.

1234 ¢. Ask to land on runway 09.

1234 d. Ask for a low approach and a visual pattern to runway 17.

Airport Runway 24br Tower | ARSA Lighted | Telephone | Maintenance

Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 brs

7753x150

7. To prepare for when marginal VFR weather makes it difficult to return to your home airfield (uncontrolled), you practice in VFR
conditions:

Rank Order Alternative

1234 a. An unofficial locally devised arrival to the pattern.
1234 b. Have devised your own arrival route to the pattern or runway.
1234 c. Practice a published IFR approach.

1234 d. Don’t do anything.

B7




Phalen, W.J., & Mitchell, J.L. (1993). Innovations in U. S. Civil airmen statistics (1993). Washington, DC:
occupational measurement technology for the Federal Aviation Administration.
U.S. military. Proceedings of the Eighth Interna- .
tional Occupational Analysts Workshop (pp 12- Weissmuller, J. J., Phalen, W. J., & Tartell J. June

16). San Antonio, TX: USAF Occupational Mea- 1 997) Defining and characterizing subgroups us-
surement Squadron ing interrater reliability and cluster analysis tech-

niques. Proceedings of the Tenth International Oc-
Staley, M. R. & Weissmuller, J. J. (1981). Interrater cupational Analysts’ Workshop. San Antonio, TX:
Reliability: The development of an automated analy- USAF Occupational Measurement Squadron.
sis tool, (AFHRL-TP-81-42, AD-A108 400). 7 .
: . . aan . Wickens, C.D., Stokes, A., Barnett, B., 8& Davis. T.
Brooks AFB, TX: Technical Services Division, Air (1987). Componential analysis of pilot decision

Force Human Resources Laboratory. making. (Technical Report ARL-87/4). Savoy,
Stokes, A.F., Kemper, K.L., & Marsh, R. (1992). IL: University of Illinois, Aviation Research

Time-stressed flight decision making: A study of Laboratory.

expert and novice aviators. (Technical Report

ARL-93-1). Savoy, IL: University of Illinois,

Aviation Research Laboratory.


iM ‘You are flying your boss from the Regional Airport to Planter's County to appear as a witness before the Grand Jury. As you
‘stop into the wind for your engine run up, your boss opens his door, leans out and vomits. When finished, he closes the door and
Says “Let’s go.” You have no airsick bags on board. You decide to:

Rank Order Alternative

1234 a. Takeoff and fly as planed to Planter's County and find some suitable airsick bags for the return trip.

1234 b. Taxi back to the FBO, pick up some airsick bags and go.

1234 ¢. Tell the boss you will not fly until he assures you he is well enough to make the flight.

1234 d. Cancel the flight and taxi back.

Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance
Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 has

7753x150

38. You have paid for and been planning this flight to the Lodge Resort at the Lake for six months. The weather is forecast good
VER with a summer haze under 3000 feet and broken scattered clouds along the route of flight. ‘The only problem is you know you
have a minor summer cold. You can clear your ears and only feel a little achy with no headache. You decide to:

Rank Order Alternative

1234 a, Take the minimal dosage of cold tablets and go.
1234 b. Cancel the flight.

1234 ¢. Call your doctor and ask for a prescription for medication.
1234

d. Stick a menthol inhaler in your pocket, take no other medication and go.

36. ‘You are 20 NM outbound from Regional Airport flying solo to deliver two coolers of medical serum to an American Red Cross
field team when departure control calls advising that someone reported a right wheel pant was found off the departure end of the
tunway and it looks like it may have separated from a 172 and is painted a white similar to the color of your airplane. You neither
heard nor felt anything unusual on takeoff and both brake pedals feel normal when you apply them, You decide to:

Rank Order Alternative

1234 a ‘Thank them for the call and ask it be delivered to Aircraft Rental and Leasing and continue your departure,

1234 >. Unfasten your seatbelt, slide over to the right seat and confirm if it is yours.

1234 ¢. Request clearance to return and request a fly-by the tower and have them determine if it is yours.

1234 d. Request clearance to return and land to inspect the airplane.

‘Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance

Runway _| Available

Regional Airport 8800x150 Yes Yes Yes Yes 24 brs

7753x150

B15


37. You need to depart the Planter's County Airport at 6:00 AM for a one hour flight to return the rented aircraft to the Regional
Airport before 7:00 AM. You slide the left seat back to climb in and start the preflight when the seat comes off of the slide tracks.
‘You get the seat back on the track and it seems to hold. You notice that two screws that hold a keeper on the back of the track are
missing and find one under the back seat. The local mechanic will not arrive for two or three hours. You decide to:

Rank Order Alternative

1234 a. Borrow a screwdriver, put in the screw and fly as is having the rental firm checked or fix the seat back at
‘the Regional Airport.
1234 b. Wait until the mechanic arrives and have him fix the seat, then fly home.
1234 ¢. Skip the repairs and fly the trip home from the right seat.
1234 d. Go find a phone, call Aircraft Rental and Leasing and request guidance on what to do.
‘Airport Runway ‘Whr Tower | ARSA Lighted | Telephone | Maintenance
Runway _| Available
Regional Airport 8800x150 Yes Yes Yes Yes 24 hrs
7753x150

38. You have taken off solo from the Regional Airport for a 45 minute flight to the Planters County Airport and have leveled at
3500 feet when you hear a banging start on the right side of the airplane. Everything checks OK so you call the FBO and ask for
advice. After a short period they ask you to find both ends of the right seat belt. You can only find one. You decide to:

Rank Order Alternative

a. Reach over, open the right door, pull in the seatbelt and close the door.
'b. Return to the Regional Airport, land and pull in the seat belt.
¢. Continue and find an airspeed where the banging stops and continue to destination.

2
2
2
2 d. Find the closest airport out of ARSA, land and pull in the seatbelt.

www
bene

39. You are looking for 172s to rent. You have decided the most important thing to look for in a rental plane is:

Rank Order Alternative

a The overall appearance, is it neat and does it look cared for.
b. Aclean engine with clean oil.

¢. New COM/NAV radios.

d. Smooth skin, no dents or dings.

ene
wewe
aeee

40. You have drawn the cross country route on your map and are going to pick your cruising altitude. You will be at altitude for
approximately 3 hours. You decide to pick your cruising altitude primarily based on:

Rank Order Alternative

a MEA
b. Aircraft cruise performance (true/fuel burn)
c. Winds aloft

1
1
1
1 d. Weather/visibility.

Rene

3
3
3
3

Ree

41, You are going to ferry an airplane to the factory and return another which has had new radios installed. You have completed all
the flight planning and decide to:

Rank Order Alternative

a. Have another pilot (not an instructor) you trust check your planning.
‘b. Find an instructor to review your planning.
c. Feel confident enough not to need a review.

1
1
1
1 d. Ask the owner of the airplane to review your planning.

RNNN
euoe
en ne

B16


FEDERAL AVIATION ADMINISTRATION PILOT SURVEY

You have been selected to participate in a Federal Aviation Administration (FAA) study designed to
identify areas of judgement where training would reduce the risk of incidents or accidents. You are especially
valuable in this research as your expressed willingness to participate will allow us to examine pilot judgement
across regions of the country, different age and levels of experience. We greatly appreciate your time and effort
in completing this survey. We think you will find it interesting.

This survey consists of two parts. The first part is an experience questionnaire to enable us to quantify
the experience factors of the pilots participating in this study. Notice that your name or other personal
identifying information is not asked for nor recorded. There will be no way to match your survey response with
you.

Please, complete the Pilot Information Form below. Enter or circle the response accurate for your
aviation experience. Be sure and circle all ratings you have had, not just the highest.

—— —
| PILOT INFORMATION
Circle or enter the correct response
Pilot Certificates Held: Instrument Rating? Engine Rating Age:
SINGLE MULTI
PP COM ATP CFI CFI YES NO
|| State of Residence: | Sox: Ever a Military Pilot? Medical:
M F YES NO Class Class Class I
FLYING HISTORY
Enter the appropriate numbers to the best of your ability
‘Years as a Pilot: Total Hours: Weather Hours: Hours of Instruction: Night Hours:
|_FLYING HOURS IN ‘THE LAST.60 DAYS
Enter the rumbers to the best of your ability’ |
Total Hours: Day Hours: Weather Hours: ‘Hours of Instruction: Night Hours:
|). AIRCRAFT INFORMATION
Enter the data for the aircraft you most frequently fly
Number of Engines: | IFR Capable? Auto-Pilot? Weather Radar? Transponder?
YES NO YES NO YES NO YES NO
Do You Own this Aircraft? Hours of Cruise Capable:
YES NO |
— =

B4




INSTRUCTION LAST 60 DAYS Years as a Pilot

~92
3 NUMBER OF YEARS PERCENTAGE (%)
: 24
6
6
A/C Most Frequently Flown 8
7
NUMBER OF ENGINES PERCENTAGE (%) 7
1 93 3
1 6 3
IFR CAPABLE? 5
Yes. ~75
No. ~25
AUTO PILOT?
Yes. 37
No.

WEATHER RADAR?
Yes.
No..

TRANSPONDER
Yes.
No.

OWN AIRCRAFT?
Yes.
No.

HOURS CRUISE CAPABILITY

NAUBON

C3


APPENDIX C

GENERAL AVIATION PILOT INFORMATION

Demographic Information

CERTIFICATES HELD N PERCENTAGE (%)
Private Pilot 242.
Commercial Pilot
Airline Pilot Transport
Certified Flight Instructor.

Certified Flight Instructor Instrument.........

RATING
Instrument
Single-Engin
Multi-Engine...

STATE OF RESIDENCE

Washington

AGE

10-19
20-29
30-39
40-49
50-59
60-69
70-79
80-89

SEX
Male...
Female...

MILITARY/PILOT CLASSIFICATION
Yes..
Class
Class II
Class III.



Table 4. Comparison of Expert
First Choices with Risk Ratings for
51 Scenarios

Rank Risk N Experts

Order Order’
1 1 42
1 2 4
1 3 5
2 1 6
2 2 35
2 3 8
2 4 2
3 1 3
3 2 9
3 3 29
3 4 10
4 2 3
4 3 9
4 4 39

‘Percentage of risk converted to rank
order; 1 = lowest risk, 4 = highest risk

remaining five had the next to highest risk alternative as
the first ranked choice. No first-ranked choice, however,
had the highest risk percentage.

Second, the highest coincidence of rank and risk
rating was for the first- and fourth-ranked choices. As
indicated, for 42 scenarios the first-ranked alternative
also had the lowest percentage of risk. For 39 sce-
narios, the fourth-ranked choice also had the highest
risk percentage.

Third, agreement between rank order and associ-
ated risk percentage was less for the second and third
ranked alternatives. Rank ordering and percentage of
tisk coincided for 35 and 29 scenarios, respectively.

Scenario-specific information for the nine scenarios
where the first-ranked alternative was not the alterna-
tive with the lowest safety risk rating is in Table 5. For
six of the scenarios, the differences of percent safety
risk were small, ranging from .70 to 8.42.

Risk percentages were higher for three other sce-
narios for which the “Why Fly” statements may have
influenced the rankings. Scenario 36 describes flight
to deliver medical serum to an American Red Cross
team, and a potential mechanical problem is reported
to the pilot. Scenario 46 involves flight to a football
game where a weather problem intervenes in the
flight. Scenario 51 describes a sight seeing trip with
friends when a potential weather problem develops.

IV. COLLECTING GENERAL AVIATION
PILOT JUDGMENTS

Data Collection Instrument

The 51 scenarios were reproduced in random order
in a booklet entitled “General Aviation Pilot Study”
(see Appendix B). To reduce the amount of time
required to complete the research task, the pilots were
asked only to rank order the alternative courses of
action for each scenario. The study booklet consisted
of a background information section, instructions for
ranking scenario alternatives, a pamphlet describing
the aircraft, airport diagrams and associated informa-
tion, and the scenarios.

Table 5. Summary of Expert Rankings and Safety Risk Ratings

Scenario Situational-Motivational

Number Category
51 Weather - Medium
46 Weather - Low
36 Mechanical - Low
34 Biological - Low
10 Biological - Medium
19 Biological - Low
05 Organizational - Low
24 Sociological - Medium
06 Organizational - Low

% Risk Rating

Lowest Lowest

Alternative Alternative Delta
b 39.54 d 13.18 26.36
b 33.75 d 17.50 16.25
¢ 30.00 d 16.42 13.58
b 16.25 d7.83 8.42
Cc 23.89 d 16.22 7.67
b 20.00 a 13.68 5.28
a 15.07 d 10.55 4.52
a 13.69 b 11.95 1.74
a 17.83 b 17.13 0.70


Table 6. General Aviation Sample and
Expert Panel Demographics and Flying

Experience Comparison

Demographic

Mean age in years
Male
Female

Military flying experience
Mean CFI/CFIl years

Percent Certificates Held

Private pilot
Commercial
ATP

CFI

CFIl
Instrument
Single engine
Multi engine

Mean flying hour logged

Total

Weather (IMC) flying hours
Instructor flying hours

Night flying hours

Mean flying hours last 60 days

Total
Day
Weather
Instructor
Night

Sample

46.9
96%
4%
3%
0.0

98%
2%
0%
0%
0%

30%

100%

11%

589
4
1)
43

10
10

Expert

45.9
90%
10%
23%
12.0

6%
35%
48%
67%
65%
97%
93%
90%

4995
581
1683
528

experts and the general aviation sample, the safety
ratings associated with each alternative, and the dif-
ference between the experts and sample.

Item Analysis and Score Generation

A scoring key was developed by taking the alterna-
tive ranked #1 by the expert panel as the keyed
alternative. This answer key is given in Table 8. Using
this key, the responses of the general aviation sample
were analyzed using the ITEMAN Item and Test
Analysis Program (Version 3.50, Assessment Systems
Corp.). This program generated item statistics (given
in Table 9) for each of the 51 scenario-items.

ITEMAN also generated number-right scores
(Safety Deviation Index, SDI) for all subjects and
produced overall scale statistics for the SDI. These
statistics are given in Table 10. The SDI scores were
merged with demographic information and were sub-
sequently analyzed using SPSS for Windows (Ver
6.0). Figure 1 shows the distribution of SDI for the
246 subjects. As may be seen from that figure, the
distribution of scores is approximately normal, with a
mean (and median) of 27.

Analysis of sample demographic variables

Correlations were computed between the SDI and
the sample demographic variables, as shown in Table
11. Although the correlations are uniformly small and
generally not statistically significant, they form an
interesting pattern. Of particular note, the correla-
tions for all the flying experience variables have a
negative sign, while the single correlation with a
positive sign is between the SDI and age.

Table 7. Mean Rank and Percentage of Safety Risk
Differences Between Experts and Sample Pilots

Scenario

35
04
48
15
39
49
51

1*-ranked
Expert

aoveot0e

Sample

oHoT0T0

c

Expert Sample Difference
Rating Rating

2.25 45.83 43.58
30.71 45.00 14.29
16.45 30.45 14.00

8.33 15.20 6.87
25.45 26.36 0.91
32.73 33.64 0.91
39.54 40.00 0.46

Note: Difference = Sample Rating - Expert Rating


Technical Report Documentation Page

7, Report No, 2. Government Accession No. 3, Recipients Catalog No.
DOT/FAA/AM-98/7

“4, Tie and Subtile %. Report Date
Evaluating the Decision-Making Skills of General Aviation Pilots February 1998

6. Performing Organization Code

7. thors)

Driskill, W.E., Weissmuller, J J., Quebe, J., Hand, D.K.;'

and Hunter, D.R.”

®. Performing Organization Report No.

9. Performing Organization Name and Address

"Metrica, Inc.
San Antonio, TX 78216

°FAA Office of Aviation Medicine

10. Work Unit No. (TRAIS)

11. Contract or Grant No,

800 Independence Ave., S.W.
Washington, DC_20591

72. Sponsoring Agency name and Address

Office of Aviation Medicine
Federal Aviation Administration
800 Independence Ave., S.W.

13. Type of Repon and Period Covered

Washington, DC 20591 14, Sponsoring Agency Code

15. Supplemental Notes

16. Abstract

An instrument consisting of 51 items was developed to assess pilot decision-making skill. Each item
consisted of a stem, a short description of an aviation scenario requiring a decision on the part of the pilot.
Four alternatives were provided, and subjects were instructed to rank order the alternatives from best to
worst solution to the scenario presented. Rank-ordered judgments of a sample of 246 general aviation (GA)
pilots (with an average of about 500 hours of total flying experience) were compared with the recommended
solutions provided by an expert panel. Results indicated that, overall, GA pilots and an expert panel of pilots
agreed in their judgments of the appropriate course of action in situations critical to flight safety. However,
the degree of agreement of individual general aviation pilots with the recommended solutions varied widely.
An index of agreement (Safety Deviation Index) was calculated that expressed the degree of agreement of
individual GA pilots with the recommended solutions. Initial evaluation of this index indicates that it
demonstrates adequate psychometric properties and that, as other research would suggest, it has little
relationship with common demographic or flight experience measures.

17, Key Words
Pilots, Aircraft Pilots, Decision-Making, Aviation Safety,
Linear Modeling, Policy Capturing

78. Distbution Statement

Document is available to the public through the
National Technical Information Service
Springfield, Virginia 22161

49, Securily Classi. (ofthis report)
Unclassified

20. Security Classi. (ofthis page) 21. No. of Pages 22. Price

Unclassified 43

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized



30. You have taken off from a resort headed back home when after 30 minutes you notice you cannot identify any landmarks or
checkpoints on your route of flight. You check the instruments and find the cage knob on the RMI not fully in and turned. When
you uncage it, it swings wildly and points back in the opposite direction you were supposed to track. You decide to:

Rank Order = Alternative

1234 a. Plot out a reciprocal heading on your sectional, turn around, call Flight Following and tell them what you
did, then determine your hours of fuel remaining and modify your flight plan.

234 b. Cail Flight Watch, tell what you did and ask for a vector back on course.

2 4 ¢. Tum around, cancel your flight plan, don’t say why, return to the airport where you took-off, land, fuel and.

‘start over.

1234 4. Call Flight Watch, cancel your flight plan, then plot a new route home and go without a flight plan.

1
1

31. You have planned a four plus hour cross country and the weather could easily force you into rather undesirable routes which
‘would take you over rough and desolate country. To match the best weather and route combination, you decide to:

Rank Order Alternative

1234 a. Select the route with which you feel the most comfortable and have the weather forecaster give you the
forecast and if VER is not recommended, repeat this process until you have a VER route.

b. Tell the forecaster your departure point, destination and have him select the best route.

¢. Give the forecaster three routes and have him give you the weather for each then you decide.

d. Delay the flight until you get VFR weather over the primary route.

vue
wow
pee

32. You have called an aircraft mechanic you usually use whose hangar is on the Justin County Airport, and arrange to have your
inoperative landing light rewired. Since you will arrive after dark, he advises you that the runway lights are out and the black
asphalt runway may cause you a problem. You decide to:

Rank Order = Alternative

a. Ask him if be will park his pickup near the threshold and shine his lights down the runway.
b. Postpone the repair until you can land in daylight.

¢. Go ahead and make a black-out landing.

d. Postpone the flight at least until the runway lights are working.

rene
wuewe
Rhee

‘Airport Runway 2br Tower | ARSA Lighted | Telephone | Maintenance
Runway _| Available
Regional Airport 8800x150 Yes Yes ‘Yes Yes 24 hrs
7753x150
Justin County Airport 3200x50 No No Yes Yes 0700-1800

33. You are one hour into a three hour cross country returning to the Regional in the mid afternoon and have been flying at 5000
feet MSL over a scattered deck of clouds and find yourself lost. The cloud deck thins out to where you can see the ground in all
directions. You decide to:

Rank Order = Alternative

1234 a. Call FSS on the radio, report yourself lost, and have them give you a position fix and vector back to course.

1234 b. Find a prominent landmark, circle it until you can find it on your sectional, fix your position and lay out a
heading to get back on course.

1234 c. Go to the nearest town, descend and read the name of the town on a water tower or other prominent
structure, fix your position and plot a heading to get back on course.

1234 d. Find an airfield, land and ask where you are, refile, then continuc.

B14





NOTICE

This document is disseminated under the sponsorship of
the U.S. Department of Transportation in the interest of
information exchange. The United States Government
assumes no liability for the contents or use thereof.




14. Your passengers decided to go into Driskill City and have someone drive out and pick them up.

The approaching thunderstorm continues to grow with reports af 80 MPH wind gusts and large hail. To save the airplane you take
off and circle northeast around the storm to avoid any possible wall cloud. You plan to stay at least 20 NM away from the storm.
You level at 3000 feet with the storm on the left wingtip and the Regional Airport VOR DME reading 27 NM. An undercast begins
to develop and the lights of the city fade. You decide to:

Rank Order Alternative
1234 a. Hold altitude, start flying counterclockwise around the 27 NM arc to clear air behind the storm.
1234 b. Contact Big City Approach control and tell them your problem and ask for vectors avoiding the storm to
any clear airport.
1234 c. Descend slowly to MEA (1000 feet) and establish VFR conditions before entering ARSA and contacting
Big City Approach.
1234 d. Make a right turn, put the storm on the tail as best you can, find clear air, and orbit until the Regional

3
3

1S, Bad weather forced you to cancel flying your boss into another city where he is to address a convention. There are openings on
a flight going to the same city departing from the airline terminal on the other side of the airport in 15 minutes. {t will take too jong
to call a taxi so he asks you to run him over to the terminal in the 172. You decide to:

Rank Order Alternative

1234 a Start the engine and ask ground control for permission to taxi to the back of the terminal, drop off a
passenger and taxi back to the FBO ramp.

1234 b. Start and ask ground control for permission to taxi around the airport for a maintenance check and
conveniently drop the boss off near the terminal,

1234 ¢. Say you're sorry but itis illegal for you to deliver passengers to the back side of the terminal and help find a
Tide through the FBO.

123 4 da Ask ground control if there is any way a representative from the airline could meet you at a door to the
ramp and escort the boss into the terminal.

‘Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance

Runway _| Available
Regional Airport 8800x150 Yes Yes Yes Yes 24 brs
7753x150

16, You have entered the left turn VFR wraffic pattern for runway 27 at the Regional Airport and hear muffled bangs and feel serious
vibrations. New dents in the top of the engine cowling confirm a cylinder has failed. The winds are 070/$ and you are just
approaching extended centerline of runway 35. You decide to:

Rank Order Alternative

1234 a. Declare an emergency, shutdown the engine and turn left to land in the crosswind on runway 35,
1234 b. Declare an emergency, shutdown the engine and turn left 180 degrees to land on the grass on the left side of
Tunway 27.
1234 ¢. Declare an emergency, shutdown the engine and turn left and stretch the tur to land on runway 27.’
1234 d. Declare an emergency, shutdown the engine and land straight ahead on the taxiway on the south side of
runway 09.
Airport Runway 24hr Tower | ARSA Lighted | Telephone | Maintenance
Runway __| Available
Regional Airport 8800x150 Yes Yes Yes Yes 2 hrs
7753x150

B10


50. ‘The enroute weather briefing for the three hour cross country was for scattered thunderstorms along the route of flight, and sure
enough there is a cluster of cells developing dead ahead on your route of flight. Other clusters have sprung up on each side of you,
and behind all close to 20 miles away. You decide to:

Rank Order Alternative

a. Proceed looking for a route around or through the rain shafts which will allow you to remain VFR.

». Fly upwind of any cloud build ups and stay VFR.

¢. Ifthe downwind route around dark cells is the only clear way, keep at least 20 miles from the closest cell.
@. Find an airport below in VMC, land, and wait until the thunderstorms pass and the route is clear.

REN
www
aaean

1
1
1
1

‘Si. It had rained all day, but the front pushed south of you and cleared the skies, ‘You are out with two friends on a sight seeing trip
to the hills 40 miles away and plan to be back before dark. With sunset still an hour away you notice ground fog beginning to form.
‘You decide to:

Rank Order Alternative

a. Apply full power and race back to the home airport.

b. Call Flight Watch and cruise back home.

¢. Call on your home airficld’s CATF to see if anyone is there and can tell you what the weather is doing.
d. Go directly to an airport you know is closer than your home airport, land and find out what the weather
is doing.

NVONK
wuuw
Pe be

THANK YOU!

PLEASE CHECK THAT YOU HAVE RESPONDED TO ALL THE SCENARIOS, WE FIND THAT SOMETIMES THE
PAGES STICK TOGETHER.

MAKE SURE YOU FILL OUT THE PILOT INFORMATION FORM.

If you want information on the results of this study, please enclose a note with your name and address (don’t put them on your
answer sheets), and we will send you a copy of the report.

B19


The second part is a set of scenarios developed from real experiences reported to the FAA through previous
studies. These scenarios contain a stem which sets the scene, and four courses of action which can be taken in
response to the problem posed in the stem. Additional information including a diagram of each airfield (except
for military installations) is provided. Airfield data relevant to each specific scenario is provided in a table
immediately following the possible actions. For each of the 51 scenarios shown in Part 2 of the survey on pages
1 to 14, do the following steps:

First, carefully read the scenario and the four listed Alternative responses. Assume you have leased the
Cessna 172 shown on the flyer from Aircraft Rental and Leasing. Feel free to use the airfield diagrams for
assistance in understanding the problem.

Second, based on your experience, decide which of the alternatives you would most likely select as your first
course of action, what would be your second choice, then your third and fourth if you were the pilot in the
scenario. To the left of each choice are the numbers (1 2 3 4), circle the 1 next to the choice you selected as
your first. Next select the choice you consider second and circle the 2 to the left of the choice, then circle your
third and fourth choices.

Third, check that you have circled only one 1, 2, 3, or 4 for each alternative.

Last, put the completed survey in the pre-addressed envelope and drop it in the mail, NO POSTAGE
NECESSARY!

This is not a test. There are no right or wrong answers, and you are not asked to do anything illegal. This is not
a test of rules but a survey of pilots who have "been there, done that" and have survived to pass lessons learned
from that experience to others just starting their flying careers. These scenarios were developed from rea/
incidents reported in previous surveys where the pilot learned a valuable lesson and was willing to share with the
aviation community.

Here is an example of what a properly completed question in the survey should look like.

15. Your airplane battery is dead and you are alone at the airfield. To start your airplane you decide to:

Rank Order Alternative

12@4 a. Get your jumper cables and move the plane close enough to the 172 parked
next to you and jump start the engine.
12 3@ b. Prime the engine, tum on the master, set the mag switch to both, slightly crack
open the throttle and swing the prop.
34 ¢, Connect the jumper cables from the airplane to your car and see if it will start.
34 d. Take the battery out and take it to be charged by an A&P mechanic.

Review the reference material on the next six (6) pages.
Following your review, turn the last reference page and
begin Part 2 by rating Items 1-51 on pages 1-14.

BS


III. CAPTURING EXPERT PILOT POLICY

Formatting of the data collection instrument, in-
structions for performing the three judgment tasks,
and criteria for selecting members of the expert panel
are described below.

Formatting and Instructions for Expert Pilot
Judgment Tasks

During the development of the scenarios, the field
review by subject matter experts made it apparent that
capturing the judgments of three factors for each of
four alternatives would be very time consuming —
possibly requiring two or more hours. Judgments of
30-35 scenarios were the most that could be captured
in one hour. Asa result, the 51 scenarios were divided
into two subsets that were reproduced as General
Aviation Pilot Study Form A and Form B. Each form
contained 33 scenarios. Fifteen scenarios were com-
mon to both Form A and B; the remaining 18 sce-
narios were unique for each form.

Instructions were provided for the three kinds of
rating tasks the experts were asked to perform. For the
first task, pilots were asked to rank each scenario
alternative with respect to “...the best course of action
for a pilot with a Private Pilot Certificate (PPL) who
has not taken the Instrument written or Flight Check,
and has approximately 500 total hours of flying time.”
Alternatives were rated a J for the best course of action
to 4 for the worst.

In the second rating task the experts were asked to
rate the plausibility of each alternative in terms of the
percentage of 500-hour pilots who might choose the
courses of action described in each scenario. The
experts used a scale ranging from 0 to 100 percent.

The third rating task asked the experts to estimate
the risk associated with each alternative. Specifically,
instructions stated “...entera number ranging from 0 to
100 which would best represent your estimate of how
risky this course of action would be for a PPL with 500
Slying hours to successfully complete.”

A page illustrating the layout of the scenarios and
alternatives for rating by the safety experts is provided
in Appendix A.

Safety Expert Criteria

Several criteria were established for selecting pilots
to form the expert safety panel for assessing their
safety risk judgments.

1. Since the objective of this research required expert
panel members be pilots whose focus was overtly on
flight safety, the first criterion for identifying ex-
perts was their employment in positions in which
flight safety is a foremost job objective. A second
criterion required prima facie evidence of the pilot’s
interest in safety above and beyond that exhibited
by the previous national samples of pilots. Pilots
meeting these criteria included FAA staff members,
senior instructional staff in university aviation pro-
grams, and pilots who frequently attended safety
seminars.

2. Asa group, the level of experience should provide
them with knowledge of the situations described in
the 51 scenarios. In addition to age, 21 other expe-
rience items were developed, ranging from total
hours logged, certification and ratings, to total and
recent night and weather flying experience.

3. The expert responses were carefully examined for
interrater agreement (r,,) and stability (ry) and
assessed for the presence of multiple policies with
regard to the rank ordering of the scenarios and
ratings of percentage of safety risk. The minimum
r,, was .20 which represents the average correlation
of each rater with each other rater. The minimum
for r,, was .90; values at this level or higher indicate
high probability that another set of raters would
provide comparable ratings. In addition, GRPREL
assesses each rater in terms of agreement with other
raters and identifies “deviant” raters. Deviant raters
are those whose rating policy does not significantly
correlate (p < 0.05) with other raters. This process
evaluates the existence of multiple policies. Usually,
only one to three raters are identified as deviant
because they did not follow instructions or reversed
the scale used. When there is a larger number of
deviant raters, GRPREL assesses them in the same
way to determine if there is a significant secondary
policy. A description of the GRPREL analysis sys-
tem and representative references are available in
Appendix E.

4. The experience level should exceed that of the na-
tional population in terms of total hours and levels
of certification.

Collection of Expert Data

Forms A and B of the General Aviation Pilot
booklet were personally delivered to pilots. They
returned their responses by mail. Judgments were


APPENDIX E
SOFTWARE AND ANALYSIS

Data analysis for this project was accomplished on a Unisys 1100 series mainframe computer at the Human
Resources Directorate of the USAF Armstrong Laboratory (AL/HR) and on the IBM RS-6000 RISC model 530
machine at the Metrica facility. All software used was developed by and is the property of the USAF Armstrong
Laboratory, Brooks AFB, San Antonio, Texas. More specifically, in addition to utility and custom software required
to reformat input files, data analysis was carried out using software packages described in the Mathematical and
Statistical Library of the Armstrong Laboratory. This library is reviewed in Albert, W. G. & Whitehead, L. K.,
MATHEMATICAL AND STATISTICAL SOFTWARE INDEX: SECOND EDITION (AFHRL-TP-85-47,
August 1986). This index. provides a brief description of each package and external references when available.
Detailed program documentation is only available on the Unisys under their “@DA*DA.ADOC” retrieval system.

The Unisys 1100 is scheduled for shut-down and replacement in October 1997 by an IBM RISC machine at AL/
HR which is already in operation, but not yet accessible to outside users. It is unknown to this contractor as to which
Unisys-resident programs are being converted for operation on the RISC platform and which will be “lost”. Metrica
has already produced the RISC-based version of ASCII CODAP.

This project used the GRPREL (Group Reliability, Staley & Weissmuller, 1981) analysis procedure. A brief
description of this package is provided below. Additional information of the use of this analytic tool is available in:
Albert, etal., 1994; Christal & Weissmuller, 1976, 1988; Goody, 1976; Phalen & Mitchell, 1993; Staley & Weissmuller,
1981; and, Weissmuller, Phalen, & Tartell (1997).

GRPREL

GRPREL is the standard interrater reliability in the CODAP system. Fora given list of items rated by a set of Subject
Matter Experts (SMEs), this program reports two measures of interrater agreement (R andR ).TheR value
indicates the reliability of the observed set of ratings— while 0.10 is considered a minimum forusablé rater agreement,
a value of 0.20 or greater is desired. The R, value is driven by the number of raters actually used. Although an R
of 0.90 is usually desired, it may not be practical i ina particular study because of a small number of raters (SMES}
that may be subdivided even further into smaller groups based on policy differences. The GRPREL program also
computes means and standard deviations for each item in the list. Item-level reports are printed in three orders: original
sequence, ordered descending y-mean value, and ordered descending on standard deviation. GRPREL computes
each rater’s correlation with the full-group mean vector and uses a probability evaluation to recommend the removal
of deviant (non-cooperative or reversed scale) raters. The program can automatically iterate and remove flagged
raters until either a sufficienct level of agreement Rg, = 0.90) is reached or no raters can be found with a probability
(of deviant rating) above 0.95.

El YYU.S. GOVERNMENT PRINTING OFFICE: 1998 -660-068/60060


The background information section, titled Pilot
Information, elicited the same demographic and ex-
perience information from the national sample as was
obtained from the expert panel of pilots. In addition,
the sample pilots were asked to provide information
about the aircraft they flew most frequently.

The instructions, after explanation of the informa-
tion supporting the scenarios, carefully described how
alternatives should be ranked, the substance of which
is quoted below:

First, carefully read the scenario and the four listed
alternative responses. Assume you have leased the Cessna
172 shown on the flyer from Aircraft Rental and
Leasing. Feel free to use the airfield diagrams for assis-
tance in understanding the problem.

Second, based on your experience, decide which of
the alternatives you would most likely select as your first
course of action, what would be your second choice,
then your third and fourth if you were the pilot in the
scenario. To the left of each choice are the numbers (1
234), circle the 1 next to the choice you selected as your
first. Next select the choice you consider second and
circle the 2 to the left of the choice, then circle your third
and fourth choices.

An example showing how rankings were to be
marked was included.

The instructions explicitly stated that “This is not a
test.” Further, they stated “There are no right or wrong
answers, and you are not asked to do anything illegal.”!

Data Collection and General Aviation Sample
Demographics and Experience

Data Collection. The data collection instrument
was mailed to approximately 1,000 private pilots in
the Eastern, Southwest, and Northwest Mountain
Regions of the FAA. Pilots were randomly selected
from current FAA records. The package contained a
postage-paid envelope addressed to the Office of Avia-
tion Medicine in Washington, D. C., for pilots to use
in returning their responses. Responses from 246
pilots were returned to the Office of Aviation Medi-
cine and key entered for data analysis.

Sample Demographics and Experience. Sample
demographics and flying experience are compared
with similar information for the 31 pilots in the expert
safety panel in Table 6. A complete summary of the
demographic and flying experience of the general
aviation sample is provided in Appendix C.

The sample statistics support two conclusions. First,
the sample satisfied the objective of the study to
obtain the judgments of pilots with about 500 total
flying hours. In addition, most of them (70%) were
not instrument-rated. Second, the criterion related to
the experience requirements of the expert panel of
pilots was achieved, as the pilots in the GA sample had
considerably less experience that the experts, nor did
any of the general aviation pilots hold CFI or CFII
certificates.

The GA pilot sample is comprised mostly of pilots
from Texas (129), New York (63), and Washington
(34), although pilots from 12 states are represented.
The most distinguishing features of the national sample
are the ownership of their own aircraft (by 50% of the
pilots) and the less than 750 hours of total flying hours
(by 76% of them). Other flying experience, as would
be expected, is similarly limited: 78% have fewer than
50 hours night flying and over one-half have fewer
than 24 hours of night flying. The aircraft they most
frequently fly is equipped with a transponder (96%),
75% are IFR-capable, 37% are autopilot equipped,
but only nine percent are equipped with weather
radar. Most of the pilots (93%) report flying single-
engine aircraft. While all are single-engine rated,
only 11% are multi-engine rated; none are certi-

fied as CFI.

V. ANALYSIS OF SCENARIO ITEM
RESPONSES

The mean vectors of rankings for the general avia-
tion sample and the expert panel of pilots were corre-
lated, and a Pearson r of 0.914 was obtained. The
general aviation sample and experts rank-ordered the
alternatives differently for only seven scenarios. Table
7 lists the seven scenarios where the differences
occurred, along with the first choice alternative of the

" After virtually all the data collection had been completed it was discovered that some of the scenarios depicted VFR flights at altitudes
normally used for IFR flight. According to the Airman’s Information Manual, VER flights in an easterly direction should be conducted
at an odd thousand feet plus 500 feet (for example, 7,500 feet); while westerly VER flights should be conducted at an even thousand
feet plus 500 feet (for example, 6,500 feet). However, some of the scenarios used in the data collection indicated that the pilot was
flying VFR at a whole number of thousands of feet. For example, the second scenario reads, “You are solo on a late night cross country
ctuising VER at 9000 feet...” While this is not illegal, it is contrary to recommended operating procedures. Neither the experts nor
the general aviation pilots commented upon this error, so it is believed not to have significantly influenced the results,