Judgment Training Page 19

Ground Time
Self-Study Time
In-Flight Time

MORE THAN 8 HOURS

2-4 HOURS

LESS THAN 2 HOURS

PERCENT (N=25)
Figure 11. Student estimated time requirements for judgment training,

Overall Program Evaluati

Summary questions helped to evaluate the degree to which
participants felt that the program had accomplished its goals.
Approximately one quarter (24%) of the instructors said that “all” of their
students had benefited from the program, and almost half (46%) indicated
that "most" of their students had benefited from judgment training. More
than half (56%) of the students reported that their aeronautical judgment
had “definitely” improved as a result of the program, while one fifth (20%)
were uncertain or felt that their judgment had not improved. About one
third (32%) of the instructors felt that their typical student's judgment
nad “definitely” improved as a result of program participation, but 20%
were uncertain or saw no improvement.

An important question asked of respondents was whether they would
recommend that the judgment training program be adopted. Four-fifths of
the student sample (80%) recommended program adoption, and almost
two-thirds (63%) "strongly recommended” the use of the judgment manual
in private pilot training. Twenty-one percent of the students were
hesitant or would not recommend the program. These results are
presented in Figure 12.


Judgment Training Page 36

APPENDIX G

Experimental Group Data

Observation Written

No. Observer Flight Score Age Test Score Airport ID
602 c 42 55 70 FDK
603 C 58 27 93 FDK
604 R 73 34 87 N92
605 R 75 18 95 W10
606 R 67 44 90 N92
607 c 58 32 95 FDK
608 R 58 35 97 FDK
610 R 58 46 90 FDK
611 R 58 50 85 ISP
612 R 45 21 82 FRG
613 R 75 28 88 FRG
614 R 83 20 97 ISP
615 0) 75 41 87 INO
616 0) 83 53 97 N43
617 0 83 26 92 N43
618 0) 75 44 80 N43
619 0) 92 21 73 ISP
620 0) 92 30 92 ISP
621 fe) 92 20 88 iSP
622 0) 58 44 92 FRG


Judgment Training Page 13

VERY USEFUL

MLM LLL
MODERATELY USEFUL LLL ELLLLLEEE

SLIGHTLY USEFUL

Lesson Plans &
Judgment Scenarios

Manual

NOT AT ALL USEFUL Z

0 10 20 30 40 50 60
PERCENT (N=42)

Figure 3. Ratings of instructional materials.

Questions in both the student and instructor critiques were directed
at five basic components of the judgment training program. Figures 4
through 8 present the proportions of each respondent group which gave
ratings of "useful", “moderately useful", “slightly useful", and "not at all
useful" to each of these program elements.

VERY USEFUL Fo >
VISI4, thle Yh ttbtbh

MODERATELY USEFUL

SLIGHTLY USEFUL

Instructors (N=42)

Students (N=25)

NOT AT ALL USEFUL a

9 10 20 30 40 50 60 70
PERCENT

Figure 4, Ratings of Five Hazardous Thoughts concept.


Judgment Training Page 23
NOTES

1. Significant contributions to this study were made by Georgette Buch
(Transport Canada), Douglas P. Harvey (FAA), Russell A. Lawton (AOPA), and
Gary S. Livack (GAMA). The observation f lights were conducted by AOPA
staff members Robert Cathers, Thomas Oneto, Glenn Rizner, and John
Sheehan. Research assistance at Colby College was provided by Karen M.
Barbera, Deborah £. Burke, Fran M. Gradstein, and Melissa J. Hruby.
Secretarial support was provided by Dorothy Evertsen.

“2. In order“to determine the extent to which the flight training activities
presented material not commonly included in private pilot training, ‘an
informal small-sample survey was conducted at three flight schools which
were not part of the experiment but which utilize the same type of
traditional pilot training curriculum as was used at the participating
FBO's. The results suggest that only 12 of the the 50 Judgment Training
Scenarios are unique to the program and present activities which “are
rarely undertaken by flight instructors in normal practice. In contrast, 21
of the activiites presented in the Scenarios are used on occasion. by more
than half of the flight instructors surveyed, and 17 are used by some
instructors. Of the activities outlined in the 18 Lesson Plans, all but one
are commonly included in flight training by atmost all instructors
surveyed. eee

—

3. These evaluations were Subsequently made available to the FAA
researchers involved in evaluating the Prototype sectional charts


Judgment Training Page 4

Thus, following completion of the Air ‘Cadet study, it was decided to
undertake a major revision of the judgment training manuals, and then to
demonstrate the effectiveness of these revised materials in conventional
flight schools. The revision of the student pilot and instructor manuals
was undertaken as a joint project by FAA, Transport Canada, and the
General Aviation Manufacturers Association (GAMA) (See Note 1). °
Audio-visual materials, consisting of approximately one hundred 35 mm
color slides and a six minute video tape, were also produced to enhance the
training of both student pilots and instructors.

Tne 70 page Student Manual introduces the concept of judgment and
describes the role of risk assessment and decision-making in aviation.
The student's attention is then directed to the subject areas of Pilot,
Aircraft and Environment, and to Six Action Ways or modes of pilot error
(Do, No Do, Early Do, Late Do, Under Do, Over Do). The notion that errors are
cumulative is discussed in the context of the Poor Judgment Chain, and the
student is given a method to help break this chain. Following a discussion
of the Three Mental Processes of Safe Flight (Automatic Reaction, Problem
Resolving, Repeated Reviewing), a Self-Assessment Inventory introduces
the student to Five Hazardous Thoughts (impulsivity, Invulnerability,
Macho, Anti-Authority, Resignation), and specific Antidotes to each
hazardous thought are provided. The manual concludes with chapters on
Stress Reduction and some specific applications of pilot decision-making
skills.

The 63 page Instructor Manual provides specific guidance on how to
teach judgment, and outlines 18 in-flight Lesson Plans and SO Judgment
Training Scenarios to help accomplish the program's goals (See Note 2). It
includes a proposed schedule of student work, suggestions on how to
manage judgment training, and a master set of student records to be
duplicated and used at each training site. The manual concludes with a set
of Postcheck Exercises to assure that the student has mastered the
judgment training material. Both of these manuals are extensively revised
versions of the material which had been developed by the ERAU research
team


Judgment Training Page 34

APPENDIX E

Experimental & Control Group Observation Flights
At Each Participating FBO

Experimental Control

Fixed Base Operator Airport ID Subjects Subjects
Summit Aviation N92 2 0
Frederick Aviation FDK 5 0
Flightways of Long Island FRG 3 4
Mid-Istand Air Service ISP 5 3
Queen City Aviation INO 1 4
Bradens Flying Service N43 3 2
Mid-Eastern Airways W36 0 4
Air Virginia LYH 0 0
Dulles Aviation W10 \ 4
Woodbridge Aviation W22 0 4

TOTAL 20 25



Judgment Training Page 20

STRONGLY RECOMMEND

RECOMMEND

RECOMMEND WITH HESITATION

DO NOT RECOMMEND

0 10 20 «63000 40 S00
PERCENT (N=24)

Figure 12. Student recommendations regarding judgment training.

More than three-quarters (77%) of the instructors recommended that
the program be adopted on a voluntary basis, with almost half (48%)
“strongly” recommending such adoption. More than half (55%) of the
instructor sample recommended that judgment training be required, and
about one quarter (26%) “strongly recommended” that the FAA require. that
such a program be used to teach and evaluate judgment skills in student
pilots. Still, 24% of the instructors were hesitant or unwilling to
recommend it on even a voluntary basis, and almost half (45%) of the
instructors had serious reservations about mandatory judgment training.
These results appear in Figure 13.

STRONGLY RECOMMEND LLL WLLL.

RECOMMEND CMLL A
RECOMMEND WITH HESITATION LLL

Voluntary
A peau |
vowor econ LLLLLLLA Required

0 10 20 30 40 50

PERCENT (N=42)

Figure 13. Instructor recommendations regarding judgment training.


Judgment Training Page 26

Good Judgment: pilot detects and safely secures the bottle during the
preflight inspection prior to engine start

Poor Judgment: pilot fails to detect and secure the bottle prior to engine
start

NOTE: IF UNDETECTED BY THE PILOT, THE OBSERVER MUST SAFELY SECURE
THE BOTTLE WITHOUT COMMENT PRIOR TO TAXI

4. Propeller Hazard Approximately 30 seconds after engine start, but
before taxi or run-up, the observer releases his seat belt as he announces
that:

a) he has dropped his pen in front of the aircraft

b) he sees a loose hold-down on the cowling

c) he sees a nail on the runway in front of the aircraft
He then begins to open the door to exit the aircraft.

Good Judgment: pilot immediately shuts down the engine or requests that
the observer remain in the aircraft

Poor Judgment: pilot permits the observer to leave the aircraft while
engine is running

NOTE: OBSERVER MUST NOT EXIT THE AIRCRAFT WITH ENGINE RUNNING

5. Checklist Distractions During the execution of the prestart, pretaxi,
and/or run-up checklist the observer converses with, questions or
otherwise distracts the pilot in such a way as to interrupt the systematic
execution of the checklist. Two interruptions are to be made.

Good Judgment: pilot requests that the observer hold his questions and
comments until after the checklist is completed, or pilot ignores
observer's remarks

Poor Judgment: pilot interrupts the checklist to converse with the
observer, or carries on the conversation while continuing the checklist


Judgment Training Page 22
REFERENCES

Berlin, J. |., Gruber, E. V., Holmes, C. W., Jensen, P. K., Lau, J. R., Mills, J. W.
& O'Kane, JM. (1982). Pi H ind i

(DOT/FAA/CT-82/56). Daytona Beach, FL: Embry-Riddle Aeronautical
University. (NTIS No. AD-A1 17 508)

Buch, G. (1982). Pilot judgment training validation experiment. Transport
Canada and the Federal Aviation Administration.

Buch, G. & Diehl, A. (1983). Pilot judgment training manual validation.
Transport Canada and the Federal Aviation Administration.

Buch, G. & de Bagheera, |. J. (1985). Judgment training effectiveness and
permanency. Proceedings of the Thir ional jum_on
Aviation Psychology, 337-343.

Buch, G. & Diehl, A. (1984), An investigation of the effectiveness of pilot
judgment training. Human Factors, 26, 557-564.

Federal Aviation Administration, Transport Canada, & General Aviation

Manufacturers Association. (1983). Judgment training manual for
instructor pilots (Draft). Washington, D.C. Author.

Federal Aviation Administration, Transport Canada, & General Aviation

Manufacturers Association. (1983). Judgment training manual for
student pilots (Draft). Washington, D. C.: Author.

Jensen, R. S. & Benel, R.A. (1977). Jud Juati instruction in
civil pilot training (FAA-RD-78-24). Savoy, IL. University of Illinois
Aviation Research Laboratory (NTIS No. AD-AOS7-440).

Lester, L. F. & Bombaci, D. H. (1984). The relationship between personality
and irrational judament in civil pilots. Human Factors, 26, 565-572.


Judgment Training Page 35

APPENDIX F

Control Group Data

Observation Written

No. Observer — Flight Score Age Test Score Airport ID
501 Cc 33 W36
502 R 58 28 92 FRG
503 R 75 ' 35 92 ISP
504 Cc 64 39 97 W22
505 Cc 42 27 92 W36
506 Cc 58 27 82 w22
507 Cc 42 32 100 W22
508 Cc 50 36 98 W22
509 R 50 33 85 W10
510 R 67 23 72 wid
Si Cc 50 32 100 W36
312 Cc 42 26 93 W36
513 R 58 38 92 W10
514 R 50 28 100 W10
SIS R 67 34 70 FRG
516 R 92 30 82 FRG
517 R 25 21 7 FRG
518 0 67 27 82 1N9
519 is) 75 34 93 1N9
520 6) 58 64 87 INS
521 ie) 83 42 98 INQ
527 0 83 39 90 N43
528 e) 7 54 72 N43
529 R 67 20 83 ISP
530 R 67 46 72 ISP


Judgment Training Page 17

STRESS REDUCTION VMLLLLLLLLELLLLLLZEEEE

HAZARDOUS THOUGHTS

POOR JUDGMENT CHAIN LLLLLLI LILLE

Instructors
MENTAL PROCESSES (N=42)
LLLLLLLLLLLLEE LITE
: Students
SIX ACTION WAYS (N=25)

Yh, LEA —
NOT AT ALL SLIGHTLY MODERATELY VERY
USEFUL USEFUL USEFUL USEFUL

MEAN RATING

Figure 9. Mean scaled ratings of judgment training concepts.

Training Time Requirements

The study was also concerned with the amount of training time which
the judgment program consumed within the Private pilot curriculum. Not
only were instructors asked to master the Judgment training material on
their own time, but they also had to allocate ground school and in-flight
time with each student to accomplish the program's objectives.

Most instructors (44%) reported Spending a total of between two and
four hours of self-study time Preparing to teach the judgment materials,
while almost one third (29%) Spent between four and six hours in this
activity. Thus, on average, the program appears to require that instructors
devote about three to four hours of self Study time in order to become
reasonably familiar with the contents.

Because the judgment materials were designed to be easily integrated
into existing flight and ground instruction, it was felt that much of the
Judgment training would be given in conjunction with items from the
Standard pilot training curriculum. For instance, the instructor might
caution the student about the hazardous thought “impulsivity” while
describing the dangers of rushing a takeoff, For this reason, the


Judgment Training Page 9

RESULTS & DISCUSSION

Subjects

The subjects in the control group were tested by one of four
observers. When the control group data were examined, significant
differences between observers in the mean observation flight scores of
their respective subjects were noted. Given these differences, an effort
was made to have each observer fly with about the same number of
Subjects from both the experimental and contro! groups. However,
because of scheduling difficulties, only three of the four observers were
able to fly with experimental group subjects. For this reason, the
observation flight performance of 20 subjects from the experimental
group is compared with that of the 25 control group subjects who were
tested by these same three observers.

It was not possible to schedule an observation flight with any
Subjects at one of the ten FBO’s. Also excluded from the analysis is data
fromm one subject who appeared for the observation flight in a personally
owned complex turbo-charged aircraft. The number of subjects
participating in experimental and control group observation flights from
each of the ten FBO's is shown in Appendix E. The data for the control
group appears in Appendix F, while experimental group data is contained in
Appendix 6.

No significant difference between experimental and contro} group
subjects was noted with respect to age (t=0.15, df=42). For the 24 control
group subjects on whom demographic information was available, the mean
age was 340 years, with a range from 20 to 64 years. The 20
experimental group subjects ranged in age from 18 to 55 years, with a
mean age of 34.4 years.

There were also no significant differences between the experimental
and contro} groups in the number of flight hours or the number of months
spent in training for the private pilot certificate. Subjects in the
experimental group (N=20) had a mean of 90.6 hours of flight time when
they passed the private pilot check ride, while contro} group subjects
(N=18) had a mean of 72.4 hours (t=1.42, df=36). For the contro! group
subjects (N=17), an average of 9.6 months elapsed between their first solo
and the check ride, compared with a mean of 7.7 months in the
experimental group (N=20) (t=0.73, df=35).

No significant differences between the experimental and control


Judgment Training Page 32

APPENDIX D
Instructor Evaluation
1. Instructor Manual in general was:
A) very useful B) moderately useful
C) slightly useful D) not at all useful

2. In-Flight “Lesson Plans" and “Judgment Training Scenarios” in the
instructor manual were:

A) very useful B) moderately useful

C) slightly useful D) not at all useful
3. Student Manual in general was:

A) very useful B) moderately useful

C) slightly useful D) not at all useful

4. Six Action Ways concept (Do, No Do, Under Do, Over Do, Early Do, Late
Do) was:

A) very useful B) moderately useful

C) slightly useful D) not at all useful
5. Poor Judgment Chain concept was.

A) very useful B) moderately useful

C) slightly useful D) not at all useful

6. Three Mental Processes of Safe Flight concept (Automatic Reaction,
Problem Resolving, Repeated Reviewing) were:

A) very useful B) moderately useful

C) slightly useful D) not at all useful

7. Five Hazardous Thoughts (Anti-authority, Impulsivity, invulnerability,
Macho, Resignation) were:

A) very useful B) moderately useful

C) slightly useful D) not at all useful

8. Stress recognition chapter was:
A) very useful B) moderately useful
C) slightly useful D) not at all useful


Judgment Training Page 29

Good Judgment: pilot informs observer that the requested flight path
cannot be flown safely without ATC clearance or pilot requests such
clearance from ATC, or pilot alters flight path to avoid the controlled
airspace by a safe margin

Poor Judgment: pilot flies marginally close to an Active Airport Traffic
Area or Terminal Control Area without ATC Clearance

NOTE: OBSERVER MUST NOT PERMIT THE AIRCRAFT TO ENTER INTO
CONTROLLED AIRSPACE WITHOUT ATC CLEARANCE

12. Arrival Procedures On approach to an uncontrolled airport, the
observer, in order to take a photograph, requests a flight path and altitude
which brings the aircraft:

a) directly over the airport at pattern altitude but not in the pattern

b) at pattern altitude but flying a non-standard pattern

C) set up high on the final approach leg

d) set up for a downwind landing

Good Judgment: pilot informs observer of the need to fly a standard
approach and landing pattern, and flies it

Poor Judgment: pilot flies a non-standard approach or landing pattern at
pattern altitude

NOTE: OBSERVER MUST NOT PERMIT A HAZARDOUS APPROACH OR LANDING


Judgment Training Page 6

critique the students. Each observer was assigned two or three airports
from which their observation flights were to be conducted,

During individual familiarization flights at each airport, specific
terrain features and targets to be used as checkpoints during the
observation flights were sglected. This flight was also used as an
opportunity for further training of the observer in the presentation and
scoring of the observation flight scenarios, Because the terrain, weather,
traffic conditions, and targets varied considerably among the ten airports,
no attempt was made to establish a rigid flight profile which would apply
to all observation flights. But each observer flew with subjects from both
the experimental and control groups and thus acted as his own blind
control for these factors.

The behavioral test of pilot judgment was in the form of a 12 item
observation flight that could be safely and reliably presented at all
evaluation sites. (tems were selected to cover pilot activities normally
occurring during the preflight, departure, cruise and arrival phases of a
typical general aviation flight conducted under visual flight rules. Each
item required a decision by the pilot, and provided a measure of judgment
rather than of knowledge, skill or experience. The 12 items and the
criterion behavior which constituted “good judgment” for each is presented
in brief form in Table 1. A complete description of the specific
procedures for administering each item and the scoring guidelines used by
the observers appears in Appendix B.

Table 1. Observation flight items.

|. Pilot checks weather and Notices to Airmen prior to boarding
aircraft.

2. During preflight inspection, pilot detects and remedies loose
fuel tank cap, loose oil filler cap, or popped circuit breaker.

3. Prior to engine start, pilot detects and removes an empty
soft drink bottle or flashlight from cockpit floor, or removes a
hat placed on glareshield.

4. Pilot shuts down engine or directs observer to remain in
aircraft when observer indicates his intention to exit aircraft
in order to check something (eg. loose cowling) shortly after
engine start. .


Judgment Training Page 5
METHOD

The experimental design compared the performance of two groups of
newly rated private pilots on a behavioral test of judgment. The study
was conducted at ten FBOs within the FAA's Eastern Region, which
consists of the geographic area within the states of Delaware, Maryland,
New York, Pennsylvania, West Virginia, and Virginia, and the District of
Columbia. The participating FBOs are listed in Appendix A. The control
group included subjects trained at these FBOs using conventional pilot
training programs. The experimental group used subjects drawn from these
same FBOs. Their training followed the same curriculum as was used in
the control group, but with the addition of a judgment component taught by
specially trained instructors. The behavioral test was in the form of an
observation flight which was administered by specially trained observers
who were uninformed of the details of the experimental design.

Cooperating instructors at each of the ten participating FBOs were
individually trained. At some FBOs only one or two instructors were
included, while at others the entire cadre of CFs agreed to participate, A
one hour slide presentation described the rationale and philosophy behind
judgment training, reviewed the content and methods used in the
curriculum, and presented a number of vignettes which illustrated ways in
which the judgment materials could be integrated into flight instruction.
Instructors were then given the Student and Instructor Manuals to study,
and were later interviewed in order to assure their familiarity with the
curriculum. All instructors participating in the program were unpaid
volunteers and were unaware that their students’ judgment might later be
evaluated. An FAA Scientist occasionally visited the FBOs to coach
instructors and monitor the progress of the program.

The observation flights were conducted by observers who were
unaware of the overall experimental design and who had no knowledge of
which type of training each subject pilot had received. The observers
were employees of the Aircraft Owners and Pilots Association (AOPA) and
had volunteered for the project. Each held an Airline Transport Pilot
certificate, along with Certified Flight Instructor, instrument and
multi-engine ratings., and each had between 5,000 and 10,000 hours of
total flight time. During training sessions, observers were introduced to
their roles in the study. It was stressed that their primary concern was to
maintain flight safety at all times, while keeping an unobtrusive record of
the subject's performance. Training underscored that the observer's role
was to present tasks and record performance, rather than to instruct or


Judgment Training Page 18

instructors’ estimates of ground and flight times devoted to judgment
training do not, per se, imply any additional time requirements.

Instructors also reported that the judgment training consumed only
one or two hours of ground time (46%) and flight time (36%) per student.
Only a small proportion of instructors reported that the material required
more than four hours in the primary flight training program. These results
are summarized in Figure 10.

Most students (60%) reported that they spent between two and six
hours in self-study with the student manual, with the modal time
requirement quite close to four hours. Half (50%) reported spending less
than two hours of ground time reviewing judgment-related material with
their instructors, and 58% said that they had devoted less than two hours
of in-flight time to such matters, However, it is not at all clear that the
student's perception of what was and what was not “judament-related”
coincided with the instructors’ actual objectives in any given instance.
These findings are illustrated in Figure 11

MORE THAN 4 HOURS i
il 4 Ground Time per Student

Flight Time per Student

2-4HOURS FY

1-2 HOURS

LESS THAN | HOUR

NONE

0 10 20 30 40 50
PERCENT (N=39)

Figure 10. instructor estimated time requirements for judgment training.


Judgment Training Page 8

area. It was explained that half of the observation flight would be flown
at low altitudes, with the other half at cruise altitudes. Subjects were
told that the major concern during the flight was how useful the currently
used chart symbols were for anew VFR pilot. The observer explained that
the flight would proceed with the subject being asked to locate certain
targets. The route of the flight was then shown, and the subject was
reminded that he was the pilot-in-command and was to make all decisions
regarding the flight. The subject was then given the prototype Los Angeles
sectional chart along with a 22 item “National Airspace Review VFR
Prototype Chart Evaluation Form” (See Note 3). Subjects were allowed as
much time as needed to study the prototype chart and complete the
questionnaire. They then could proceed to flight planning activities and
the preflight aircraft inspection.

Observers did not volunteer detailed information about their own
flying background. If a subject inquired, the observer told them only that
he was arated pilot. If pressed for more details, the observer allowed
that he “used to be a flight instructor" but was not actively involved as
such any longer. This satisfied virtually all subjects and helped to
minimize the tendency to attribute authority figure status to the
observers. Items were presented by the observers in such a manner as to
never require them to take control of the aircraft or to divulge the true
purpose of the flight. Each itern was scored and unobtrusively recorded by
the observer at the time it was presented according to the specific
scoring guidelines using a dichotomous “good judgment” or “poor judgment”
scoring system.

Following completion of the private pilot check ride, each subject
who had received judgment training was asked to complete a 12 item
evaluation of the program (See Appendix C). Instructors completed a
similar 16 item form (See Appendix D).


Judgment Training Page 24

APPENDIX A

Participating Fixed Base Operators

City State Airport Fixed Base Operator ID
Middletown DE Summit Summit Aviation N92
Frederick MD Frederick Frederick Aviation FDK
Farmingdale NY Republic Flightways of Long Island FRG
Islip NY islip-Mac Arthur Mid-Island Air Service 1SP
Allentown PA Queen City Queen City Aviation iN9
Easton PA Easton Bradens Flying Service N43
Chesapeake VA Chesapeake Mid-Eastern Alrways W36
Lynchburg VA Lynchburg Air Virginia LYH
Manassas VA Manassas Dulles Aviation wid

Woodbridge VA Woodbridge Woodbridge Aviation ‘W22


Technical Report Documentation Page

T. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT /FAA/AM-87 /6

4. Title ond Subtitle 5. Report Date

PRIVATE PILOT JUDGMENT TRAINING May 1987

IN FLIGHT SCHOOL SETTINGS 6. Performing Organization Code
8. Performing Organization Report No.

7. Author! s)

Alan E. Diehl and Lewis F. Lester
9. Performing Organization Name ond Address 10. Work Unit No. (TRAIS)

FAA Office of Aviation Medicine

800 Independence Avenue, SW 11. Contract or Gront No.

Washington, D.C. 20591

13, Type of Report and Period Covered

12. Sponsoring Agency Name and Address
Office of Aviation Medicine
Federal Aviation Administration
800 Independence Avenue, S.W. 14. Sponsoring Agency Code
Washington, D.C. 20591

15. Supplementary Notes

16. Abstract

Pilot judgment errors have long been recognized as an important factor in aviation
accidents. Previous studies have demonstrated that specialized training procedures
can significantly reduce the number of decisional errors made by newly certified
private pilots during in-flight tests. However, the subjects in these studies
were all college-age students enrolled in full time aviation training programs
which were taught by highly motivated instructors. The present study examined
the utility of revised judgment training materials with typical private pilot
applicants in conventional flight school settings at ten fixed based operat ions
within the FAA's Eastern Region. The performance of a sample of subjects who
received judgment training was compared with that of a control group drawn from
these same FBOs. The behavioral test of judgment was in the form of an observation
flight administered by observers who were uninformed of the details of the
experimental design. -Students and instructors also completed a critique of

the program materials, The results of the study suggest that improvements in
pilot decisionmaking skills can be achieved in the less formal instructional
climate which characterizes many conventional flight school programs. The revised
judgment training program and instructional materials are acceptable to the

user community, and most participants found them to be very useful.

17. Key Words 18. Distribution Statement
sudgrent Maki Document is available to the public
Traini ing through the National Technical

aaning Information Service, Springfield,
Virginia 22161
19. Security Clossif. (of this report) 20. Security Classif. (of this page) i No. of Poges | 22, Price
Unclassified Unclassified 36

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized


a

Judgment Training Page 11

There was considerable variability in the mean observation flight
scores assigned to subjects by each of the three observers, as well as
great differences among subjects from different FBO's. Because of the
small cell frequencies and the partial confounding of observers with FBO's,
it was not possible to statistically isolate the effects of these extraneous
variables. However, these influences were examined in two ways. When
the observation flight performance of the experimental and contro} group
subjects run by each of the three observers are compared, improvements
of between four and eight percent are noted in the case of each observer.
These results are shown in Table 3.

Table 3. Mean observation flight score by observer.

Observer A Observer B Observer C
(NET 1) (N=20) (NF 14)

Control Group 476 615 73.5
Experimental Group 52.7 65.8 81.3

Moreover, when the overall analysis is restricted to those FBO's
which were represented in both the experimental and contro! groups, the
improvement in observation flight performance which results from
judgment training is noted to be even greater (t-2.22, df=28, p<.05). This
comparison is shown in Table 4

Table 4. Observation flight results for FBO's
represented in both experimental and control groups.

Correct Responses

Control Experimental
Group Group
a (PERCENT)
65.7 Mean 778
15.6 S.D. 13.6

17 N 13

These supplemental analyses suggest that the finding that judgment
training improves observation flight performance is quite reliable in spite
of the high variablility among FBO's and between observers.


Judgment Training Page 10

groups were noted in the amount of time which elapsed between the
private pilot check ride and the observation flight (t=0.54, df=35). On
average, control group subjects (N=17) were tested 5.4 months following
the check ride. Experimental group subjects (N=20) were tested after 6.0
months.

There was no significant difference between the experimental and
control groups on the scores earned when they passed the written
examination for the private pilot certificate (t=0.36, df=42). Subjects in
the control group (N=24) earned a mean score of 87.5, while experimental
droup subjects (N=20) had a mean score of 88.5 on this examination. Two
subjects, one in the control group and one in the experimental group, did
not pass the written examination on the first attempt. One subject in the
control group did not pass the private pilot practical test on the first
attempt.

Thus, any difference between the experimental and control groups in
observation flight performance cannot reasonably be attributed to
differences in age, flight experience, aeronautical knowledge, or skill.

Observation Flight Performance

The experimental group, which received judgment training, did about
10% better on the observation flight than did the control group. This
difference is statistically significant (t=2.13, df=43, p<.05S), and suggests
that the judgment training program is effective in conventional flight
school settings. However, the improvement in pilot judgment may be
considerably less in these settings than was suggested by previous
studies. Results of the observation flight are presented in Table 2.

Table 2. Observation flight results.
Correct Responses

Control} Experimental
Group Group
— (PERCENT) =
59.9 Mean 70.0
16.3 5. D. 15.4

25 N 20


Judgment Training Page 31

7. How much self-study time did you spend on the Judgment Training
Manual?

A) less than 2 hrs, B) 2-4 hrs. C) 4-6 hrs.
D) 6-8 hrs. E) more than 8 hrs.

8. How much total time did your instructor spend discussing judgment
with you in-flight?

A) less than 2 hrs, B) 2-4hrs. C) 4-6 hrs.

D) 6-8 hrs. E) more than 8 hrs.

9. How much total time did your instructor spend discussing judgment
with you on the ground?

A) less than 2 hrs. B) 2-4 hrs. C) 4-6 hrs.
D) 6-8 hrs. E) more than 8 hrs,

10. Do you feel that reading this manual and/or discussing these concepts
improved your aeronautical judgment?

A) definitely B) probably
C) uncertain D) no

11 Would you recommend the adoption of the Judgment Manual for use in
private pilot training?

A) strongly recommend B) recommend
C) recommend with hesitation D) do not recommend

12. How could this manual be improved?


Judgment Training Page 30

APPENDIX C
Student Evaluation

The Federal Aviation Administration (FAA) and the General Aviation
Manufacturers Association asked several Flight schools to examine a
prototype Pilot Judgment Training Program. We understand that you
participated in this important program, and we would be interested in your
views on its acceptability. Please circle your response and feel free to
explain it below the question, or on the back of the page if necessary. A
self-addressed, postage paid envelope is enclosed for your convenience.
Your cooperation in this critique is deeply appreciated.

1. The Judgment Training Manual in general was:
A) very useful B) moderately useful
C) slightly useful D) not at all useful

2. Six Action Ways concept (Do, No Do, Under Do, Over Do, Early Do, Late
Do) was:

A) very useful B) moderately useful

C) slightly useful D) not at all useful
3. Poor Judgment Chain concept was:

A) very useful B) moderately useful

C) slightly useful D) not at all useful

4. Three Mental Processes of Safe Flight concept (Automatic Reaction,
Problem Resolving, Repeated Reviewing) were:

A) very useful B) moderately useful

C) slightly useful D) not at all useful

S. Five Hazardous Thoughts (Anti-authority, Impulsivity, Invulnerability,
Macho, Resignation) were:

A) very useful B) moderately useful

C) slightly useful D) not at all useful

6. Chapter on Identifying and Reducing Stress was;
A) very useful B) moderately useful
C) slightly useful D) not at all useful


Judgment Training Page 7

5. Pilot requests that observer hold questions and comments
followin interruptions during preflight, taxi, or run-up
checklisf.

6. Pilot declines observer request to make a marginally safe
intersection takeoff, non-standard turn from pattern, or low
altitude turn immediately after takeoff.

7. Pilot declines observer request to overf lya araet structure
(eg. friend's house) in a built-up area by barely 1,000 feet.

8. Pilot declines observer request to execute a low steep turn
over a target structure.

9. Pilot declines observer request to fly at an unsuitably low
altitude over water, mountainous, or heavily wooded terrain.

10, Pilot adjusts altitude to comply with VFR cruising
standards following observer request to turn to new heading.

11. Pilot avoids controlled airspace or obtains ATC clearance
following observer request for flight into active Airport
Traffic Area.

12. Pilot declines observer request to fly a non-standard
pattern, directly over airport at pattern altitude, high final
approach leg, or downwind landing at uncontrolled airport

ln order to permit a realistic assessment of pilot judgment, it was
necessary to disguise the true purpose of the study. Thus, the observation
flight was described to subjects as being part of an “aviation safety
survey” being conducted by AOPA. Subjects initially received a letter
stating that volunteers were being asked to participate in a short
cross-country flight in connection with the planned revision of sectional
aeronautical charts. The incentive for these newly licensed pilots was
they would be given the opportunity to fly a rented aircraft from their
home airport free, in exchange for their participation in this study. The
letter requested that pilots call a toll-free telephone number if they
were interested in participating. | When a subject telephoned, they
generally spoke with the observer who would conduct their observation
Flight. The observer introduced himself, explained the proposal in greater
detail and, if the subject was still willing, made arrangements to meet at
the pilot's home airport.

Upon arrival at the airport, the subject was greeted by a casually
dressed observer who explained the study of the revised sectional charts
and the need for evaluations from newly rated private pilots. The subject
was told that he would first complete a questionnaire on a prototype
chart, and then fly the survey flight using the standard chart for his home


Judgment Training Page 21
IMPLICATIONS & CONCLUSIONS

Based upon the findings.of this study, the following conclusions are
offered:

1) The Student and Instructor Manuals are highly acceptable to the
user community, and most participants found them to be very useful.

2) Of the major didactic tools used in the program, the Five
Hazardous Thoughts concept, the Poor Judgment Chain model, and the
stress reduction material are most useful. They should be retained in
future versions of the judgment training program.

3) The concepts of the Six Action Ways and the Three Mental
Processes of Safe Flight may be too pedantic for general use, and might be
dropped from subsequent training manuals.

4) Participating students and instructors felt that the program
improved pilot decision-making skills.

5) Both students and instructors strongly recommended adoption of
the program, although instructors were not particularly supportive of
required judgment training.

6) The observation flight data suggest that the program is effective
in actually improving pilot judgment, even in the informal atmosphere
which characterizes most FBO training programs. However, the program's
effectiveness in these settings may be considerably less dramatic than
was suggested by the earlier studies.


Judgment Training Page 27
6.___Departure Procedures The observer requests one of the following

non-standard departure procedures:
a) a takeoff from a marginally safe intersection
b) aright (or non-standard left) turn immediately after takeoff while
still in the pattern at an uncontrolled field
C) a low (200 Feet AGL) turn immediately after takeoff

Good Judgment: pilot declines request
Poor Judgment: pilot executes potentially hazardous departure procedure

NOTE: OBSERVER MUST NOT PERMIT THE PILOT TO EXECUTE A HAZARDOUS
DEPARTURE

2__Low Flight Over Obstruction The observer requests a flight path and
altitude which will take the aircraft barely 1,000 feet above some object
or building in a built-up area

Good Judgment: pilot notes conflict and either climbs or diverts in order
to pass the object at a safe distance

Poor Judgment: pilot flies low over vertical obstruction

NOTE: OBSERVER MUST NOT PERMIT HAZARDOUSLY LOW FLIGHT OVER
OBSTRUCTION

8. __Low Steep Turn Over Building The observer requests that the pilot
Circle above a certain building so that the observer can take a photograph.
The observer requests an increasingly steeper turn until a 45 degree angle
of bank is established.

Good Judgment: pilot climbs to a safe altitude, or does not exceed a 30
degree angle of bank at low altitude, or declines request

Poor Judgment: pilot attempts to execute a hazardous low steep turn

NOTE: OBSERVER MUST NOT PERMIT A HAZARDOUSLY LOW/STEEP TURN


Judgment Training Page 3

A second Transport Canada study was undertaken in order to
determine whether the use of the self-paced student manual could improve
pilot decision-making skills without the involvement of flight instructors
(Buch and Diehl, 1983). This study, conducted at Confederation College of
Applied Arts and Technology (Thunder Bay, Ontario) used a group of 17
university students. All were licensed private pilots with “relatively
little flight time". Subjects were first evaluated using a well-disguised
observation flight procedure, then given the self-paced manuals and
instructed to complete them on their own. Following the self-study
program, a second observation flight was scheduled. The use of the
judgment training manuals alone resulted in a significant improvement in
observation flight performance, from 53% on the pretest to 62% on the
post-test.

Another Canadian study, this one on the effectiveness and permanency
of judgment training over time, is currently under way at a community
college in Chicoutimi, Quebec (Buch and de Bagheera, 1985). In this study,
subjects’ aeronautical decision-making skills will be evaluated up to three
years after their initial judgment training.

While pilot judgment and decision-making may be related to
relatively inflexible aspects of pilot personality or to the interaction of
personal and situational variables (Lester and Bombaci, 1984), these
experiments strongly suggest that the use of the judgment training
curriculum can significantly improve pilot performance and enhance
aviation safety. However, there were a number of limitations to the
studies which compromise the external validity of these findings. Most of
the subjects were young males who had only recently completed their
secondary education. Both the ERAU students and the Canadian Air Cadets
were enrolled in highly concentrated aviation training programs. Most of
the Air Cadets already held glider licenses, and all of the subjects in the
Confederation College study were licensed private pilots at the time the
judgment training was begun.

It was, therefore, felt necessary to demonstrate that the judgment
training curriculum would be effective with the general population of ab
initio student pilots and in the type of training environments typically
encountered at operational flight schools. Further work was also needed
in order to produce training materials which were acceptable to students
and instructors in these settings.


Judgment Training Page 14

VERV USEFUL ULL

MODERATELY USEFUL

SLUG US| ~ Fy
SUGHTLY USEFUL instructors (N=42)

Students (N=25)

NOT AT ALL USEFUL

0 10 20 40 40 50 60 70
PERCENT

Figure 5. Ratings of stress reduction material.

VLU

MODERATELY USEFUL

LLL LLL

SUGHTLY USEFUL instructors (N=42)

Students (N=25)

NOT AT ALL USEFUL

9 10 20 30 40 50 60
PERCENT

Figure 6. Ratings of Poor Judgment Chain concept.


Judgment Training Page 16

AS may be seen, almost half (48%) of the student pilots rate the Five
Hazardous Thoughts concept as “very useful". The concept of the Poor
Judgment Chain is also given high ratings by the students, with 42%
considering this concept to be “very useful”. However, 16% of the student
sample indicate that these two concepts have been of little or no value in
their training. Forty-four percent of the students rate the chapter on
stress reduction as “very useful", although more than one third (36%) place
little value on this material. The Six Action Ways and the Three Mental
Processes of Safe Flight are given markedly lower ratings by the students.
Although one fifth (21%) of the sample view the Six Action Ways as "very
useful", an equal proportion report it to have little or no utility. And while
one third (33%) of the students rate the Three Mental Processes of Safe
Flight as “very useful", an even greater proportion (42%) see little worth
in this concept.

Almost two-thirds of the instructor sample rate both the stress
reduction chapter (64%) and the concept of the Five Hazardous Thoughts
(65%) as "very useful". No low ratings at all are given to the stress
reduction material, although 8% of the instructors consider the Hazardous
Thoughts to be only “slightly useful’. The Poor Judgment Chain model is
rated as "very useful” by more than half (55%) of the instructors, with 7%
viewing it as only “slightly useful". Relatively lower ratings are given to
the concepts of Three Mental Processes of Safe Flight and the Six Action
Ways. The’ Mental Processes are rated “very useful" by 41% of the
instructors, with 12% noting that they are "slightly useful". And while
over one third (36%) of the instructor sample consider the Action Ways to
be “very useful”, an almost equal proportion (31%) see only slight utility to
this concept.

The ratings assigned to each component by each respondent were
scaled, and the mean ratings are shown in Figure 9. Although instructors
gave consistently higher ratings to each component than did the student
pilots, there was general agreement as to the relatively greater value of
the stress reduction material, the Hazardous Thoughts concept, and the
Poor Judgment Chain model. Respondents also agreed on the lesser value
of the concepts of the Three Mental Process of Safe Flight and the Six
Action Ways, and their utility in the judgment training program could be
questioned.


Judgment Training Page 2

enrolled in an ERAU flight program and had a mean age of 19 years. One
group of subjects was trained under the conventional ERAU primary flight
program, while a second group's flight and ground training incorporated the
newly developed judgment materials. Both groups were evaluated on the
observation flight, and the group which had received judgment training did
significantly better than the control group on this behavioral test.

In 1982, the FAA entered into a joint research agreement with
Transport Canada to further examine the ERAU prototype materials and to
independently evaluate judgment training in the Canadian pilot population.
Several experiments have thus far been conducted under the provisions of
that agreement.

The subjects in the initial Canadian study were civilian Air Cadets
participating in a summer flight training program in the Ontario Region.
All were in their late teens and early twenties, and more than half of the
subjects already held glider licenses. This study used geographically
isolated experimental and control groups, and the observation flight was
administered under circumstances which permitted its true purpose to be
better disguised than in the ERAU study. Here too, those subjects who
received the judgment training did significantly better on the observation
flight than did control subjects (Buch, 1982; Buch and Dieh}, 1984). The
results of both the ERAU and Air Cadet studies are shown in Figure |.

CANADIAN AIR CADETS

N50) LZ, YY; WY Judgment

Training

Conventional

ERAU STUDENTS. (N#50) Training
VMMMM000,

0 10 20 30 40 50 60 70 80 90
PERCENT

Figure 1. Results of ERAU and Canadian Air Cadet Studies.


Judgment Training Page 25

APPENDIX B

Observation Flight
Administration & Scoring Guidelines

|. Flight Information During the mission preview, the observer indicates
to the pilot that the flight will proceed to a specific destination airport.
If possible, this field should be one which has a currently valid NOTAM
and/or one with which the pilot is relatively unfamiliar. As the observer
and pilot later walk to the aircraft, the observer will casually inquire if
there are any NOTAMs on the destination airport and/or if the weather will
be all right for the flight.

Good Judgment: pilot indicates that he has checked an appropriate source
for NOTAMs and/or weather

Poor Judgment: pilot fails to check NOTAMs and/or weather prior to
boarding the aircraft

2._Aircraft Condition Prior to the time that the pilot arrives at the
airport, or while he is completing the aeronautical chart survey form, the
observer:

a) loosens, but does not remove, one of the fuel tank caps

b) loosens, but does not remove, the oil fill cap/dip stick

C) pulls a fuse or circuit breaker

Good Judgment: the pilot detects and remedies the defect during the
preflight inspection prior to boarding aircraft

Poor Judgment: the pilot fails to detect and remedy the defect prior to
boarding aircraft

NOTE: IF UNDETECTED BY THE PILOT, THE OBSERVER MUST REMEDY THE
DEFECT PRIOR TO BOARDING AIRCRAFT

3._Cabin Hazard Prior to the time that the pilot arrives at the airport, or
while he is completing the aeronautical chart survey form, the observer
places a completely empty soft drink bottle on the Cockpit floor in front of
the right seat halfway between the front of the seat and the rudder pedals.


Judgment Training Page 1

PRIVATE PILOT JUDGMENT TRAINING
IN FLIGHT SCHOOL SETTINGS

INTRODUCTION

In 1976, the Federal Aviation Administration (FAA) contracted with
University of Illinois researchers to investigate the degree to which
faulty pilot judgment contributed to civil aviation accidents and to
determine whether and how pilot judgment might be taught and evaluated.
This study reviewed all general aviation accident data between 1970 and
1974, and concluded that faulty pilot decisional activities were involved
in 35% of all nonfatal general aviation accidents and in 52% of all fatal
accidents (Jensen and Benel, 1977). The authors offered the following
two-component definition of judgment:

|. The ability to search for and establish the relevance of all
available information regarding a situation, to specify
alternative courses of action, and to determine expected
outcomes from each alternative.

2. The motivation to choose and authoritatively execute a
suitable course of action within the time frame permitted by
the situation (1977, p. 34).

These researchers concluded that pilot judgment could be taught and
objectively evaluated, and they proposed some ways in which this could be
accomplished. Two years later, the FAA contracted with Embry-Riddle
Aeronautical University, Daytona Beach, Florida (ERAU) to develop
judgment training materials for student and instructor pilots, devise
procedures for the objective measurement of pilot judgment, and
demonstrate the effectiveness of such training. A revised definition of
pilot judgment was developed by this group:

Pilot judgment is the mental process by which the pilot
recognizes, analyzes, and evaluates information regardin
himself, the aircraft, and the outside environment. The fina
step in the process is to make a decision pertaining to the safe
operation of the aircraft and to implement the decision in a
timely manner (1982, p. 4).

The ERAU project (Berlin, Gruber, Holmes, Jensen, Lau, Mills and
O'Kane, 1982) produced prototype student and instructor pilot training
manuals, and developed an observation flight protoco! to measure pilot
judgment. Using this technique, the ERAU team then carried out a limited
validation study of the training materials. The subjects were ali students


Judgment Training Page 33

9. How much self-study time did you spend (reading the Student and
Instructor Manuals, planning in-flight exercises, etc.) in order to permit
you to teach the judgment concepts? Do NOT include the time spent in the
training sessions with FAA scientists.

A) less than 2 hrs, B) 2-4 hrs. C) 4-6 hrs.

D) 6-8 hrs. E) more than 8 hrs.

10. About how much ground time did you spend teaching judgment to each
of your students?

A) None B) less than 1 hr. C) 1-2 hrs.

D) 2-4 hrs. E) more than 4 hrs.

11. About how much flight time did you spend teaching judgment to each
of your students?

A) None B) less than 1 hr. C) 1-2 hrs.

D) 2-4 hrs. E) more than 4 hrs.

12. Would you recommend the voluntary use of these manuals for private
pilot training?

A) strongly recommend B) recommend

C) recommend with hesitation D) do not recommend

13. Do you recommend that FAA require a Program such as this to teach
and evaluate judgment skills of student pilots?

A) strongly recommend B) recommend

C) recommend with hesitation D) do not recommend

14, Do you think participating in this program improved your "typical"
students’ judgment skills?

A) definitely B) probably

C) unsure D) no

15. What proportion of your students who participated in this program do
you feel benefited from this training?

A) all B) most C) some

D) few E) none

16. How could this program be improved?


Judgment Training Page 28

9, Low Flight Over Unsuitable Terrain The observer requests a flight

path and altitude over unsuitable terrain such that the ability to execute a
safe emergency landing or power-off glide to shore would be seriously
compromised.

a) heavily wooded terrain

b) mountainous terrain

C) large body of water

Good Judgment: pilot declines request or climbs to a safe altitude
Poor Judgment: pilot flies low over unsuitable terrain

NOTE: OBSERVER MUST NOT PERMIT HAZARDOUSLY LOW FLIGHT OVER
UNSUITABLE TERRAIN

10. VFR Cruising Altitude The observer requests a flight path and

altitude consistent with VFR cruising altitude regulations, and requests
the pilot to locate a specific terrain feature along this flight path. Upon
reaching the terrain feature, the observer requests the pilot to fly a new
heading such that a climb/descent is required in order to maintain an
appropriate VFR cruising altitude. The observer makes no mention of
altitude however.

Good Judgment: pilot notes conflict and either climbs or descends to an
appropriate VFR cruising altitude

Poor Judgment: pilot turns to the new heading but fails to change altitude
to comply with VFR cruising altitude regulations

NOTE: OBSERVER MUST NOT PERMIT A HAZARDOUS VIOLATION OF VFR
CRUISING ALTITUDE REGULATIONS

1}. Controlied Airspace Encroachment The observer requests a flight

path and altitude which brings the aircraft marginally close to the
horizontal or vertical limits of an active Airport Traffic Area of Terminal
Control Area, while the aircraft is not in radio contact with the
controlling facility.


Judgment Training Page 12
men! ini rial

Twenty-five pilots who received judgment training provided critiques
of the training program and materials. Forty-two participating
instructors also evaluated the program. Because the Student Manual
represented a major revision of the ones used in both the ERAU and Air
Cadet studies, it was particularly important to establish its user
acceptability in this manner. These student and instructor critiques were
quite helpful in identifying the strengths and weaknesses of the various
program components. As may be seen in Figure 2, the Student Manual was
very well received. More than half of both the student and instructor
samples indicated that the Student Manual was "very useful”, while 12% of
the students, and none of the instructors, felt that it was “not at all
useful",

Instructor evaluations of their teaching materials was also favorable,
but not quite as positive as in the case of the Student Manual. As may be
seen in Figure 3, 43% felt that the Instructor Manual as a whole was "very
useful’, while 32% gave high ratings to the in-flight Lesson Plans and the
Judgment Training Scenarios. However, 10% of the respondents felt that
the Instructor Manual was only “slightly useful’, and 14% saw little value
in the Lesson Plans and Judgment Training Scenarios.

VERY USEFUL

MODERATELY USEFUL tS

SLIGHTLY USEFUL

Instructors (N=42)

Students (N=25)

SA

NOT AT ALL USEFUL

9 “10 20 30 40 50 60
PERCENT
Figure 2 Ratings of Student Manual


Judgment Training Page 15

VERY USEFUL F

WLM hs

MODERATELY USEFUL

MLL :

: Instructors
SLIGHTLY USEFUL (N=42)
VMMMMU“IL
Students
E (N=25)
NOT AT ALL USEFUL
0 10 20 30 40 50 60

PERCENT

Figure 7. Ratings of Six Action Ways concept.

VERY USEFUL
MODERATELY USEFUL
VLU,
SLIGHTLY USEFUL Instructors (N=42)
Students (N=25)
NOT AT ALL USEFUL F*
0 10 20 30 40 50

PERCENT

Figure 8. Ratings of Three Mental Processes of Safe Flight concept.