**Air Carrier**

**Training**

**Research Review**

**June 1997**

**Air Carrier Training Research**

The FAA Office of the Chief Scientist for Human Factors (AAR‑100)
directs an air carrier training research program centering on methods
for effective pilot training and assessment. Semi-annual research
reviews are conducted by this office. The following report summarizes
the June 1997 review.

**Introduction**

Although basic technical and Crew Resource Management (CRM) concepts are
widely accepted, much remains to be learned regarding the appropriate
methods for effective training and valid and reliable assessment of
training programs. The general research philosophy guiding efforts to
improve training and assessment is that research must consider distinct
segments of aviation training systems. Individuals comprising the crew,
instructors who train and evaluate crews in the classroom, the simulator
and on the line as well as the management personnel responsible for the
safety climate of the carriers should all be considered. Additionally,
this research must regard the variables important to Line Oriented
Flight Training (LOFT) development, implementation and evaluation. Thus,
this research centers on (1) crew training and assessment, (2)
instructor training, (3) LOFT development strategies, and (4)
organizational and systematic influences on pilot performance.

Eleana Edens, Ph.D.

AAR‑100

\(202\) 267‑7867

\(703\) 960‑1085

EEdens@Mail.FAA.GOV

**Project Title:** *Identification of Hard to Train CRM Skills*

**Performing Agency:** University of New Mexico, Albuquerque

**Background:** Traditionally, CRM training has focused on the
high‑level CRM concepts or elements of pilot performance. The training
and assessment of high‑level concepts is fraught with many problems.
Among them, the validity of course content is questionable and the
reliability and validity of CRM assessment has not been established.
Prior research has shown that assessment of observable behaviors, rather
than global concepts, especially within LOFT event sets, can improve the
reliability of CRM assessment. Therefore, observable behaviors have
provided a more meaningful and lower level of detail for the CRM
assessment process. However, the research and operational community have
not yet developed a comparable level of detail for the CRM training
process. It is impractical to try to train observable behaviors. Thus,
at present, it is likely that training would be on the global level but
assessment could take place at the observable behavior level. This
training paradigm, although improved from all CRM training activities
being at the global level, still lacks the training vigor employed, for
example, in technical training. An approach to solving this dilemma is
to attempt to identify the knowledge, skills and abilities underlying
CRM concepts and then to train at that level. Observable behaviors could
be identified and assessed for skills in situation specific training
events.

**Project Description:** This research project has been investigating
two distinct segments of air carrier training, skill analysis and
reliable performance assessment.

Work is being conducted with major carriers to identify the key CRM
skills that can be trained systematically in crews undergoing transition
and recurrent training. CRM concepts, or CRM knowledge, that are
important across several air carriers were identified. These are the
concepts that are less affected by individual corporate climate and
should be the foundation of CRM knowledge at most US based air carriers.
Additionally, a general framework for skill analysis was developed that
incorporated an analysis methodology with skill categories that could
have significant training implications for air carriers. The skill types
under investigation are: 1) procedural skills, 2) representational
skills, and 3) strategies. It may be possible to design skill training,
employing this taxonomy. This idea will be explored in follow up
research.

Work continues in the validation of methods that can be used by air
carriers to identify and link CRM skills within their fleet‑specific
Advanced Qualification Program (AQP) databases. Currently, most carriers
insert place holders similar to high‑level CRM elements in their AQP
databases, rather than enter detailed CRM skills. Once developed, these
new methods will allow air carriers to systematically identify
appropriate CRM skills and to link those skills with the observable
behaviors presently used in the CRM assessment process.

Efforts to reliably assess pilot performance, during training, have
traditionally been less than successful. During FY96 and FY97, this
research team has been working with an air carrier to improve training
assessment by designing rater training and new data collection systems.
This work focuses on rater calibration, development of grade sheets,
development of videos used in the calibration sessions, presentations to
the instructors on the importance of quality data, collection and
analysis of the calibration data, and reporting results. Several
calibration studies have been completed and additional sessions are
planned. Thus far, the results of these calibration studies show (1) how
minor changes in the wording of observed behaviors listed on gradesheets
can have a significant effect on ratings, (2) large individual
differences among raters, (3) higher inter-referent than inter-rater
reliability, and (4) a significant increase in reliability when
improvements to the calibration sessions were implemented (wording of
the gradesheet, quality of the video, better briefing of the event
sets). The frequency of the higher inter-referent reliability is
important to note because there are differences among carriers in their
views of whether a referent should be established and instructors
trained to the referent or whether instructor's inter-rater reliability
is adequate. AAR-100 is investigating this question further through work
being done by George Mason University at several other carriers
employing the inter-rater reliability methodology.

This team has begun work on the development of a PC-based system that
will effectively present a complete calibration session to individual
evaluators. Upon entering a pin, the system will 1) provide the
evaluator with a prebriefing of the LOE that is to be presented; 2)
display the gradesheet that is to be used; 3) present an audio/video of
the first event set; 4) collect ratings to the event set and the
observed behaviors; 5) continue to present and collect ratings for the
entire LOFT or LOE; 6) analyze the evaluators ratings and present
evaluator's performance relative to other evaluators and a referent
score based on the fleet's qualification standards; 7) allow the
evaluator to select any segment of the video and receive instructions as
to what the qualification standards were and why a specific rating was
deemed appropriate.

This project has produced a document describing what it means to have
quality data, why quality data are necessary for training under AQP, and
the methods for achieving quality in performance assessment. This
document is intended for evaluators and instructor/evaluator supervisors
and is available to industry.

This project is also examining the statistical properties of a large
amount of maneuver validation data. The goal is to be able to identify
statistically significant changes in maneuver validation performance
across time in general, and more specifically in first-look performance.

**Products:** Skill training and assessment strategies incorporated into
training guidelines; Instructor/evaluator training guidelines

**Project Title:** *Identifying Leadership and Followership Skills in
CRM*

**Performing Agency:** Western Michigan University

**Background:** The University of New Mexico research described earlier
will center on CRM components: Crew Communication, Crew Climate,
Decision Making, and Workload Management. Western Michigan University
will provide the CRM leadership component skill analysis information.

Historically, leadership has been viewed as a linear, one‑way (i.e.,
downward) action with the primary aim of task accomplishment. This
perspective has dominated the cockpit of air carriers. However, as the
role of the modern cockpit crew changes, the skills required of the crew
members change also. It is now thought that leadership is an activity
that involves both leaders and followers as they interact to accomplish
goals and that followership skills are as important as leadership skills
to the safe and efficient performance of flight crews. It is unclear how
to train leadership/followership as the skills underlying these concepts
have not been identified.

**Project description:** This project will analyze cockpit performance
to determine the behavioral components of leadership/followership
skills. When these are identified, this research will provide training
guidelines. This project will provide the skill analysis of cockpit
leadership and be employed in the more general skill study at the
University of New Mexico.

In phase I of this two‑year project, a model of cockpit leadership was
developed which guided the skill analysis. This model of effective and
efficient cockpit operations points out that the crew must use many
skills to deal effectively with environmental variables largely out of
their control. These variables include the regulatory environment, the
corporate environment, the market environment, and the physical
environment. Analyses are being conducted based on the assumptions that
(1) a minimum amount of leadership/followership is required to achieve a
safe flight, (2) individuals constantly oscillate between leader and
follower roles, and (3) weak leader or follower skills will be
compensated for by other crew members. Preliminary analysis is complete
and reveals that to deal effectively within the framework described in
the model, crews must possess skills in: envisioning, modeling,
receptiveness, influence, adaptability, and initiative. The analysis
also showed that these skills are common to both leadership and
followership.

During FY 96, data from a line audit at a major carrier revealed that
the components described in the model are related to overall crew
effectiveness. In general, the more severe an abnormal situation the
less effective the crew was in exhibiting leadership/followership
skills. Captains tended to be less likely to articulate a vision for the
flight, meet company standards or obtain commitment from other
crewmembers or be adaptable. Both Captains and First Officers were less
likely to initiate actions in response to an operational deficiency when
the severity of abnormals was high. When handling a complex situation,
but not an abnormal, these crews displayed \"outstanding\"
leadership/followership skills. These ratings were not dependent on
position or time together as a crew.

The data also showed that Captains do, in fact, set the tone in the
cockpit. When Captains articulated a vision for the flight the First
Officer would initiate a response to an operational deficiency without
further direction from the Captain. When Captains exhibited good conduct
and high standards, First Officers exhibited similar behavior. When
Captains were receptive, First Officers were likewise receptive.

Maintaining vigilance during the flight seems to be dependent on each
crew member at different phases. During predeparture, takeoff, climb and
cruise, vigilance was related with the Captain\'s envisioning, modeling
and receptiveness. However, during the descent and approach phases,
vigilance was related to the First Officer\'s conduct and standards.­­­­­­­­­­­­­­­­­­­­­­­­­­­­

Workload and task distribution was dependent on the Captain\'s conduct,
standards, and receptiveness. Establishing guidelines for automated
systems for all phases of flight were related to the Captain\'s
articulating a vision, conduct and standards (modeling), receptiveness
and use of interpersonal skills to obtain commitment from others.

The next phase of this study is to determine the effects of
leadership/followership skills on cockpit performance in LOE\'s and
another major line audit. Data collection began January, 1997. When
completed, guidelines for an integrated approach to training will be
developed. It is likely that the skills identified in the preliminary
analysis will be modified by future work.

Challenges facing the development of curriculum have been identified,
and include the development of classroom exercises extending beyond
role-playing, the creation of event sets that address critical
leadership/followership skills, and addressing company philosophy and
policy issues. During FY 97, curriculum has been developed for Alaska
Airlines recurrent pilot training.

**Products:** Leadership/Followership training guidelines; curriculum
materials for addressing leadership/followership awareness and skills
training; and a web site for distribution of curriculum development
materials.

**Project Title:** *Analysis of Cockpit Management System in Multiple
Carrier Environments*

**Performing Agency:** George Mason University

**Background:** It is widely believed that CRM training can result in
significant improvements in flightcrew performance and flight safety.
Historically, it has been possible to assess the effects of CRM training
on pilot attitudes regarding the general management of cockpit duties
with self-reporting inventories administered pre- and post‑CRM training.
It is thought that a change in pilot attitude will result in a change in
behavior in the cockpit. It has also been possible to subjectively
evaluate the effects of CRM training on pilot performance in LOFT and on
the Line by noting the differences in CRM performance between crews that
have had CRM training and the crews that have not. Further, performance
differences between air carriers and between fleets within one air
carrier have been investigated. While this approach to assessment has
contributed to the advancement of CRM, it is hampered by the practical
constraints of self report and global measurement.

**Project Description:** This research project will help determine the
most efficient method to train and assess CRM and whether conceivable
performance differences pre‑ and post‑ CRM training can be empirically
substantiated. This research project is being conducted with one
regional carrier and one major carrier. This project focuses on the
design and implementation of a prototype CRM training program based on a
task analysis methodology. The research will evaluate the effects of
this specific CRM training on the Line and LOFT performance of crews.
This study also integrates CRM performance requirements or procedures
into the standard operating procedures of the air carrier. CRM
procedures are the implementation of specific calls, checks, and/or
guidance into one or more of the following: normal checklists, Quick
Reference Handbook, Abnormal/Emergency Procedures, Flight Standards
Manual, and additional Job Aids. This can be viewed as translating
critical CRM principles into CRM procedures.

In developing the prototype, advanced CRM course data from line
operations, instructor comments and ratings, and findings from the NTSB
commuter safety study were used as the basis for the CRM training
course. These data were incorporated into a proceduralized management
system specifically tailored to the needs of this regional carrier. This
advanced CRM course is being given to pilots in the research fleet.
Their performance in a LOE was assessed prior to this training and will
be assessed after the training course is completed with an appropriate
time lapse. The central focus of this research is to determine if a
proceduralized system will increase pilot performance, thereby
increasing safety. This research has thus far proceduralized: Team
Management, Crew Communication, Decision Making, and Situation Awareness
in the operational environment of the regional carrier.

The proceduralized CRM training course is complete, and has been given
to flight crews. The Quick Reference Handbook has been rewritten to
reflect proceduralized CRM. The normal procedures have been rewritten to
incorporate the proceduralized CRM. Baseline performance data are being
collected during LOEs and preliminary analyses are underway.

One analysis of the baseline performance data showed some interesting
results. In the research process, it was necessary to train evaluators
to collect data during the LOEs. The evaluators are line pilots working
in the training center. One of the critical questions centered on
whether the evaluator training was successful and would result in
reliable and valid CRM assessment. The analysis showed that the LOE
assessment process, which was taught in the evaluator training, going
from ratings of observable behaviors to technical and CRM ratings, to
overall Pilot In Command (PIC), Second In Command (SIC) and Crew ratings
for each event set was confirmed. Observable behavior ratings
significantly contributed to both technical and CRM evaluations of each
crew. This confirms that with appropriate training and calibration,
evaluators can use specific, observable behaviors as the basis for
judgments which should increase the objective component of CRM
evaluation in an LOE and ensure more fair, unbiased evaluations of each
pilot.

CRM ratings, across event sets, as well as technical ratings
significantly contributed to the overall evaluation of PIC and SIC. This
confirms that evaluators, when properly trained and calibrated, can
consistently integrate CRM evaluation with technical performance to
obtain overall evaluation of crew members on each event set. This is
evidence that the LOE evaluation was reliably tapping CRM as well as
technical performance. This is important because in the past there were
questions as to whether CRM ratings were just a reflection of technical
performance. It has been suggested that evaluators could not or would
not rate CRM behavior as a separate performance domain, but part of the
entire crew performance.

Additionally, the analysis showed that Captains and First Officers were
evaluated differently. In evaluations of the entire LOE, CRM performance
was more important than technical performance for the PIC, while
technical performance was more important than CRM performance in rating
the SIC. These differences in evaluations are congruent with holding the
PIC more responsible for CRM as part of the Captain\'s role. These
findings are also congruent with previous findings that Captains state
that the most important feature of a good first officer is technical
competence and First Officers state the most important feature of a good
Captain is CRM competence.

Further analysis showed stronger relationships between Captain
evaluation and crew performance than between First Officers and crew
performance. This provided additional evidence that Captains are held
more responsible for crew performance, particularly CRM performance.
This empirically confirms that Captains are the focal point for CRM and
crew performance and suggests additional training on establishing and
maintaining CRM should be performed in upgrade training when a First
Officer transitions to Captain.

LOE data were compared with the data from traditional proficiency check
of specific maneuvers. The proficiency check evaluations could predict
about 50% of the overall evaluations of PIC and SIC on the LOE. This
confirms that the technical proficiency emphasized by the traditional
check is relevant to LOE performance. However, the remaining 50% of the
variance unique to the LOE confirms that the LOE is also tapping a
distinct type of performance in addition to technical proficiency. More
detailed evaluation of proficiency check assessments showed that only
evaluations of the SIC do not predict LOE results. Additional analyses
are required to better understand the role of rating SIC performance in
proficiency checks.

Analysis of different event sets found significant differences in the
performance of the targeted observable behaviors. For specific event
sets, the targeted behaviors (e.g. specific briefings) were observed
significantly less often. Isolation of performance problems across crews
can be used to rationally and efficiently change pilot training. The
lack of specific observable behaviors can be tied to specific
instructional objectives using an instructional database such as the
model AQP database. The database can also be used to find the specific
training components for these objectives that must be strengthened. This
analysis shows that isolating poor performance across crews can be used
to directly change relevant components of training.

During FY 96, work was initiated with several carriers to identify
appropriate questions, data collection strategies, and statistical
methods to analyze pilot performance data. This data collection system
will allow carriers to evaluate their AQP training system effectiveness
in many dimensions. The overall goal of this part of the project is to
determine the appropriate data necessary to extend training intervals
for AQP programs.

Also in FY 96, a formal workshop was developed for regional carriers to
introduce the instructor/evaluator training methods used in this
research project and to introduce the data analysis methodology. The
workshop was attended by 50 air carrier representatives from 30
different carriers.

During FY97, data collection was completed and new LOE and associated
forms for Year 3 were developed. IRR sessions for new LOE were conducted
in May-June 1997. A second IRR workshop is being planned. The future
work will center on developing and evaluating CRM training and future
AQP data analysis.

Another area of this research is IRR Indoctrination Training for
Instructor/Evaluators (I/E). The overall goal of this project is to
assist a carrier with incorporating interrater reliability training
strategies into existing instructor/evaluator (I/E) indoctrination
training. This project involves: selecting and evaluating existing
videotapes of crew performance to use as \"gold\" standards for the I/E
indoctrination class; developing behaviorally‑based grading sheets for
the selected video tapes; and collecting and analyzing data from at
least one I/E indoctrination class on the levels of agreement (i.e.,
interrater reliability) that emerge as a result of training. Results
from this investigation will be summarized in an FAA technical report
that documents the extent to which the I/E indoctrination class can be
used to do the initial rater agreement training and to foster
observational skills for LOFT debriefing.

**Products:** A cockpit management system; Research report on
performance differences due to proceduralized CRM training; Industry
papers on Inter-rater reliability training; Training data analysis
software

**Project Title:** *Crew Resource Management: Design and Evaluation of
Human Factors Training in Aviation.*

**Performing Agency:** The University of Texas, Austin

**Background:** This research consists of several projects that center
on evaluating the impact of CRM and LOFT and developing methods to
optimize training and performance evaluation. This research group has
maintained an international, longitudinal database of pilot performance
since the late 1980\'s. Prior findings from this work indicate that
formal CRM programs combined with LOFT have a significant, measurable
positive effect on crew behavior and attitudes and by inference on
system safety. However, the data also indicate that all human factors
problems in aviation have not been solved by the current CRM and LOFT
programs. Specifically, the following negative outcomes have been
documented through this research program:

\(1\) There is great variability in the impact of programs and in
observed crew performance, even after implementation of CRM training.
Incomplete acceptance of CRM concepts and practices by instructors and
evaluators has been identified as one source of problems.

\(2\) In the absence of continuing reinforcement, behavior and attitudes
regress almost to pre-training levels.

\(3\) LOFT programs in several organizations show great variability in
execution and impact.

\(4\) Considerable controversy remains regarding the evaluation of
individual and team performance, as required of organizations
participating in the Advanced Qualification Program (AQP).

\(5\) There is an absence of reliable data on the human factors of
incidents occurring in the aviation system.

\(6\) Recent investigations of major accidents, including air carrier
crashes, have shown that organizational cultures have a profound impact
on operations and on the effectiveness of human factors programs.

\(7\) Human Factors training and awareness in Air Traffic Control (ACT)
has lagged behind programs in air carrier operations.

\(8\) Human Factors aspects of cockpit automation are not being
addressed in most CRM/LOFT programs and have been identified as critical
in a number of accidents.

**Project Description:** The research program is designed to build on
current knowledge and to address the issues noted above. This research
program has developed and maintains a massive longitudinal database on
the effectiveness of CRM in the Aviation industry. Operational line and
LOFT observation data comprise the database. Confidential data are
entered into the database and reports are delivered to each
participating carrier. This information is used to remedy the weaknesses
apparent in the observations or to highlight strengths. It is common
practice for carriers to use these reports on which to base the
following year's recurrent training. This research group disseminates
research findings to scientific meetings and is widely published each
year.

In addition to collecting line and LOFT data from a cross section of air
carriers, work with Southwest Airlines and Conquest Airlines currently
is underway to explore concepts and issues important to each carrier.

The research group is collecting data from many different air carrier
cultures to determine the effects of culture on CRM performance. The
goal is to develop training guidance for carriers employing
cross‑cultural cockpits or for carriers operating with non‑Western
cockpits.

Because flight deck automated systems can execute most control functions
performed by pilots, the flight management computer has been likened to
an \"electronic crewmember.\" This view of automation has led to
speculation that the manner in which crew members interact with the FMC
may influence crew behavior and performance. The link between
interpersonal flight management attitudes and performance in aviation
suggests that attitudes regarding flight deck automation are likely to
be related to pilots\' use of automated systems and overall performance.
This research project has collected data from more than 10,000 pilots
from more than 10 cultures addressing attitudes towards automation.
Analysis of these data showed significant fleet, organizational and
national differences in both general attitudes regarding the conduct of
the flight and attitudes regarding the use of flight deck automation.
Currently, this research project is investigating these issues with an
expanded methodology and will further explore the link between national
differences in attitudes toward automation and a multi‑dimensional model
of national culture.

The research is correlating the behavioral markers developed under this
project with a world wide accident database. This is being done in order
to determine the influence of certain cockpit behaviors and accidents
worldwide rather than just in the United States.

A large number of direct observations along with instructor and line
check airmen comments have been analyzed regarding incidents that were
successfully and safely resolved. These data can be used to develop
training scenarios and are available to air carrier training
departments.

A segment of the work done recently by this program was to analyze the
particular CRM requirements of the regional carriers coming under Part
121 (NPRM 94‑35) which requires CRM training for pilots, flight
attendants, and dispatchers (rule issued December 14, 1995). This
research suggests that regional CRM programs should focus on specific
elements of the commuter environment. The programs must address low
experience level and high turnover of both pilots and flight attendants.
Also, pilots may be thrust into automated equipment without extended
training in non‑automated equipment which may affect trust in the
automated systems. Also, these programs must address the environmental
variables: high frequency, low altitude, turbulence and small airports.

One project of this research program focuses on the effectiveness of
LOFT. The goal is to determine the key components for high quality LOFT.
Thus, this research seeks to determine what set of LOFT attributes best
predicts overall LOFT quality and to find the relative importance of
each predictor. In addressing this issue, scenarios were analyzed for
appropriate workload, creative problem solving required, problem
realism, and simulator mechanical condition. Briefings were analyzed for
how the instructor integrated the crew into briefings, handled a review
of CRM concepts, and conveyed enthusiasm. Scenario execution was also
analyzed for simulator operation, flight attendant roles, adherence to
script and radio calls. Debriefings were analyzed for crew involvement
with self‑appraisals, the comprehensiveness of the debrief, and the
integration of technical and CRM skills. Findings showed that overall,
the briefing and debriefing quality were the most important aspect of
LOFT quality, followed by the quality of the scenario and then its
execution. These findings will be made available to the Airline
Transport Association subcommittee on LOFT development.

This research program investigated the role of group dynamics and
performance in the Air Traffic Control system. This project was
conducted with the Southwest Region to investigate the relationships
between the behavior of organizations, work teams and individuals and
three possible operational outcomes: loss of separation, acceptable
performance (absence of remarkable event) or outstanding performance in
the presence of increased challenge. Data from loss of separation events
and acceptable performance events have been collected. Data from
outstanding performance in increased challenge events are currently
being collected. Analyses of the data to date indicate that differences
between unremarkable outcomes and undesirable outcomes (loss of
separation) rests on a combination of environmental challenge,
interpersonal sensitivity, and information sharing present during an
operational time frame. Undesirable outcomes are characterized more by
the absence of desirable behaviors than by the presence of undesirable
ones. Environmental challenge seems to force a decrease in interpersonal
sensitivity and controllers report perceiving increased demands under
these conditions. High levels of information sharing seem to moderate
the effects of high environmental challenge and decreased sensitivity on
operational outcomes. A technical report summarizing this study is in
progress and should be distributed Spring 1997.

During FY 96, this research group conducted 3 line audits for major
carriers. This work included the training of air carrier personnel to
collect line data as well as the collection of line data by the
researchers. The results were reported to the carriers, allowing them to
address weaknesses or strengths highlighted on the audits.

The research group produced three doctoral dissertations: (1) \"The role
of group dynamics in error tolerant systems,\" (2)\" Rising to the
occasion: foundations, processes and outcomes of emergent leadership\",
and (3) \" National culture and work attitudes in commercial aviation: A
cross-cultural investigation."

Additionally, the research group produced fifteen academic papers on
aviation crew issues. These papers and dissertations can be obtained
from the Internet. For instruction on how to access these papers, call:
Lou Montgomery (512) 480-9997.

During FY97, a fifth generation model of CRM was developed to be used in
error management. This requires a new approach to training of
instructor/evaluator groups, as well as development of new evaluation
tools.

Research is also concentrating on validating the findings from the NTSB
1994 report on commercial aviation accidents using line audit data. This
validation study will examine whether the accident causes identified by
the NTBS are observable in line operations.

**Products:** Reports to air carriers; Maintenance of International
CRM/LOFT database; Information for Advisory Circulars and regulatory
considerations for Flight Standards and industry training groups.

**Project Title:** *An Investigation of Training Issues Concerning the
Advanced Qualification Program (AQP)*

**Performing Agency:** Battelle Memorial Institute, Columbus, Ohio

**Background:** In an attempt to encourage the use of innovative
training programs, the FAA has proposed the concept of Advanced
Qualification Program (AQP) training. Central to AQP is
proficiency-based training. Under AQP, carriers who have applied for
inclusion in the program can develop their own proficiency objectives
which must address the range of conditions and contingencies that might
be faced by pilots working within the carrier\'s operational domain.
These proficiency objectives define the set of skills and tasks a pilot
must be able to perform to be proficient on a given aircraft type within
the carrier's operational domain. The goal is to ensure that the
training program meets each carrier\'s specific requirements and does so
in the most efficient way possible. Utilization of a strong analytical
framework for developing a carrier\'s program helps to ensure that
training is systematically oriented towards those objectives of greatest
relevance to the individual carrier and also supports meaningful crew
and program evaluation. One of the objectives of AQP is to provide
seamless integration of CRM and technical skills within the curriculum
to ensure that CRM skills are practiced together with all other flight
skills and procedures as required by each flight situation. One goal of
AQP is that CRM skills should be utilized as a normal and inherent part
of aircraft operation little different from operating the automation or
performing a proper checklist.

While AQP permits significant departures from the traditional FAR
requirements for training and checking airmen, the price of that
regulatory flexibility is a detailed front end analysis, the methodology
for which is described in Advisory Circular (AC) 120‑54. Accomplishing
the analyses necessary to create AQP qualification standards has proven
to be particularly challenging for participating air carriers. Training
developers have had difficulty selecting an appropriate level of
analysis detail, efficiently executing their analyses and determining
how to incorporate cognitive and crew resource management considerations
in a manner that will generate meaningful proficiency objectives,
standards, and conditions. These difficulties stem in part from the fact
that as a new program, AQP lacks concrete examples for reference
purposes. The methodology was developed by the FAA on an a priority
basis, with the explicit intention that it be refined subsequently as
experience with the new programs accumulate. There was a need for a
methodology that if faithfully followed would produce an effective AQP.

**Project Description:** The goals of this research project are to: (a)
assess whether the Instructional Systems Design (ISD) process can be
effectively used to develop a prototype AQP, including specification of
the process that derives learning objectives from the task analysis, (b)
determine whether the ISD methodology needs to be modified to best
support AQP development, and (c) verify the modified ISD methodology
that has been standardized through the development of a database
comprised of templates and instructional guidance for developing an AQP.

The approach to integrating CRM into technical training adopted by this
research utilizes ideas from several sources: (1) ISD methodology used
for AQP curriculum development, (2) the event set approach developed by
Carlow for the FAA, and (3) the situation assessment model being
developed by Battelle for the Model AQP. Each of these sources provides
a useful and unique perspective. The ISD methodology, as currently
implemented, focuses on the technical skills, knowledge and procedures
required to accomplish specified tasks and subtasks. The methodology is
especially effective for activities that occur at predictable times and
in a standard order. The event set approach, in contrast, focuses on a
selected sequence of situations which attempts to mimic real‑world
situations with all of the attendant complexity. The objective is to
evaluate crew performance in situations that require both technical and
CRM skills. This approach is currently used by many carriers to support
line operational simulation. This research project\'s situation
assessment model attempts to provide a cognitive perspective by focusing
on those factors that influence a flight crew\'s assessment of a
situation and subsequent management of available resources. In the past,
there has been little continuity between the task-oriented front end
analysis provided by the ISD methodology and the situation‑oriented
event set approach. This lack of continuity is exemplified by
differences in types of CRM skills addressed by each approach. The ISD
methodology is best suited to handle phase‑specific skills, that is,
those activities which are always performed by the crew for a given task
or subtask. Traditional ISD\'s behavioral orientation supports its
emphasis on the specific tasks and subtasks that must be performed to
complete a job.

This task orientation does not support those aspects of the pilot\'s job
which fall outside of the sequential tasks and subtasks found in a task
list. Instead, the unique dynamics of the aviation environment
necessitate a change in focus to the situation as a whole, including
conditions under which a task or subtask must be performed (e.g.
weather, aircraft system, failure) and the requirement to utilize
phase‑independent flight management skills either on a need basis or
continuously to ensure that the flight is properly managed. Appropriate
utilization of phase-independent skills depends upon crew judgment:
accurate assessment of the requirements of the situation together with
effective utilization of those skills and information sources most
likely to be useful in that situation. This judgment depends upon an
understanding of the situation as a whole, not simply the task in
isolation.

This situational focus is the strength of the event set approach. An
effective training program will enable flight crews to experience these
situations so as to allow them the opportunity to practice the
phase‑independent skills required to cope with these situations.

Both the ISD and event‑set methodologies bring important and unique
perspectives to an AQP. Merging both into a coherent approach will
support the development of a complete training program. One means by
which this integration can occur is to place the focus of flight
training on situations rather than tasks throughout the program rather
than waiting until LOFT. A situational orientation throughout training
helps to ensure that phase‑specific and phase-independent technical and
CRM skills are practiced in an integrated fashion. In addition, the
situational orientation gives flight crews the opportunity to practice
those skills involved with assessing situations.

This transition from the task focus of the ISD methodology to a
situation orientation takes place in the Model AQP by means of the
concept of an event. An event includes a specific task (i.e., a maneuver
or set of procedures) together with the conditions (weather,
malfunctioning aircraft system, etc.) under which the task is to be
performed. To handle an event successfully requires that the crew
quickly and accurately assess the situation, plan how to manage the
event and utilize the technical and CRM skills appropriate for that
event. In addition, the set of events included in the curriculum can be
selected to ensure that important technical and CRM issues are
addressed. Each event centers on a specific topic or theme. The flight
training curriculum can be designed by strategically selecting and
positioning events in accordance with these themes.

One of the strengths of the event concept is its applicability to both
ground and flight training. Continuity throughout all parts of an AQP is
a critical goal for the Model AQP project. Events can be used as the
building blocks for both the ground and flight training curricula. For
ground school, one of the goals of the Model AQP is to utilize
scenario‑based training in which students would be required to not only
acquire new information but also learn how to apply that information to
solving problems. Events are a natural tool for designing a
scenario‑based ground school curriculum. Similarly, events can serve as
the individual units for FTD and simulator training. Finally, they will
continue to serve as the building blocks for LOE scenario development.

The Model AQP, which has been targeted to regional carrier training, has
been completed and is being used by approximately 20 carriers. During FY
96, two workshops were conducted to train air carrier personnel in the
use of the model. During FY 97, appropriate updates and modifications to
the model were accomplished.

AQP training and LOE evaluation is exceeding the objective of integrated
crew training. Data analysis is providing critical information to focus
training. AQP performance analysis is providing critical safety
information to initiate changes in pilot training.

This model AQP has introduced several novel approaches to pilot
training. In summary, this research project and the Model AQP views
flight tasks, including CRM behaviors, as being either phase-specific or
phase-independent. Also, this model proposes that pilot training should
begin event training at the start of a program and not wait to the end
to incorporate a set of individual tasks into the Line Operational
Evaluations.

**Products:** Model AQP and research report delineating process,
methodology and lessons learned. The Model has been delivered to many
air carriers. During FY 97, modifications to the model were completed.
The 2nd generation database included conversion to MSAccess with more
powerful user information. AQP methodology enhancements were recommended
which included task identification guidance, EO design guidance, and
rapidly reconfigurable event sets. A performance/proficiency database
was also created.

**Project Title:** *LOFT Debriefing Study*

**Performing Agency:** NASA Ames

**Background:** Training effectiveness of LOFT simulations is heavily
influenced by the debriefing that occurs afterward. The LOFT simulation
is a very busy, intense, and sometimes stressful experience. Thoughtful
discussion after the experience is necessary in order for the crew to
sort out and interpret what happened and to consolidate the lessons
learned into long-term memory in a form that can be used later in actual
line operations. The debriefing is a window on the entire CRM process.
LOFT debriefings can demonstrate how well crews are able to analyze
their performance along CRM dimensions. It is thought that in order to
implement CRM effectively in day‑to‑day line operations, crews must have
the skill and the habit of analyzing their own performance in terms of
CRM.

**Project Description:** The purpose of this project is to determine
which techniques are actually being used by LOFT instructors, how
effective the techniques seem to be, the extent to which those
techniques are consistent with FAA Advisory Circular guidelines, how
practical the guidelines are for real‑world training and what obstacles
instructors encounter in trying to teach according to these guidelines.

Data have been collected from five major US air carriers. At each
carrier, five to eleven instructors from different fleets made
audiotapes of the debriefings they conducted. The audiotapes were
transcribed and coded, generating a large database. The data from the
first two carriers have been analyzed and results have been reported to
the respective carriers. Data analysis from the other three carriers is
in progress and will be completed in winter 1996.

Preliminary findings are as follows:

\(1\) Typically, the debriefing is scheduled after the practice of
proficiency maneuvers, rather than immediately after the LOFT.
Interposition of the proficiency maneuver practice constitutes an
\"interference experiment\" that probably impedes the ability of the
crew to remember what happened in the LOFT.

\(2\) The similarities among carriers are greater than differences in
how they conduct LOFT and LOFT debriefings. There are large differences
among individual instructors within carriers in facilitation
effectiveness.

\(3\) Most instructors conscientiously attempt to elicit crew
participation, but some unwittingly sabotage their own efforts with
behaviors that discourage participation.

\(4\) Most instructors fail to make clear at the beginning of the
debriefing the nature of the participation that they expect of the crew
and do not explain why it is important that the crew take an active
role.

\(5\) Although instructors have been told to facilitate crew
self‑debriefing, they have been given only very general advice on how to
do this. Training departments should provide much more explicit hands‑on
training in facilitation and should mentor new instructors as they start
to facilitate.

\(6\) The content of the debriefing is strongly driven by the
instructor\'s observations and questions, rather than the crew\'s self
analysis.

\(7\) The crews are responsive to the instructors, but are more reactive
than proactive.

\(8\) Most crews would probably have difficulty conducting a deeply
analytical debriefing on their own, however they appear to be quite
conversant with and accepting of CRM. With practice, they could develop
debriefing skills and apply them routinely on the line.

The data and analysis have resulted in major carriers developing a list
of do\'s and don\'ts for facilitators. It is being used as the core of
annual recurrent training for instructors.

A field test of debrief criteria is now underway by Northwest Airlines.

Further research will examine distractions and attention lapses in the
cockpit. The question is \"During multi‑tasking performance, how do
humans switch from one task to another effectively?\" Cockpit attention
problems include the failure to return to interrupted tasks, returning
to the wrong place of a task, tunneling of attention, premature
interpretation of situations, and inefficient task management. This
research will analyze NTSB and ASRS reports, field observations, video
from Orasanu/Prince project and experimental studies to develop
techniques crews can use to manage attention effectively.

**Products:** CRM Debriefing training guidelines were distributed in
February 1997. A handbook is being developed and will be distributed to
carriers in January 1997. Training video produced with Delta Air Lines.
Draft technical report of International Journal of Aviation Psychology.

**Project Title:** *The Development of Reliable and Valid Scenario Event
Sets for Line Operational Evaluation, Based on Empirical Data.*

**Performing Agency:** NAWCTSD; University of Central Florida

**Background:** In practice, training and assessment of CRM takes place
during LOFT. LOFT occurs in a full mission simulator and aircrews are
required to fly scenarios that include specific CRM events, or event
sets, which are designed to target CRM behaviors. Under AQP, CRM
proficiency will be evaluated in the simulator during LOE. Therefore,
the procedure by which CRM scenario event sets are developed will have
an impact on the CRM assessment process. Quality event sets will need to
be incorporated into LOE scenarios to ensure that an adequate number of
CRM behaviors are elicited by an aircrew so that check pilots can make
judgments regarding a crew\'s proficiency in CRM.

In theory, scenarios are designed in such a fashion as to require crews
to demonstrate CRM skills. The FAA provides guidelines, Advisory
Circular on LOFT, on the development of scenarios. However, little or no
information is presented on how to develop scenarios that are of
reasonable difficulty and that adequately distinguish between various
crew performance levels. It has been suggested that crews should have
more than one opportunity to display CRM skills of interest in order to
validly assess the crew\'s level of performance. This point is not in
any formal guidance from the FAA nor has this idea been tested in
operational settings.

Scenario event sets can be thought of like items on a test. Therefore,
scenario events could be, and perhaps should be, generated in a fashion
similar to that of test development. Reliable and valid measurement of
CRM skills will likely be achieved if two conditions exist in the
measurement process. First, CRM event sets, embedded in scenarios, must
possess reasonable levels of difficulty and effectively discriminate
different levels of aircrew performance. Second, there must be multiple
event sets that address each CRM skill assessed. In test development,
reliability and validity are maximized by selecting the highest quality
items and employing multiple items to measure a single variable of
interest. For each item, data are collected on the item\'s difficulty
and discriminability, and then the best items are selected based upon
these indicators.

A unique aspect of this project is that several sources of safety data
collected by the carrier are being used for event set design rather than
subjective judgment of issues currently effecting the carrier.

**Project Description:** This research project will develop and test a
methodology for establishing reliable and valid CRM event sets and
determine the number of event sets required to measure CRM skills
reliably.

**Products:** A series of reliable and valid CRM event sets that can be
incorporated into any number of CRM scenarios for the purpose of LOE.

**Project Title:** *Rapidly Reconfigurable Event-Set Based Line-Oriented
Evaluations (LOE) Generator*

**Performing Agency:** NAWCTSD; University of Central Florida

**Background:** Line-Oriented Evaluations (LOEs) are a methodology used
in Advanced Qualification Programs (AQPs) to evaluate pilot training
performance and establish trainee proficiency. LOEs consist of flight
simulation scenarios that are developed by the training organization and
approved by the FAA.

In the past, LOEs were developed and apprised individually. That is,
each LOE was separately conceived, developed and tested by the training
organization and individually reviewed and approved by the FAA. The
development and approval of new LOEs was costly and time consuming. As a
result, training organizations usually had only a limited number of LOEs
available for evaluation, each of which were only approved for a limited
time period. In some cases, this may have increased the risk of LOE
scenarios being compromised within a training organization, thus
reducing the validity and reliability of the scenarios.

One way to reduce the negative aspects of the previous LOE design
process will be to develop a set of LOE event sets (modules) which upon
FAA approval could be assembled by the training organization into a
number of unique LOEs without requiring additional, specific approval of
each individual LOE by the FAA.

This project will develop appropriate software to allow the
reconfiguration of separate event sets into entire LOEs. This will
require that the difficulty of each event set is established. This
project was started in July, 1997. The requirements document has been
written and approved. This is a three year effort.

**Project Title:** *Integrating CRM into Crew Procedures: Checklist
Standardization*

**Performing Agency:** NASA AMES

**Background:** Non‑compliance with checklists has become a potentially
serious problem for air carriers and the FAA. In today\'s aviation
system, procedural changes occur frequently and are instituted for a
large variety of reasons: company mergers, changes in air traffic
procedures, changes in airport environments, introduction of new
technologies and new aircraft, etc. While one obvious goal is to
maintain compliance and standardization of crew member performance,
there must not be a sacrifice in the match between procedures and the
operational realities in which the procedures are to be used. Not only
must procedures reflect a reasonable match to technical task demands,
they must reflect consistency with company and system requirements.
Within an air carrier for example, philosophy, policies and procedures
should be made consistent both within teams (within aircraft type) and
across teams (across fleets within the company) to the extent that
standardization is feasible. Since there are legitimate aircraft and
mission differences across fleets, a means of evaluating the appropriate
level of standardization must be developed and reviewed when changes are
anticipated.

**Project Description:** The goal of this research project is to
integrate CRM principles into crew roles and procedures in order to: 1)
improve the match between current procedures and operational reality,
and 2) develop a process for adapting procedures to anticipated changes.

The basic approach is to analyze procedures from a team perspective and
to conduct a team task analysis in order to assess whether current
procedures match the task flow including team processes. The integration
of CRM principles into the task analysis provides the basis for
reviewing and re‑designing procedures. The development of standard
methods and measures for evaluating procedure usage is important for
assessing both current and proposed procedures. Measures used during the
research process will be refined for validation purposes as well as
eventual use in training.

The objectives of this research project are to develop a systematic
approach to checklist standardization across all fleets, to develop
guidelines for implementing and evaluating procedure changes that
optimize: l) within company standardization, 2) compatibility with
aircraft and operational differences and 3) procedures consistent with
CRM principles. This research is in phase II and has thus far
established company‑wide philosophy and policy statements: Operations,
Automation, Training, and Checklists. Also, methods for data collection
have been identified and developed. Pilot feedback on proposed checklist
changes has been completed. This project has been completed and the
report and guidelines are being written.

**Project Products:** Guidelines for standardization of procedures.

**Note:** This project led to the following project described on the
next page.

**Project Title:** *Operating Documents Human Factors Project*

**Performing Agency:** NASA; George Mason University

**Background:** The original project described on the preceding page was
intended to develop a systematic approach to checklist standardization
and to develop guidelines that would optimize within-company
standardization for eight fleets, to insure compatibility with aircraft
and operational differences and be consistent with CRM principles. This
project considered the form of the checklist with respect to content,
structure, logic, format, terminology, roles and usage. However, this
approach was inadequate as checklists are part of the whole operating
document system. It became apparent that operating documents must show
internal consistency across fleets (aircraft types, and route
structures), across departments (training, evaluation, safety, cabin
crew, maintenance, etc.), and across documents (FOM, training documents,
pilot's handbooks, company policies, etc.) Further, these documents must
be externally consistent with regulations and manufacturers. Document
systems must be consistent across philosophies, policies, procedures and
practices. This concept is not in place in the air carrier industry.

This research team developed surveys and held workshops to bring
together air carriers (including regionals and cargo), manufacturers,
and the FAA to work on a better, more human centered approach to these
documents. Two workshops have been held in FY 97. The most recent
workshop was attended by 78 participants.

Based on the information provided on the surveys and in the workshops,
guidelines will be written for the development of operating documents
that will include: the Organization of Documents, Standardization of
Documents, Usability of Documents, Developing and Maintaining Documents,
and Transition to Electronic Media.

**Products:** Focus group presentations and guidelines document.

**Project Title:** *Team Situational Awareness*

**Performing Agency:** Naval Air Warfare Center, Orlando

**Background:** There is little question that situational awareness is
critical for safe flight. Much research has centered on attempts to
identify and quantify individual situational awareness. However, since
cockpit crew performance depends heavily on team processes, it is
necessary to determine exactly what aspects of individuals and the
environment either enhance or degrade team situational awareness.

**Project Description:** This research project will identify a model of
team situational awareness,

develop effective instructional strategies to improve situational
awareness training, develop

situational awareness assessment tools, and develop guidelines for
situational awareness scenario

development.

During Phase I of this project, an initial framework for team
situational awareness has been completed. This framework will serve as a
basis for the team situational awareness model. Preliminary training
guidelines based on this framework, which will be amended as the
research progresses, follow:

\(1\) Tailor training to the experience level of the crew in terms of
time in aircraft, time in position, and familiarity with locality. For
example: Consider a newly hired first officer or flight engineer that is
transitioning to or learning a new aircraft and flying out of a new
area. It is important to ensure that the crew member is familiar with
cockpit roles, the geographic area, and the aircraft equipment before
expecting him/her to have the awareness to handle dynamic problem
situations in that environment.

\(2\) Ensure job/task analysis provides information about the cognitive,
perceptual and behavioral demands required for situational awareness.
For example: A job analysis of a pilot who flies a highly automated or
\"glass cockpit\" aircraft should solicit information particularly
relevant to that environment, emphasizing factors such as information
seeking from different sources, collation and interpretation of
information from complicated displays, scan through the FMS, mode errors
with the FMS, manual data entry task demands, and communications
associated with the advanced technology.

\(3\) Include training which specifically focuses at the team level of
situational awareness in addition to training which focuses on
individual skill development. For example: individual skill development
in situational awareness may include: providing information that
increases the individual\'s knowledge base so that he/she knows when and
what information to seek, helping in the development of planning skills
so that he/she is better prepared for completing tasks within the
flight, and providing tips that help free up time from basic tasks in
the cockpit so that more attention may be paid to the situation. Team
skill development in situational awareness should include team process
skill training such as communication and leadership behaviors to enhance
and maintain crew awareness.

\(4\) Based on the complexity of the situation, provide training which
emphasizes adequate information, demonstration, and active practice and
feedback. All are needed to produce a behavioral change. For example: to
enhance the overall training experience provided in a realistic LOFT
scenario emphasizing situational awareness, the crew members should
receive advanced preparation about the training objectives, including
specific information on the skills to be trained and demonstrations or
examples of effective and/or ineffective behaviors relevant to those
skills. Providing general information about situational awareness
concepts would be insufficient to expect the crew to obtain the full
impact of LOFT. Crew members who are at a low experience level need to
have specific information on the kind and timing of information transfer
that will enhance crew situational awareness. Alerting them to the
importance of communication, if they do not know what and when to
communicate, would not prepare them to practice good communication
skills in the LOFT session.

\(5\) Design training scenarios to be realistic and to elicit the
desired cognitive, perceptual and behavioral task demands encompassing
the situational awareness construct. For example: consider that
including an unfamiliar event, such as icing for crews who have only
flown in Florida, would require them to recognize the relevant cues,
comprehend their meaning and project a course of action. A very
different type of problem, such as a gradual change in an instrument due
to a developing condition, may require perception, comprehension and
projection as well, but the cognitive and behavioral demands related to
this problem are very different from those imposed by icing conditions.
Including both types of problems within a scenario would allow the
trainee to exercise these different cognitive and behavioral skills and
would help him/her learn to generalize the application of these skills
across various problem situations.

\(6\) Conduct training which focuses on improving team or process skills
related to situational awareness (e.g., planning, communication,
leadership). For example: training focusing on communication skills
related to situational awareness can include training that encourages
the flow of the information that needs to be transmitted in the planning
phase of the flight (e.g., what questions the Captain can ask to solicit
information, what questions the First Officer should ask, what
information should be covered with Flight Attendants). Situational
awareness training can also include practice in the communications that
are most effective (e.g., what information to provide to another crew
member or when to communicate the information to the other).

\(7\) Include training which emphasizes developing accurate and
appropriate knowledge structures for expectations related to situational
awareness. For example: familiarizing crews with a specific airport's
traffic flow prior to arrival increases their readiness to recognize an
abnormal situation.

\(8\) Use realistic and dynamic scenarios to provide training on
information seeking and processing, detecting anomalous information,
prioritizing attention to cues especially when there is information that
may be related to different problems, and taking appropriate actions
based on cues. For example: design realistic scenarios with events
requiring crew members to look for and use information, handle multiple
cues, determine when information is signaling a problem or possible
problem, and act on the information. Intermittent fluctuations of an
instrument during a scenario require crew members to recognize the
fluctuations and to diagnose the problem. This may include cross
checking instruments, discussing among themselves, and seeking
additional information. Putting this event in the scenario at the time
of moderately heavy workload or during another troubleshooting event
gives crew members practice in dividing their attention and determining
priorities.

\(9\) Design and develop performance measurement tools for the
behaviors, knowledge and perceptions required for situational awareness.
For example: a realistic scenario provides information about situational
awareness behaviors; an observation scale can be developed for those
behaviors. Tapping into the crew member\'s knowledge and perceptions may
require asking questions, either during the scenario or after it is
completed to determine if he/she saw the problem and understood its
possible impact. For pilots with low experience levels, questions can be
either a part of the scenario (e.g., ATC can question the crew members)
or the scenario can be stopped briefly for questions to be asked.

\(10\) Measurement tools should measure individual situational awareness
skills as well as team interactions related to situational awareness.
For example: an observation scale can be designed that measures what
individual team members contributed to the team situational awareness
and what actions they performed independently that demonstrated their
situational awareness. The same scale should include opportunities to
document team interactions (e.g., planning, communications, and
leadership) that helped the crew members gain and maintain team
situational awareness.

\(11\) Situational awareness measurements of behaviors should include
both outcome and process measurements. For example: outcome is the
action taken by the crew in response to an event (e.g., the crew
diverted to Airport A) and process is how they achieved the outcome
(e.g., the Captain gathered information from others about the decision,
crew members discussed the merits of each airport, the crew members
noticed that the runway at Airport B was not adequate for their needs).
Both are important to understanding performance.

\(12\) Include instructions and information about training objectives to
instructors about each training method or tool. For example: write out
the specific objectives to be accomplished at the top or on a cover
sheet of scenario instructions. Include clear objective behaviors as
part of the scenario goals. The instructor is then informed about how to
use the measurement tool to re‑enforce the training objectives.

\(13\) Link the measurement criteria to the scenario events and to the
debrief plans, for consistency in training. For example: if a generic
measurement instrument is used, prepare the raters to recognize and
document specific examples observed in the scenarios that relate to the
general behaviors on the rating form. It is likely that the raters will
need thorough training so they can map the scenario events and behaviors
observed to the rating form. Thus, in the debrief, crew members will
know what specific actions they took and how those relate to general
requirements for situational awareness.

\(14\) Multiple measurements over time are necessary to yield a good
measurement of situational awareness; single measurements at one time
are insufficient. For example: in a single LOFT scenario, build a number
of opportunities for crew members to demonstrate situational awareness
(e.g., NOTAM information, ATC calls to other aircraft, PIREPs, subtle
changes in instruments, weather, unexpected circumstances) and ensure
that observers know when, where, and how to observe and document
situational awareness actions. The same could be accomplished by using a
number of small scenarios (each with opportunities to observe
situational awareness) and then evaluating the aviator\'s situational
awareness across the scenarios.

\(15\) Use specific feedback tools along with realistic and dynamic
scenarios to provide feedback that is specific to the trainees on their
situational awareness skill performance, reinforces the development and
maintenance of relevant knowledge, and stresses adapting knowledge to
meet situation demands. For example: give observers feedback forms with
the specific situational awareness elements that were built into the
scenario. Observers should use these forms to document behavior they see
in the scenario. This is then used to give specific feedback on what the
crew members did, how they did it, and the context of the behaviors.
Explanation of the rationale for the feedback helps the crew members
develop and add to their existing knowledge.

\(16\) Train instructors on making ratings and giving feedback. For
example: this training can be done by providing information to the
raters on situational awareness, showing them examples of the behaviors
associated with situational awareness (so raters can be knowledgeable
about the behaviors), giving them practice in rating those behaviors,
and giving raters practice in providing feedback on specific situational
awareness behaviors.

Phase II of this research project concentrates on other challenges. The
goals will include making the methodology generic to airlines and
aircraft, creating data exchange protocols with existing databases,
creating intuitive and adaptable user-interfaces and validating the
methodology under field conditions.

**Products:** The above guidelines have been expanded, are under review
and will be distributed 11/97.

**Project Title:** *Training Pilot Not Flying (PNF) Back‑up Skills*

**Performing Agency:** Naval Air Warfare Center, Orlando

**Background:** Regional airline Captains indicate that one of the
greatest difficulties they experience is working with inexperienced or
new First Officers who lack the skill to effectively provide back‑up
when serving as Pilot Not Flying (PNF) when it is needed. Since the
complex environment of regional flying necessitates the coordination of
the two crew members, it is critical that this issue be addressed.

**Project Description:** This research project will identify the
knowledge, skills and attitudes necessary for PNFs to properly perform
back‑up duties. In conducting this research, (1) structured interviews
with pilots from the regional airlines were conducted to inquire about
effective and ineffective back‑up behavior, (2) the 100 most recent
submissions to the ASRS database were analyzed using the descriptors of
\"regional airlines\" and (3) analysis of the recent NTSB report which
reviewed flightcrew-involved major accidents of U.S. air carriers that
occurred from 1978 through 1990 and which identified poor
monitoring/challenging behavior on the part of the non‑flying crewmember
as a causal factor in seventy‑six percent of these accidents. Combining
information from these sources, three broad problem areas requiring
additional training were identified: (1) First Officer or PNF
assertiveness in ambiguous situations, (2) monitoring and challenging by
the First Officer or PNF, and (3) proficiency in technical and
communication procedures. Presently, a study is being conducted to test
the training potential of several techniques for improving PNF back‑up
performance.

The direction of future research will be toward metacognitive training
and CRM-/team-training. Metacognitive training is a viable addition to
CRM and is particularly useful for teaching abstract, difficult to
observe cognitive processes. Team training will allow the training of a
good backup, and that behavioral training will not "boomerang."

**Products:** "Identifying Critical Training Needs for Junior First
Officers" was distributed 5/97.

**Project Title:** *Decision-Making and Judgment on the Flight Deck*

**Performing Agency:** NASA-Ames and the Naval Systems Warfare Center,
Orlando

**Background:** The NTSB reports that in the four year period,
1983‑1987, crew judgment and decision making was implicated in 47% of
fatal accidents. The NTSB and ASRS cite many more reports where poor
decision making was evident in many potentially serious incidents.
Decision making is a component of most CRM courses and is encouraged by
the FAA Advisory Circular 120‑51, 1993. However, scientific research has
not been available to support this training. Decision making in the
dynamic, time-constrained cockpit environment does not mirror decision
making in a static laboratory setting. Thus, findings from generic
decision making research do not necessarily generalize to aviation
settings.

**Project Description:** Decision making is an essential component of a
Captain\'s expertise. The Captain is responsible for making the hard
decision (for example: choosing where to divert after a system
malfunction, when fuel is short and weather is deteriorating;
determining how to cope with a passenger\'s medical emergency,
evaluating whether to take‑off with a placarded system given past
experience with the projected weather and traffic at the destination).
The Captain\'s judgment is most critical when conditions are ambiguous
and no clear guidance is provided in manuals, checklists, or company
policy.

This research project has been in progress for several years and has
developed a model of factors that make flight decision efforts
difficult, along with a taxonomy of decision problems. This model
incorporates the nature of the problem, the time available to solve the
problem and the risks associated with the options. This work was based
on analyses of crews coping with abnormal and emergency events in full
mission simulators and further supported by NTSB accident analyses and
ASRS reports. For example, analyses revealed differences in the
importance of risk and time pressure between Captains and First
Officers. For Captains, risk was the most important variable in the
decision strategy, while First Officers viewed time pressure as most
important. As might be expected, experience does affect decision
strategies. Experienced crew members are more sensitive to available
decision time, better at handling uncertainty and have better
prioritization of information needs. Based on these results it is
suggested that to encourage effective decision strategies, crews are
trained to: (1) understand and verify the problem, (2) assess time and
risk, (3) not rush to judgment, (4) consider constraints and
consequences of options so as not to oversimplify the problem, (5) use
\"worst case\" instead of \"best case\" reasoning, (6) plan for
contingencies, and (7) manage the situation by setting priorities,
assigning tasks and using all available resources.

Presently, this research project is developing decision making training
scenarios for major carriers to validate during recurrent training. This
research project focuses on designing and validating assessment methods
and scales for the decision making scenarios. This work also is
developing training guidelines to manage risk in ambiguous situations,
and assess LOS/LOFT and debriefing tools.

**Products:** Decision making training and assessment guidelines for air
carriers to incorporate into their CRM training or into AQP will be
published in 1998. Article written, "Stress and Naturalistic Decision
Making: Strengthening the Weak Links."

**Project Title:** *Pilot Training and Evaluation: Airplane Simulation
Human Factors*

**Performing Agency:** Volpe Center

**Background:** Flight simulator technology has advanced significantly
during the past decade, one result of which is that device capability
has increased while cost declined. However, for regional airlines, cost
continues to be an obstacle to flight simulator access, particularly for
recurrent training. Currently, airplane FTD and simulator qualification
criteria are contained in Advisory Circulars AC 120-40B and 45B/C and
related documents. Changes both in the air carrier training environment
and in the field of simulator technology demand a reassessment of these
criteria. The shift in training philosophy from a time-based list of
specific training events to the Advanced Qualification Program (AQP)
training approach based on individual training needs requires that much
of the training formerly performed in aircraft will move to the
simulator. Considering the initiative that Part 135 carriers be held to
the same safety standards as Part 121 carriers, the success of AQP
depends on whether regional carriers will be able to afford simulators
that are appropriate for training 121 proficiency using an AQP training
approach. AAR-100 is undertaking an extensive investigation of the pilot
cueing requirements to enhance affordability without degrading the
standards or quality of performance for such equipment.

**Project Description:** An extensive literature review of the motion
vs. non-motion studies for pilot training has been completed. The
research study has been designed, an air carrier has been secured to
conduct the research, the simulators have been programmed to collect the
necessary data and the instructor pilots have been trained in data
collection. Data collection will begin 10/97 and is expected to continue
for approximately 1 year.

**Products:** Report on the necessity of motion for recurrent pilot
training