j 

DOT/FAA/AM-96/24 

Office of Aviation Medicine 

Washington, D.C. 20591 

Flight Inspection Crew 

Resource Management 

'fraining Needs Analysis 

Lawrence L. Bailey, 

Rogers V. Shaw 

Civil Aeromedical Institute 

Federal Aviation Administration 

Oklahoma City, Oklahoma 73125 

September 1996 

Final Report 

This document is available to the public 

through the National Technical Information 

Service, Springfield, Virginia 22161. 

DTIC QUALITY mSPEL'TED 8 

U.S. Department 

of Transportation 

Federal Aviation 

Administration 

• 

• 
NOTICE 

·This document is disseminated under the sponsorship of 

the U.S. Department of Transportation in the interest of 

information exchange. The United States Government 

assumes no liability for the contents or use thereof 
Technical Report Documentation Page 

1. Report No. 12. Government Accession No. 3. Recipient's Catalog No. 

DOT/FANAM-96/24 

4. Title and Subtitle 5. Report Date 

Flight Inspection Crew Resource Management Training Needs September 1996 

Analysis 

6. Performing Organization Code 

7. Author(s) 8. Performing Organization Report No. 

Lawrence L. Bailey, Ph.D. and Rogers V. Shaw, M.S. 

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) 

FAA Civil Aeromedical Institute 

P.O. Box 25082 11. Contract or Grant No. 

Oklahoma City, OK 73125 

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered 

Office of Aviation Medicine 

Federal Aviation Administration 

800 Independence Ave., S.W. 

Washington, DC 20591 14. Sponsoring Agency Code 

15. Supplemental Notes 

This work was performed in response to the the National Transportation Safety Board's recommendation that crew 

resource management (CRM) training should be implemented as outlined in the Federal Aviation Administration's 

CRM Advisory Circular 120-51. 

16. Abstract 

On October 26, 1993, there was a fatal crash of a Federal Aviation Administration (FAA) flight inspection 

aircraft. During the accident investigation, the National Transportation Safety Board (NTSB) cited 

ineffective crew resource management (CRM) as one of the causal factors and recommended CRM 

training for flight inspection aircrews. As part of the FAA's response to the NTSB recommendation, a 

CRM training needs analysis was conducted. Cluster analytic results of the identified training needs 

suggested three categories affecting crew performance: ( 1) technical skills, (2) crew coordination skills, 

and (3) the organization context- in which flight inspection crews perform. Implications for CRM 

awareness training are discussed. The purpose of this reort is to document the flight inspection CRM 

training needs that emerged from the analyses and to recommend steps for developing a flight inspection 

CRM training program. 

17. Key Words 

Crew resource management 

CRM 

Training 

19. Security Ciassif. (of this report) 

Unclassified 

Form DOT F 1700.7 (8-72) 

authorized 

18. Distribution Statement 

Document is available to the public through the 

National Technical Information Service 

Springfield, Virginia 22161 

20. Security Classif. (of this poge) 21. No. of Pages 22. Price 

Unclassified 17 

Reproduction of completed page 

DTIC QUALITY mSPEC'.rED 3 
FLIGHT INSPECTION CREW REsOURCE MANAGEMENT 

TRAINING NEEDS ANALYSIS 

The training of aircrews has changed markedly 

from the early days of aviation. Initially, the emphasis 

of air crew training focused on developing the technical skills of the individual crew members. The underlying assumption was that if crew membets were 

technically proficient at their respective jobs, they 

would automatically be able to operate effectively as a 

crew {for historical reviews, see Driskell & Adams, 

1992; FAA, 1993, Hartel & Hartel, 1995; Helmreich 

& Foushee, 1993). During the 1970s, however, evidence from airline accident reports, flight simulator 

transcripts, and interviews with crew members suggested that the above assumption was inherently flawed. 

Technical competence, by itself, did not ensure a 

successful mission. Instead, mission success was dependent on the manner in which technically competent crews coordinated their individual efforts {Cooper, 

White & Lauber, 1979; Lauber 1987; and Ruffel 

Smith, 1979). Training to develop aircrew coordination skills became known as Crew Resource Management or simply CRM (FAA, 1993). More recently, 

organizational factors (in particular those which determine the consequence of performance, and those 

which provide flight crews with the resources necessary to perform their jobs) have also been found to be 

important determinants ofaircrew performance {Hackman, 1993). These findings suggest that aircrew performance depends on at least three factors: {1) technical 

competencies, (2) crew resource management skills, 

and {3) the organizational context in which crews 

perform. 

Issues surrounding the above performance factors 

were identified in a National Transportation Safety 

Board (NTSB) report of the October 26, 1993, fatal 

crash of a Federal Aviation Administration {FAA) 

flight inspection aircraft, N82 (NTSB, 1994). This 

was the second fatal flight inspection aircraft accident 

in five years. In reviewing the factors contributing to 

the October crash, the NTSB issued one urgent action 

and seven priority action recommendations to the 

FAA. Included in the latter was the recommendation 

to institute Crew Resource Management {CRM) training, as outlined in the FAA CRM Advisory 120-51, at 

each of Flight Inspection Area Office {FIAO). 

Prior to the accident ofN82, the FAA was already 

in the final stages of developing a CRM training 

course for all FAA aircrews. With the advent of the 

accident, the FAA Administrator elevated the priority 

of this initiative, and a CRM task force led by the Civil 

Aeromedical Institute (CAMI) was created to guide 

the process of developing a CRM course specifically 

for the flight inspection mission. Also included in the 

CRM task force was the FAA's Senior Flight Safety 

Officer and representation from six FIAOs. 

One of the first steps taken by the CRM task force 

was to conduct a CRM training needs analysis based 

on issues ~ddressed during the November 1993 safety 

meetings conducted at each FIAO. These meetings 

were mandated by the F AAAdministrator in response 

to the accident ofN82. The purpose of the meetings 

was to identify organizational and crew issues that 

impacted the safety of all FAA flight operations. 

"Flight inspection" refers to the airborne tests conducted to ensure that navigational aids are sending 

accurate signal-in-space guidance, and to ensure that 

instrument flight procedures are accurate and will 

safely guide aircraft to their destination. A flight 

inspection crew consists of a pilot in command (PIC), 

a second in command (SIC; co-pilot), and an electronics technician (ET). The flight inspection mission differs from other forms of flying {such as air 

transport) in that most flight maneuvers are conducted within the terminal area, at low altitudes, and 

at times running counter to the established air traffic 

flow pattern. This requires a high degree of traffic 

vigilance, as well as, coordination between the flight 

inspection crew, air traffic control, and a ground 

electronics technician. 

Although generic CRM programs modeled after 

commercial airlines have been developed, the FAA's 

CRM Advisory Circular {120-SIA) warns against 

using such programs without first customizing them 
to reflect the nature and needs of the specific flying 

mission. Following that guidance, a CRM training 

needs analysis was conducted to ensure that the mission-specific concerns of the flight inspection crews 

were addressed in their CRM awareness training. 

Specifically, the focus of the needs analysis was to 

identify: (1) the phases of the flight inspection mission that were most problematic from a CRM perspective, (2) the CRM dimensions that needed to be 

especially emphasized in a CRM awareness course, 

and (3) organizational factors that needed to be addressed to support a long term CRM initiative. 

This report presents the results of the needs analysis, along with a discussion of the CRM training 

implications that emerged. However, before proceeding, the reader may find helpful the following background information about CRM, the FAA's prior 

CRM training initiatives, and information concerning the flight inspection aircraft accidents that highlighted the need for flight inspection CRM awareness 

training. 

CRM Background 

Crew Resource Management (CRM) refers to the 

"effective utilization of all available resources-hardware, software, and personnel-to achieve safe, efficient flight operations" (p. 8, Driskill & Adams, 

1992). CRM differs from other forms of team work 

training both in the nature of decisions that must be 

made, as well as the consequences of those decisions. 

Typically, aircrews must make CRM related decisions in a matter of seconds and the results of those 

decisions have life and death consequences. Although 

CRM training began with aviation, it now has extended into such other fields as operating rooms 

(Howard, Gaba, Fish, Yang, & Sarnquist, 1992), 

nuclear power plants (Gaddy & Wachtel, 1992), and 

combat units (Andrews, Wagg & Bell, 1992). 

The need for specialized training in CRM stems 

from the developmental tract that many technical 

fields require of their professionals. For the most part, 

the emphasis of technical training is on the development of individual knowledge, skills, and abilities. 

Teamwork is often represented as the simple aggregation of individual efforts. Because of the lack of 

formal training in teamwork, technically proficient 

2 

individuals may have well developed strategies for 

working by themselves, but these same strategies may 

not be the most effective for use in a group (Salas, 

Dickinson, Converse, & Tannenbaum, 1992). For 

example, persons who are used to making decisions on 

their own may not take the time to inform others of 

the rationale behind those decisions. Whereas the 

individual may understand what is going on, others 

may have only a limited view of the situation, a view 

that can also be colored by their own perspective. 

One of the main emphases, then, of CRM training 

is to develop the resource management skills necessary 

to ensure that all group members are operating from 

a common frame of reference, and that 'this reference 

is consistent with what is actually occurring. Specific 

skills developed in CRM awareness training commonly include: (1) communication skills, such as 

inquiry, advocacy, and assertion; (2) methodologies 

for identifying problems and making decisions under 

severe time constraints; (3) self-monitoring skills for 

critiquing decisions and actions of the crew; (4) conflict resolution skills; (5) skills associated with crew 

leadership, followership, and concern for the task; (6) 

interpersonal skills necessary for maintaining a professional crew climate; (7) situational awareness and 

distraction avoidance skills; (8) workload planning 

and distribution skills; and (8) identifying personal 

stressors and developing effective stress reduction 

techniques (FAA, 1993). 

As an organizational intervention, CRM typically 

involves a three-phase approach (FAA, 1993). This 

includes formal classroom training, practice and feedback, and organizational reinforcement. In the first 

phase, the awareness phase, CRM principles are taught 

in a classroom setting and are reinforced through the 

use of accident recreations, case history discussions, 

role-playing exercises, etc. The purpose of this training is to provide crew members with a common 

understanding of how CRM principles apply to their 

mission, and to practice some basic skills. Once general principles are acquired, they are further refined 

during the second phase of training, practice and 

feedback. Ideally, this phase of training should be 

conducted on an annual basis in a simulated mission 

training environment. Crews then perform mission 

oriented tasks while being video-taped. Afterwards, 
crew members view the video playback of their performance and perform a self critique of their CRM skills. 

Finally, in the third phase, the reinforcement phase, the 

organization ensures that CRM principles become 

part of the organization's culture by adopting policies, 

procedures, and practices that are consistent with, and 

reinforce, CRM principles. Among other actions, this 

involves incorporating CRM principles in all forms of 

training, requiring managers to become effective role 

models for CRM behaviors, and establishing CRM 

criteria for use in selection, promotion, and performance appraisal decisions. For a more complete review of CRM training and its history, see Driskell & 

Adams (1992), Hartel & Hartel (1995), and Wiener, 

Kanki & Helmrei~h (1993). 

FAA CRM Training 

Two flight inspection CRM trammg mltlattves 

preceded the accident of N82. In early 1990, CRM 

awareness training was delivered to all flight inspection pilots. The course was developed and conducted 

by United Airlines. Although the course exposed 

participants to CRM principles, it was not customized to the mission needs of flight inspection crews. 

Furthermore, electronic technicians were not included 

in the training. Finally, the course itself was delivered 

as a stand-alone program, with no formal attempts 

made to integrate it into a long-term CRM program 

as described in the previous section. 

To address the above weakness, in 1992, under the 

sponsorship of the FAA's Flight Safety Program, a 

CRM initiative began, with the goal of developing a 

long-term CRM training effort for each of the five 

FAA aircraft programs: ( 1) flight inspection, (2) flight 

standards, (3) Washington flight program, (4) research and development, and (5) training academy. 

The Flight Safety Program targeted the flight inspection aircrews as having the most pressing need for 

CRM training. Thus, work began to develop a prototype flight inspection CRM training effort that would 

serve as a guide for the remaining aircraft programs. 

Guiding this effort was an informal taskforce, consisting of the FAA Senior Flight Safety Officer, CAMI, 

and representation from flight inspection crews. 

Throughout 1993, the taskforce reviewed existing 

aircarrier CRM programs, as well as the results from 

3 

published CRM research. With the advent of the N82 

accident, a formal CRM taskforce was developed and 

charged to deliver flight inspection CRM awareness 

training during the second quarter of fiscal year 1994. 

The new taskforce agreed to developed a one-day 

post-accident CRM awareness training course which 

would be followed by a three-day course following the 

completion of a CRM needs analysis. 

Flight Inspection Aircraft Accidents 

Although CRM training can be viewed as a proactive means for an organization to ensure the safety of 

its flight operation, some organizations wait until · 

they have experienced an aircraft accident before they 

initiate a CRM training program (Helmreich & 

Foushee, 1993). Described below are the three flight 

inspection aircraft accidents (two fatal and one nonfatal) that preceded the development of the FAA's CRM 

awareness training for its flight inspection aircrews. 

Information concerning these accidents is documented 

in the NTSB accident report on N82 (NTSB, 1994). 

On October 26, 1993, N82, a Beech Super King 

Air 300/F, crashed into mountainous terrain near 

Front Royal, Virginia. All three crew members died in 

the crash. The plane was owned by the FAA and 

operated by the Atlantic City, New Jersey, Flight 

Inspection Area Office (FIAO). The NTSB determined that the most probable cause of the accident 

was the failure of the pilot-in-command to operate 

under visual flight rules over mountainous terrain. 

Subsequent investigations conducted by the NTSB 

revealed that the PIC had a history of ignoring visual 

flight rules, failing· to adhere to checklists, and not 

acknowledging inquiries by the second-in-command 

when such matters were brought to his attention. 

Furthermore, when crew members reported these infractions to the Atlantic City flight operations/scheduling supervisor (FO/SS), the supervisor failed to take 

corrective action (NTSB, 1994). 

In addition to the formal report on N82, two earlier 

flight inspection aircraft accidents also involved Groblems with situational awareness, communications, 

and overall crew coordination. On November 2, 1988, 

a Rockwell Jet Commander (1121A) operated by the 

Atlantic City FIAO crashed near Oak Grove, Pennsylvania, killing all three crew members. Based on their 
investigation, the NTSB determined that the aircraft 

had entered an area of moderate icing conditions and 

was in a holding pattern while the crew checked on 

flight inspection equipment malfunctions. The crew 

apparently noticed ice accretion and activated the 

surface de-icing system. Ice broke lose and entered 

engine intakes, causing both engines to flame out. 

During emergency descent, the crew attempted to 

restart engines with no success. In addition, both 

pilots were reported to have experienced stress in their 

private lives that could also have affected their performance (NTSB, 1994). 

An earlier, non fatal accident occurred on September 29, 1986, when a Rockwell Sabreliner (NA-265) 

operated by the Oklahoma City FIAO was destroyed 

while landing near Liberal, Kansas. The NTSB determined that the flying pilot (in this case the SIC) 

touched down 21 feet short of the runway. Due to the 

impact the landing gear collapsed and the aircraft 

traveled the full length of the runway before stopping 

on a golf course a quarter of a mile away. Also cited as 

contributing to the accident was the PIC's failure to 

follow proper flight procedures.(NTSB, 1994). 

All three of the above accidents contained elements 

of ineffective crew resource management. Thus, the 

accident reports themselves, served as one means of 

identifying CRM training needs. In addition to accident reports, CRM training needs were also identified 

by analyzing the results of safety meeting discussions 

following the N82 accident, and by asking subject 

matter experts to provide narratives of problematic 

situations that they had encountered while performing a flight inspection mission. This report summarizes 

the training needs derived from the last two sources. 

METHOD 

Participants 

Subjects matter experts (SMEs) were recruited during a one-day post accident CRM awareness training 

course conducted at each of the FIAO. Training 

participants were advised that they could participate 

individually in the data collection for the training 

needs analysis, or they could provide their input to 

their FIAO' s CRM representative. Fifty-eight flight 

inspection SMEs chose to provide their individual 

4 

input. This represented 30% of the flight inspection 

workforce. Subjects consisted of PICs, SICs, and 

electronics technicians. To protect the anonymity of 

the individuals and their respective FIAOs, no demographic data were collected. In addition, all surveys 

were destroyed following data entry and analysis. 

These measures were taken to assure the participants 

that they could be candid with their responses, and 

that no punitive action could result from their participation in the survey. 

Instruments 

Two written questionnaires were used to collect 

data for the training needs analysis. The purpose of 

the first questionnaire (Safety Survey) was to identify: 

(I) CRM dimensions that needed to be addressed in 

awareness training; and (2) the organizational factors 

that needed to be addressed to support a long term 

CRM effort. The Safety Survey presented SMEs with 

109 issues, extracted verbatim from written summaries of theN ovember 1993 safety meeting discussions. 

For each safety issue, SMEs indicated which of 13 

performance categories most applied (see Table 1). 

Multiple categories could be assigned to a given safety 

issue. Definitions for the first 10 categories were 

derived from commonly accepted CRM dimensions 

(Bowers, Morgan, Salas, & Converse, 1993; FAA, 

1993). These categories represented potential CRM 

awareness training modules. Categories 11 and 12 

(skill proficiency and organizational factors ) were 

included, based on the literature previously reviewed. 

Category 13 (CRM Dimension Not Specified) was 

included for completeness. 

Two kinds of analysis were planned for the Safety 

Survey. First, safety issues would be sorted by the 

performance categories to which they most applied. 

Second, cluster analytic techniques would be used to 

determine the hierarchical structure of the safety issues and how they related to overall mission success. 

Whereas the first analysis would identify specific 

training needs, the second analysis would provide a 

broader structural context for interpreting those needs. 

The purpose of the second questionnaire (Incident 

Survey) was to identify the phases of flight that were 

most problematic from a CRM perspective, as well as 

to provide insight concerning the cause of those 
TABLE 1 

Performance Categories 

1. Mission Analysis*- includes monitoring, allocating, and coordinating the resources of the crew and the 

aircraft; prioritizing tasks; setting goals and developing plans to accomplish the goals; and creating 

contingency plans. 

2. Situational Awareness* -refers to identifying the source and nature of problems, maintaining an accurate 

perception of the aircraft's location relative to the external environment, and detecting situations that 

require action. 

3. Decision Making* - includes identifying possible solutions to problems, evaluating the consequences of 

each alternative, selecting the best alternative, and gathering information needed prior to arriving at a 

decision. 

4. Communication* - includes sending, receiving, and acknowledging information among crew members in 

a way that facilitates the accurate transfer of information 

5. Crew Interpersonal Climate (CC)** - Refers to the overall interpersonal atmosphere of the crew. It 

includes the way interpersonal conflicts are resolved, the degree to which members enjoy working 

together, the shared values they have about their profession, and the degree of comfort the crew has with 

the way crew activities are coordinated. 

6. Leadership* refers to directing the activities of others, monitoring and assessing the performance of crew 

members, motivating members, and communicating mission information. 

7. Adaptability*- refers to the ability to alter one's course of action as necessary, maintain constructive 

behavior under pressure, and adapt to internal or external changes. 

8. Assertiveness* - refers to the willingness to make decisions, demonstrating initiative, and maintaining 

one's position until convinced otherwise by facts. 

9. Workload Management*** - refers to the ability to schedule, structure, and coordinate mission activities 

so as not to jeopardize situational awareness during any phase of the mission. 

10. Life Stress*** - refers to stressors outside of the context of work which interfere with a person's ability to 

perform as expected by self and crew members. 

11. Skill Proficiency- refers to technical skills, the absence of which adversely impacts the crew's confidence 

in their ability to carry out the mission. 

12. Organizational Factors- refers to formal (regulations) and informal (imposed by a given manager) 

policies and procedures which constrain the way a crew should ideally function. 

13. CRM Not Specified - refers to any CRM principle not specified in the above list. 

*After Bowers, et al. (1993) definitions of crew coordination demands. 

**After Feldman (1968), 3-factor model of cohesion. 

***After FAA Crew Resource Management Training Advisory Circular, 120-51A (1993). 

5 
problems. SMEs were given written instructions asking them to describe problematic situations that they 

had experienced while performing a flight inspection. 

The instructions were modeled after Flanagan's ( 1954) 

critical incidence methodology, commonly used to 

identify behaviors that differentiate between varying 

levels of performance. In describing their situations, 

SMEs were instructed to include: (1) the specific problem that was encountered, (2) the phase of the flight in 

which the problem occurred, (3) what led up to the 

incident, ( 4) what crew members did that was effective or 

ineffective, and (5) the impact that the problem had on 

the success of the overall mission. Data from this survey 

were subject to content analyses by the authors. 

Procedures 

Both the Safety and Incident surveys were administered during the one day post accident CRM awareness training conducted at each of the FIAOs. The 

training seminar had a five fold purpose: ( 1) to present 

an overview of what was meant by CRM, (2) to 

outline the steps that would be used to develop a three 

day flight inspection CRM awareness course, (3) to 

establish trust and rapport with the flight inspection 

crews, ( 4) to deliver a training module on aeronautical 

decision making, (5) to conduct small group discussions after viewing an aircraft accident scenario similar to the N82 accident. After outlining the CRM 

course development process, time was set aside for 

volunteers to participate in the identification ofCRM 

training needs .. Volunteers were briefed on the purpose of the training needs analysis and the part it 

would play in the course development process. Afterwards, a question and answer period was conducted to 

alleviate any doubts or concerns participants had 

about the two surveys. Verbal and written instructions were then provided before administering each 

survey. A survey administrator was present to answer 

any additional questions. No time limit was imposed. 

RESULTS 

Safety Survey 

The first step in analyzing the Safety Survey was to 

construct a 109 by 13 frequency matrix in which the 

rows contained the 109 issues extracted from the 

6 

safety meetings, and the columns contained the 13 

performance categories identified in Table 1. Cell 

values represented the frequency with which a given 

safety issue was matched to a particular category. The 

maximum value for any cell was 58, corresponding to 

the number of people who participated in the survey. 

From this frequency matrix, each column was sorted 

based on cell values. This produced a priority listing 

of safety issues associated with each training category. 

The results of this analysis appear in Table 2. 

In the first column of Table 2, training categories 

are presented in descending order based on the frequency with which they were endorsed. The percentages of endorsement are indicated in parenthesis. 

Over half of the problems discussed during the safety 

meetings concerned problems associated with management (20%), crew interpersonal climate (11 %), 

situational awareness (I 0%), and leadership (1 0%). 

Decision making, life stress, adaptability, and 

assertiveness together accounted for the bottom 11% 

of the endorsements. Presented in the second column 

ofT able 2 are the training needs that were identified 

for each training category. These include both training for management to support CRM (indicated under Organizational Factors) as well as training for 

crew members themselves. Notice that no training 

needs were identified under the "assertiveness" category. This does not suggest that there is no need for 

assertiveness training. It simply means that survey 

respondents did not indicate that assertiveness was 

related to the issues raised during the safety meetings. 

To determine the hierarchical structure of the Safety 

Survey, the frequency matrix was converted into a 

proximity matrix, using squared Euclidean distances 

as a measure of similarity. Clusters were then formed 

using Ward's method in SPSS for Windows version 

6.0. Ward's method forms subgroupings (clusters) 

based on maximizing the ratio of between-duster 

variance over within-cluster variance. The resulting 

dendogram depicting the hierarchical relationship of 

five interpretable clusters is shown in Figure 1. These 

clusters include: (1) technical skills, (2) organizational stressors, (3) ~rew stressors, (4) situational 

awareness, and (5) planning and decision making. 

Training themes associated with each of five clusters 

are shown in Table 3. 
TABLE 2 

Crew Resource Management Training Needs 

Performance Categories Ranked by Training Needs 

Percentage of Endorsement (%) 

Organizational Factors (20%) Need for management to: "walk the talk;" develop and enforce well 

defined policies and procedures; understand the nature of the flight 

inspection mission and how managerial performance affects crew 

performance; establish standardization; and create an atmosphere in 

which candid exchanges about safety can occur. 

Crew Interpersonal Climate (11 %) Need for the entire crew to participate in creating a safe flying 

environment; to be clear on their respective team roles; and to resolve 

interpersonal conflicts before flying. 

Situational Awareness (10%) Need for the entire crew to participate in altitude and aircraft position 

awareness. 

Leadership (10%) 

Communications (9%) 

Mission Analysis (8%) 

Skill Proficiencies (8%) 

Workload Management (7%) 

CAM Dimension not specified (6%) 

Decision Making (4%) 

Life Stress (3%) 

Adaptability (2%) 

Assertiveness (2%) 

Need for the Pilot In Command to clarify roles each crew member will 

play prior to flying the mission; and to distribute the workload so that 

the PIC is not over tasked. 

Need for a pre-flight briefing highlighting the major phases of the 

mission so that the entire crew is prepared, and the need to.create a 

professional climate for open communication exchanges among crew 

members. 

Need to include organizational factors affecting the mission, in 

particular, the scheduling of flight inspections, the availability of aircraf 

and the conduct of pre-flight briefings 

Need to maintain currency and the need to train to standardization. 

Need to coordinate with air traffic control, and to reduce scheduling 

pressures. 

No training needs identified. 

Need for the crew to take charge of decisions affecting the success of 

the mission, in particular deciding on whether or not it is safe to fly. 

Need to quickly resolve organizational uncertainty (office closings and 

relocations). 

Need to adjust to flight check restrictions imposed by air traffic control 

without creating an adversarial situation, and to adapt to itinerary 

changes rather than being distracted by them. 

No training needs identified. 

7 
Figure 1 shows that mission success consists of two 

clusters, one that deals with Technical Performance, 

and one that deals with Crew Participation. Technical 

performance is further divided into issues related to 

technical flying skills, as well as stressors that act to 

interfere with the performance of those skills. This 

interference consists of factors residing within the 

organization and factors that reside within the crews. 

Crew participation is comprised of issues related to 

maintaining situational awareness, and planning and 

decision making. 

Table 3 shows that the training needs that emerge 

from the cluster analysis may be further divided into 

two categories: (1) areas of personal concern, and (2) 

areas of personal control. This division is also shown 

in Figure 1 by the dashed line. Contained in areas of 

personal concern are those issues that concern flight 

inspection crews, such as technical skills and organizational stressors, but are outside their personal control. For example, flight crews can request changes in 

the kind of training that they receive, however, they 

do not have the power to make those changes. In 

contrast, areas of personal control, such as crew stressors, 

situational awareness, planning, and decision making, are more strongly associated with factors that 

crew members themselves have the power to change. 

Incident Survey 

In analyzing the results of the Incident Survey, a 

problem arose due to incomplete data. Of the five 

factors that SMEs were asked to include in their 

narratives, only the first two were reported with any 

degree of clarity. Subsequent discussions with the 

SMEs revealed they were more comfortable talking 

about problems than writing about them. Because of 

problems with incomplete data, the results of the 

content analyses are reported for heuristic purposes 

only, and should not be construed as representing 

scientific rigor. 

Data from the CRM Incident Survey were coded by 

the authors according to the phase of the mission in 

which problems occurred, as well as the most probable 

cause of the problem (i.e., weather, person, or equipment failure). This information is provided in the first 

four columns of Table 4. The numbers recorded in 

each cell refer to the frequency with which a probable 

Figure 1 

~ 0 t.l ('.1 

fl) 

111 

('.1 

I 

-

0 

Hierarchical Classification of Training Needs 

Technical 

Flying Skills 

Concern 

Organizationa,l 

Stressors 

8 

Control 

Crew 

Stressors 

Situational 

Awareness 

Planning and 

Decision Making 
Clusters 

Areas of Concern 

Technical Skills 

Organizational Stressors 

Areas of Control 

Crew Stressors 

Situational Awareness 

Planning and Decision 

Making 

TABLE 3 

Cluster Themes 

Dominant Themes 

Train to a standard, and maintain currency with equipment 

Mission-safety conflict, aircraft maintenance problems, and 

lack of organizational support 

Role conflict, role ambiguity, low morale 

Altitude and position awareness 

Preflight briefings, aircraft safety 

TABLE 4 

Probable Cause of Problems Encountered at Each Phase of the Flight Inspection Mission 

Number of Number of Number of 

People Weather Equipment Row Frequencies 

Mission Phase Problems Problems Problems Row% 

Pre-Departure 4 0 0 4 6.9% 

Take-off 8 0 2 10 17.2% 

Enroute 5 5 3 13 22.4% 

Inspection 20 3 2 25 43.2% 

Landing 2 2 2 6 10.3% 

Column 

Frequencies 39 10 9 58 

Column% 67.2% 17.3% 15.5% 100% 

9 
cause occurred within a given phase of the mission. In 

addition, Table 4 also shows column and row frequencies and percentages. 

The most probable cause of the problems reported 

in Table 4 was associated with people-related issues 

(67.2%) involving interactions among crew members 

and with air traffic control and airway facilities ground 

maintenance. This was followed by weather-related 

problems (17.3%), and equipment malfunctions 

(15.5%). Most of the incidents occurred while performing the flight inspection (43.2%), followed by 

enroute (22.4%), take-off(17.2%), landing (10.3%), 

and pre-departure (6.9%). A content analysis of the 

people-related problems suggested three problematic 

areas involving: (1) errors in information processing 

(e.g., situational awareness, and decision making); (2) 

errors in information transfer (i.e., communications); 

and (3) factors associated with the way crew members 

related to each other (e.g., interpersonal climate). 

A second content analysis of the CRM incidents 

was conducted to identify the CRM performance 

dimensions associated with a given incident. Following the same procedures used by SMEs in the Safety 

Survey, the authors identified the performance categories (see Table 1) that most applied. Percentages of 

incidents associated with a given performance category were then derived as reported for the safety 

survey. This information is presented in column two 

ofT able 5. For comparison purposes, the results of the 

Safety Survey are also shown in column 3. For the 

most part, the rank order (based on percentage values) 

of the CRM categories are the same for both surveys. 

Notice, however, that there is an extreme difference 

associated with two CRM categories, decision making 

and organizationai factors. Whereas decision making 

was a significant factor (22%) associated with the 

CRM incidents, it played a relatively minor role in the 

CRM safety issues (4%). Similarly, organizational 

factors were the dominant issues discussed in the 

CRM safety survey (20%), but they contributed 

little (1 %) to the factors associated with the CRM 

incidents. These differences have important implications for CRM training and will be discussed 

later in this report. 

TABLE 5 

Performance Category Comparisons between Incident Survey and Safety Survey 

Incident Survey Safety Survey 

Percentage of Percentage of 

Performance Categories Occurrences Endorsements 

Decision Making 22% 4% 

Crew Interpersonal Climate 22% 11% 

Situational Awareness 20% 10% 

Communications 15% 9% 

Mission Analysis 8% 8% 

Assertiveness 7% 2% 

Leadership 3% 10% 

Workload Management 2% 7% 

Organizational Factors 1% 20% 

Skill Proficiencies 0% 8% 

Life Stress 0% 3% 

Adaptability 0% 2% 

CRM Dimension not specified 0% 6% 

10 

1 

, 
DISCUSSION 

The cluster analytic results of the training needs 

analysis support earlier findings that flight crew performance is dependent on three factors: (1) technical 

skills, (2) resource management skills, and (3) the 

organizational context in which flight crews operate. 

As shown in Figure 1, technical performance and crew 

participation form two distinct classifications of training needs, with organizational and crew contextual 

factors acting as stressors that interfere with the technical performance of flight inspection crews. Furthermore, the training needs that emerge from this 

classification may be further divided into two categories: (1) those factors over which flight crews have 

control, and (2) factors that concern flight crews, but 

control of which resides within the organization. 

Using the structural framework of Figure 1, several 

training implications are especially worth noting. 

First, crew members reported problems with the 

technical training they received. In particular, pilots 

complained that some of them were not getting enough 

flying time which made them feel not as technically 

proficient as they would have liked. In addition, pilots 

were not always checked out on equipment modifications prior to conducting a flight inspection mission. 

Since the single most important resource that crew 

members possess is technical skill, technical training 

deficiencies, such as these, must first be addressed for 

crew resource management training to have a positive 

effect on crew performance. 

Second, the results of the needs analysis suggested 

that crews would benefit by more active crew participation, particularly with regard to three areas: (1) premission briefings, (2) decisions about safety, and (3) 

maintaining aircraft situational awareness. The importance of a pre-mission briefing cannot be over 

emphasized. It is during this briefing that crews develop what Cannon-Bowers, et al. (1993) and others 

have called a shared mental model of the mission (for 

a review of the literature see Klimoski & Mohammed, 

1994). A shared mental model may be thought of as a 

common set of expectations about what will occur 

during a mission. Included in this mental model are 

expectations concerning the time-sequencing of mission events, the tasks to be performed, and how 

'11 

individual efforts will be coordinated (Cannon-Bowers, et al., 1993). When a pre-mission briefing is 

lacking, crew members must rely on past experiences 

to guide their performance. The implicit assumption 

is that everyone is operating with the same set of 

expectations. Unfortunately, it is usually under nonroutine conditions that the fallacy of this assumption 

surfaces (Orasanu, 1994). 

In addition to establishing a shared mental model 

of the mission, the pre-mission briefing is an excellent 

time to address the leadership, communications, and 

crew climate concerns, as indicated in Table 2. How 

a PIC conducts a pre-mission briefing sets the stage 

for the communication patterns that will emerge 

among crew members (Hackman, 1993). If the PIC 

provides a well organized briefing and solicits input, 

then he or she establishes an atmosphere of professional competency in which crew members feel free to 

voice concerns. Furthermore, to the extent that crews 

can resolve differences of opinions prior to flight, they 

are less likely to be distracted by those differences 

during the course of the mission. By involving all 

three crew members (PIC, SIC, and ET) in decisions 

regarding flight safety, crews create a climate in which 

flight safety is a shared responsibility. The need for 

such involvement is reflected in the decision making 

and situational awareness CRM training needs presented in Table 2. 

A third implication derived from the needs analysis 

concerns the effects that organizational and crew 

stressors have on the technical performance of a flight 

inspection mission. Crew stressors, such as role ambiguity and role conflict, have already been addressed as 

issues that should be clarified during a pre-mission 

briefing. Organizational stressors, however, require 

further elaboration. 

Flight inspection crews are mission oriented. Their 

job is to fly the aircraft over a prescribed course under 

the guidance of ground-to-air flight navigation equipment, while onboard electronic equipment re.cords 

the aircraft's relative position in space. The ground 

facility is certified when the recorded data indicate the 

aircraft flew the desired course within specific tolerances. Facilities must be flight checked and certified 

prior to use when ground navigation aids or equipment are installed, repaired, removed, or replaced and 
when required periodic inspections are due. Navigational aids may also be flight checked following aircraft accidents. 

Due to a variety of reasons (such as a facility outage 

at O'Hare International, a high density traffic airport), there can be increasing pressure on flight crews 

to perform flight checks during marginal weather or 

during off peak traffic hours late at night when crews 

tend to be fatigued. Crews consider these conditions 

to be unsafe, and problems can arise when flight crews 

perceive (correctly or incorrectly) that their management is more concerned about getting the job done 

than they are about their safety. 

In a study conducted by Witt, Hilton, & Helman 

(1994), workplace safety perceptions were found to be 

directly related to perceived managerial support for 

safety. Moreover, management was perceived to be 

most supportive of safety when employees experienced a work climate that was "relatively free from 

politics and unfairness, and in which everyone shared 

a common view regarding the importance of safety." 

The authors concluded that how managers deal with 

everyday problems sets the tone for how they will deal 

with safety related issues. This conclusion is further 

supported by the organizational factor concerns reported in Table 2. In particular, flight crews expressed 

the need for management to understand how managerial performance affects crew performance, as well as 

the need for management to create an atmosphere in 

which candid exchanges about safety can occur. 

Concerns about organizational stressors are valid 

and need to be addressed by the organization; however, caution is advised when addressing these issues 

during CRM awareness training. The inclusion of 

organizational factors is likely to shift the focus of 

CRM training from what Covey (1989) calls "areas of 

control" to "areas of concern." Areas of concern are 

those issues in which crews have a vested interested, 

but little personal control over the outcome. In contrast, areas of control represent issues in which the 

crews, themselves, have the power to effect change. 

Covey further notes that there is a tendency for people 

to spend a considerable amount of time attempting to 

address areas of concern while neglecting areas of 

control. This division is shown in Figure 1 and is 

reflected in the frequency that organizational factors 

12 

were discussed during the safety meetings. Of all 

CRM categories, organizational factors was the most 

frequently discussed, but it is the area over which 

crews have the least personal control. 

One way of keeping CRM awareness training focused on areas of control is to develop training scenarios around situations that crews have actually 

experienced while in flight. This can be seen by the 

Table 5 comparison between the rank ordering of 

CRM categories derived from the Safety Survey, and 

those derived from the CRM Incidents. Whereas 

organizational factors represented 20% of the issues 

discussed in the safety meetings, they represented only 

1 o/o of the CRM incidents reported by flight inspection crews. 

The above results further suggest that by simply 

changing the field of reference from problems in 

general {as discussed in the November 1993 safety 

meetings) to problems in the air (as reported in the 

Incident Survey), crew members are more apt to take 

personal responsibility for their actions. This projected outcome forms the basis of the practice and 

feedback phase ofCRM training (FAA, 1993). In this 

phase, instead of evaluating the performance of others 

(as in watching accident recreations), crews are videotaped while performing under mission simulated conditions. Afterwards, crews view the video playback of 

their performance and critique themselves on their 

resource management skills. Factors affecting crew 

performance are often self-evident, however, a trained 

facilitator may be necessary to ensure that crews perform a self critique on more than just their technical 

performance (Butler, 1993). 

Finally, in support of developing CRM awareness 

training, data from the CRM incident survey (see 

Table 4) suggest that most flight inspection CRMrelated problems (43.2%) occurred while performing 

aircraft maneuvers during the airborne testing phase 

of flight (i.e., the actual flight inspection). These 

problems involve communication errors, errors in 

information processing, and factors associated with 

the crew's interpersonal climate. Although generic 

CRM training materials in those areas exist, the FAA 

CRM Advisory Circular (120-51A) cautions against 

using such material without first adapting it to reflect 

the nature and needs of the organization. 
The most successful CRM training programs are 

those which are internally developed (Captain J. E. 

Carroll, retired United Airlines Vice President of 

Safety and Training, personal communications, May 

17, 1994). This is because crew resource management 

_ is more than just a course, it is a philosophy or way of 

knowing and thinking about flying a mission. This 

philosophy is manifested in an organization's culture. 

By developing an internal course, organizations must 

address such questions as: (1) what is the purpose of 

the flying mission and what are its priorities, (2) how 

do organizational policies and practices affect the 

flying mission, (3) what really takes place when a 

flight crew flies a mission, (4) how prepared are flight 

crews for accomplishing their mission, and (5) what 

habits have flight crews developed that enhance or 

impede the success of a mission? When a generic 

course is used to deliver CRM training, there is a 

tendency for organizations to side-step many of these 

harder questions, and instead just implement a course 

as a "quick fix." As a consequence, such CRM programs tend to fade over time. 

There is a down side to developing a CRM program, however, when internal resources are used: it 

takes time, and mistakes will be made. Thus, unless an 

organization is willing to commit the necessary resources to improve upon initial efforts, its CRM 

program is likely to fall below expectations. In addition, flight crews should understand that their CRM 

program will evolve over time, with its success dependent upon the degree to which flight crews are actively 

involved in course development, program implementation, and program evaluation. For these reasons, the 

following recommendations are made: (1) develop a 

CRM awareness course from within the FAA, rather 

than trying to fit A VN to a generic CRM course; (2) 

create a CRM program that can be sustained over the 

long term, rather than attempting to implement a 

"quick fix;" (3) involve the flight crews in the course 

development process; ( 4) develop a common language 

between the flight crews and A VN management for 

discussing CRM issues; (5) develop a mission-based 

flight inspection simulation training environment for 

the practice and feedback phase ofCRM training; and 

(6) adopt organizational reinforcers to support the 

principles of CRM. 

13 

REFERENCES 

Andrews, D., Waag, W., & Bell, H. (1993). Training 

technologies applied to team training: Military 

examples. In R. Swezey & E. Salas (Eds.), Teams: 

Their training performance. Norwood, NJ: Ablex, 

pp. 283-327. 

Bowers, C., Morgan, B., Salas, E., & Prince, C. (1993). 

Assessment of coordination demand for aircrew 

coordination training. Military Psychology, 5, 95-

112. 

Butler, R. (1993). LOFT: Full-mission simulation as 

crew resource management training. In E. Wiener, 

B. Kanki & R. Helmreich (Eds.), Cockpit resource 

management. San Diego: Academic Press, 231-

259. 

Cannon-Bowers, J., Salas, E., & Converse, S. (1993). 

Shared mental models in decision making. InN. 

Castellan (Ed.),Individualandgroupdecision making. Hilsdale, NJ: Lawrence Erlbaum Associates, 

pp. 221-246. 

Cooper, G., White, M., & Lauber, J (Eds.) (1979). 

Resource management in the flight deck (NASA 

Conference Publication No. 2120). Moffett Field, 

CA: NASA Ames Research Center. 

Covey, S. (1989). Seven habits of highly effective people: 

Restoring the character ethic. New York: Simon & 

Schuster. 

Driskell, J., & Adams, R. (1992). Crew resource management: An introductory handbook (DOT/FAA/RD92/26). Springfield: VA: National Technical Information Service. 

Federal Aviation Administration (1993). Crew resource 

management training(Advisory Circular 120-51A). 

Washington, DC: author. 

Feldman, R. (1968). Interrelationships among three 

bases of group integration. Sociometry, 31, 30-46. 

Flanagan, J. (1954). The critical incidence technique. 

Psychological Bulletin, 51, 327-258. 

Gaddy, C., & Wachtel, J. (1993). Team skills training 

in nuclear power plant operations. In R. Swezey & 

E. Salas (Eds.), Teams: Their trainingperformance. 

Norwood, NJ: Ablex, pp. 379-396. 
Hackman,]. (1993). Teams, leaders, and organizations: 

New directions for crew oriented flight training. 

In E. Wiener, B. Kanki, & R. Helmreich (Eds). 

Cockpit resource management. San Diego: Academic 

Press, 47-70. 

Hartel, C., & Hartel, G (1995). Controller resource 

management - What can we learn from aircrews 

(DOT/FAA/AM-95/21). Springfield, VA: National Technical Information Service. 

Helmreich, R., & Foushee, H. (1993). Why crew resource management? Empirical and theoretical bases 

of human factors training in aviation. In E. Wiener, 

B. Kanki & R. Helmreich (Eds.), Cockpit resource 

management. San Diego: Academic Press, 3-45. 

Howard, S., Gaba, D., Fish, K., Yang, G., & Sarnquist, 

F. (1992). Anesthesia crisis resource management 

training: Teaching anesthesiologists to handle critical events. Aviation, Space, and Environmental 

Medicine, 63, 763-770. 

Klimoski, R., & Mohammed, S. (1994). Team mental 

model: Construct or metaphor. journal of Management, 20, 403-417. 

Lauber,]. (1987). Cockpit resource management: Background studies and rationale. In H. Orlandy & H. 

Foushee (Eds.), Cockpit resource management 

training: Proceedings of the NASA/MAC workshop,. (Tech. Rep. No. NASA CP-2455). Moffett 

Field, CA: NASA-Ames Research Center. 

National Transportation Safety Board (1994). Aircraft 

accident report: Controlled flight into terrain, Federa/Aviation Administration, Beech Super King Air 

300/F, N82, Front Royal Virginia, October 26, 

1993 (NTIS PB94-910405). 

14 

Orasanu, J. (1994). Shared problem models and flight 

crew performance. InN. Johnston, N. McDonald, 

& R. Fuller (Eds.),Aviation Psychology in Practice. 

Brookfield, VT: Ashgate, 225-285. 

Prince, C., Childester, T., Bowers, C., & CannonBowers, J. (1993). Aircrew coordination- Achieving teamwork in the cockpit. In R. Swezey & E. 

Salas (Eds.), Teams: Their training performance. 

Norwood, NJ: Ablex, 329-354. 

Ruffell Smith, H. (1979, January). A simulator study of 

the interaction of pilot workload with errors, vigilance, and decisions. (NASA TM-78483). Moffet 

Field, CA: NASA-Ames Research Center. 

Salas, E., Dickinson, T., Converse, S., & Tannenbaum, 

S. (1992). Toward and understanding of team 

performance. In R. Swezey & E. Salas (Eds.), 

Teams: Their training performance. Norwood, NJ: 

Ablex, 3-29. 

Statistical Software Package for Social Sciences for Windows, Version 6.0 (1993). SPSS, Inc.: Chicago. 

Swezey, R., & Salas, E. (1993), Teams: Their training 

performance. Norwood, NJ: Ablex. 

Wiener, E., Kanki, B., & Helmreich, R. (1993). Cockpit 

resource management. San Diego: CA.: Academic 

Press. 

Witt, Hilton & Helman (1994) Management influences 

on perceived safety conditions. Presented at the annual conference of the American Psychological 

Society: Washington. 

-tfu.S. GOVERNMENT PRINTING OFFICE: 1996 • 559-079/40013