NAWCTSD/UCF/FAA

Partnership for Aviation Training

**Guidelines**

**For Situation Awareness Training**

**Carolyn Prince, Ph.D.**

# NAWCTSD/FAA/UCF Partnership for Aviation Team Training

![](media/image1.emf){width="1.49375in" height="1.2604166666666667in"}
![](media/image2.emf){width="1.3611111111111112in"
height="1.3680555555555556in"} ![](media/image3.emf){width="1.93125in"
height="1.1979166666666667in"}

Naval Air Warfare Center Federal Aviation Administration University of
Central Florida

Training Systems Division AAR-100 Department of Psychology

*Orlando, Florida Washington, DC Orlando, Florida*

**INTRODUCTION TO THE GUIDE FOR TRAINING**

**CREW SITUATION AWARENESS**

**Underlying Assumptions for This Document**

This guide was developed with the following assumptions:

1.  Crew situation awareness is the result of each crewmember's
    situation awareness and team processes.

2.  Although situation awareness may differ from one person to another
    and may be partially the result of experience, personality, other
    inherent characteristics, and motivation, there are actions that
    crew members can take that help build and maintain situation
    awareness.

3.  The major purpose of CRM training is to reduce human error caused
    accidents. Problems with situation awareness can be readily linked
    to human errors. The best way to train is not to focus on the errors
    themselves but to build the teamwork skills required for effective
    performance.

4.  Training should include active practice and feedback. Depending on
    the level of skill being trained, this can be accomplished in LOFT,
    SPOT, or in the use of realistic scenarios with training devices
    that do not necessarily have high fidelity.

5.  Training should follow a pattern that includes: Information,
    Demonstration, Practice, and Feedback. Lecture, discussion,
    videotape observation should be followed by opportunities for
    crewmembers to practice actions and to receive feedback (debrief).

6.  Consistency in the training is important. Actions and ideas
    introduced in the classroom should be practiced in the simulation.
    These actions should then be de-briefed by the instructor.

```{=html}
<!-- -->
```
7.  The quality of feedback, or debriefing is important and instructors
    should be trained in observing and providing specific debriefs.

**Useful Definitions**

> **Individual situation awareness** has been defined in a number of
> ways. Three definitions are:

1.  Situation awareness is the perception of the elements in the
    environment within a volume of time and space, the comprehension of
    their meaning and the projection of their status in the near future
    (Endsley, 1987)

2.  Situation awareness refers to the up to the minute cognizance
    required to operate or maintain a system (Adams, Tenney and Pew,
    1995)

3.  Situation awareness is adaptive, externally directed consciousness.
    At a very simple level, situation awareness is an appropriate
    awareness of a situation. (Smith and Hancock, 1995)

> **Team or crew situation awareness** has been defined less often than
> individual situation awareness. Two current definitions are:

1.  Team situation awareness involves two critical but poorly understood
    abstractions: individual situation awareness and team processes in a
    highly interactive relationship (Salas, Prince, Baker and Shrestha,
    1995)

2.  Team situation awareness is the crew's understanding of flight
    factors that affect (or could affect) the crew and aircraft at any
    given time (Wagner and Simon, 1990).

> **Situation assessment** is defined as the process of achieving
> situation awareness. It is the process of information acquisition and
> interpretation (Adams, Tenney and Pew, 1995) that leads to the
> product, defined as situation awareness.

**Sources for the Guide**

The information in this guide has come from research in aviation:
experiments, ASRS analysis, structured and critical incident interviews
with crewmembers, and observations of aviation crews. Each guideline
refers the reader to a brief description of the research from which it
was derived. Research summaries can be found at the end of this document
in Appendix B. A complete citation for the research is provided with
each description in the Appendix for more information. A reference
section is included for those who would like additional information on
situation awareness to add to their understanding of the concept.

Research conducted with funding from the Federal Aviation
Administration, Office of the Chief Scientist as part of a project on
crew situation awareness is described in Appendix B. Other relevant
research is listed in the Reference section found at the end of the
Guidelines before Appendix A.

**Organization of this Document**

These guidelines for training situation awareness are grouped according
to the area of training they address and are presented in four sections.
The sections are:

> **Pre-Training Considerations:** These guidelines for developing a
> training program for situation awareness address some elements for the
> training developer to consider before designing a program.
>
> **Guidelines for Training Program Content:** The guidelines in this
> section include information about what the crewmembers should know and
> do to maintain awareness as a crew. They also describe situations that
> pose a particular threat to maintaining awareness.
>
> **Evaluation:** These guidelines address training instructors to
> observe and evaluate crew members' situation awareness for
> de-briefing. They contain information useful for training crewmembers
> to observe and evaluate situation awareness actions of others and of
> themselves.
>
> **Specialized Training:** (training for Captains and upgrade training
> for new Captains). This section contains guidelines that can be used
> in training programs for Captains, since there are certain actions
> that are the Captain's responsibility to initiate. These actions help
> maintain their crew's awareness.
>
> **Appendix A** General Training Tips
>
> Training Tips for Scenario Based Training
>
> Suggested Program Design
>
> for Recurrent Training
>
> for New Hires
>
> **Appendix B** Research Summaries
>
> **Appendix C** Training Materials

**THE GUIDELINES**

**Pre-Training Considerations**

The guidelines in this section will help the training designer in
deciding on class composition and on the use of class materials. Some of
these training decisions should be based on the experience level of
those crewmembers being trained. This section's format is:

I-III Guideline (in bold), followed by a short explanation (in italics)

> An example of the Guideline
>
> A training tip or an example of training based on the guideline

I.  **Recognize that crewmembers with different levels of experience and
    positions (e.g. 1,500 hours and 15,000 hours; Captains and Second
    Officers) are likely to define and conceptualize situation awareness
    differently based on their experience**. **This leads to three
    considerations for actions.**

II. consider separate classes for different experience levels and
    positions.

III. ensure that instructors are aware of the possible differences in
     conceptualizing situation awareness.

IV. carefully review already developed program materials for their
    relevance to the level of crewmember experience that will be
    represented in the training (e.g., a new hire first officer in a
    regional airline may need a different course emphasis from that for
    experienced Captains).

> *Crewmembers extend their description of situation awareness elements
> and include more elements as they gain experience. This continues
> throughout their careers. More experienced crewmember report different
> cues for recognizing a problem with situation awareness of another
> crewmember and describe more specific preparation for a flight. (See
> Appendix B Research Summary 1.)*
>
> [Example]{.underline}: Interviews with pilots at different levels of
> experience (under 1000 hours, an average of 6000 hours, and an average
> of 12000 hours) revealed clear differences in responses to questions
> that asked: 1) What is situation awareness? 2) How do you know when
> another crewmember may be having a problem with situation awareness?
> and 3) What do you need for situation awareness? Those with the least
> experience usually have flown less complex flights with a need to
> attend to fewer elements during flight. They describe situation
> awareness in terms of immediate surroundings. As crewmembers gain
> experience, they show an appreciation for their flight as part of a
> larger system and include a wide variety of elements as important for
> their awareness. Low experience level pilots use obvious signs as an
> indication that another is having awareness problems (off course, off
> altitude, staring). More experienced crew members look for subtle cues
> and even indications in situations that may lead to a loss of
> awareness (e.g., the other crew member has been momentarily distracted
> by a call from the cabin). They recognize that when they suspect that
> another has lost awareness, it may be an indication that they,
> themselves, are having a problem. In discussing what is needed for
> situation awareness, all levels of pilots emphasize preparation and
> planning. With low experience level aviators, there is more concern
> about personal physical preparation (well rested, ready for flight)
> and as experience level grows, there is more emphasis on becoming
> familiar with all possible aspects of the current flight (e.g., who
> other crew members are, cabin crew plans, runway conditions for
> take-off and landing, equipment).
>
> **[Training Tip]{.underline}**: *The training designer needs to keep
> in mind:*

-   *the term situation awareness does not mean the same to all
    crewmembers. Make the instructor aware so he/she can adjust to the
    appropriate level*

-   *low experience level crew members may not appreciate the amount of
    specific preparation on flight details needed for each flight to
    maintain crew awareness*

-   *newly hired crew members with little experience flying as a crew
    may need to be trained on how to recognize situation awareness
    problems in experienced crew members or in recognizing situations
    that may lead those crew members to need assistance with elements
    for awareness (e.g., making a PA)*

I.  **Design the course to emphasize the active role that crew members
    must take to gain and maintain crew situation awareness**

> *The objective of these guidelines is to develop training that will
> give crewmembers the information about, and practice in, actions that
> will increase the crew's awareness. Many crewmembers will enter the
> training believing that situation awareness is based on an individual
> trait, "Either you have it or you don't". (Appendix B, Research
> Summary 1). Training that involves crewmembers in active participation
> will help emphasize the active role crewmembers must take to maintain
> crew situation awareness. It may be useful to introduce the concept of
> situation assessment, the active process that leads to situation
> awareness*. *Do not make this an information/demonstration only course
> with a passive role for crewmembers.*
>
> [Example]{.underline}: The following is from the ASRS database (Report
> Number 271107*).* Took off LMT VFR and picked up clearance (IFR) to
> RDD, asked for 13000 feet and for a heading for direct RDD. Clearance
> was given as requested. After performing the After Takeoff and Cruise
> checklists (after 10000 feet) I, as PNF pulled out a magazine to look
> through. I mentioned an article in the magazine. We were both
> discussing the article when the Captain noticed we had reached an
> altitude of 13700 feet when the controller queried whether we wanted
> 14000 feet as a final. He stated, "no", and proceeded back to 13000
> feet. **I believe my complacency of kicking back before reaching
> cruise altitude and inattention by both of us contributed to the
> altitude bust**. Also, lack of aircraft standardization, since this
> plane we were flying is one of only a couple in the fleet that doesn't
> have an altitude alerter installed, also played a part. No conflict
> was reported. Supplemental information: There were a number of
> circumstances that conspired to allow me to make this error! 1) We had
> an early show and I have not been sleeping very well**. This was the
> last leg of the day and my mind was wandering.** 2) The FO does not
> always make altitude call-outs. In fact, he is quite sloppy about
> this.
>
> **Training Tip**: *Some points from the example that may be used in
> training:*

-   *Neither crew member was consistently monitoring the instruments The
    FO did not always provide needed information (altitude call-outs)*

-   *These crew members may have relied, somewhat, on equipment that was
    not there*

-   *Crewmembers are all familiar with the lower level of situation
    awareness that occurs when the crew is lulled into complacency or is
    fatigued. In a seminar format, use the example above, one of your
    own, or get the crew members to contribute an example of a case of
    an otherwise "good" crew that had a problem related to a lowered
    level of situation awareness.*

> **Training Tip for Practice and Feedback**: *Include as much active
> practice as possible in classroom training (e.g., with problem
> exercises, scenarios on tabletop trainers). Another technique that
> helps involve everyone during a class is to stop a videotape of a crew
> in a problem situation and ask class members to assume the roles of
> the crewmembers and say what they would do next.*
>
> *Be sure to include LOS, with feedback, as much as possible.*

III. **Design all the course elements (information, scenario events,
     de-briefing guides) to reflect the same training philosophy and the
     same skills**. **That is, introduce an important element in the
     Information/Demonstration phase of training and design Practice to
     include that element. Follow up with specific Feedback on that same
     element.**

> *This is basic training information that can be easily overlooked when
> training includes diverse elements such as classroom discussions and
> realistic scenarios. The scenarios may be designed for a number of
> training purposes in addition to situation awareness training, such as
> demonstrating a new procedure, or training both technical and other
> CRM skills. When a scenario is developed, the training developer
> should examine it for the learning points included in the situation
> awareness training and develop a debrief guide for instructors that
> includes those points*.
>
> [Example]{.underline}: A crew, flying a LOFT scenario, began a
> take-off roll without flap extension. Crewmembers caught their error
> before take-off. They attributed the error to a problem with their
> crew situation awareness, but were not sure how the breakdown had
> occurred. When debriefed, the instructor did not mention the problem
> nor their solution. The crewmembers elected not to bring it to his
> attention.
>
> **[Training Tip]{.underline}**: *This lapse on the part of the
> instructor left the crewmembers confused. They felt that despite the
> potential danger of the situation and the emphasis on such factors in
> the class discussions, the problem with crew awareness was not of
> sufficient importance to the instructor in the simulator to even raise
> it for discussion. Although they wanted some information from an
> observer's perspective, they did not want to remind the instructor of
> their error*.

**Training Content**

Guidelines in this section are based on information from research that
was conducted with pilots. They are important to consider when
determining the content of a training program in situation awareness. In
general, these guidelines have the following format:

I-XI Guideline, followed by an explanation or expansion of the guideline
(in Italics)

Example

Training tip for Information/Demonstration training

Training tip for Practice/Feedback training

(More general training tips are found in Appendix A.)

I. Inform crew members that 40% of the reported incidents in the ASRS
data base occurred when only one crew member had a problem with
situation awareness.

> *In an analysis of the ASRS database, approximately 25 percent of the
> incidents occurred when the Captain was not aware of an important
> element in the situation. Ten to 15 percent of the incidents occurred
> when either the First Officer or the Second Officer alone had a lapse
> in awareness. This suggests that crew members need to monitor one
> another and talk about the situation (as well as listen to one
> another) so all individuals in the crew can maintain a level of
> situation awareness sufficient for the required* *tasks***.**
> *(Appendix B, Research Summary VII.)*
>
> [Example]{.underline}: *An example of this problem can be found in the
> ASRS database, Report Number 255739*. After landing on Runway 32,
> Tower Controller cleared us to turn left on Taxiway A, to hold short
> of Runway 22. Although the FO later recalled hearing and reading back
> these instructions, both the clearance and the acknowledgment somehow
> escaped my attention and he failed to ensure that I was "in the loop".
> During subsequent taxi, completion of After Landing checklist, and
> other distractions, we became focused on our taxi route, which was
> complicated by the partial closure of Taxiway A and several lighted
> barricades associated with this. In the course of negotiating this
> obstacle course, and without further communications from the FO or the
> Tower Controller, I proceeded across Runway 22 without clearance, as
> another aircraft (commuter, type unknown) was landing on that runway.
> Although collision was not imminent and evasive action not required,
> it was luck alone which prevented this (or much worse) from occurring.
>
> **Training Tip:** *This example helps illustrate a Captain's error,
> based on incomplete or erroneous knowledge of the situation. The FO
> had apparently assumed that the information from the Tower Controller
> had been heard by the Captain and had not made sure that the Captain
> had the needed instructions. Crewmembers in the training class should
> be able to supply their own examples of situations where one
> crewmember could have made the others aware before an incident
> occurred. This guideline demonstrates that crew situation awareness is
> a requirement until the crew leaves the cockpit**.***
>
> **Training Tip, Practice/Feedback*: ****When de-briefing crew members
> after a training scenario, call attention to any incident where one
> crew member reminded another, or one questioned another. This is an
> indication that one was aware and that the reminder or the question
> helped increase the awareness of the other crewmember. It helps
> establish the value of one's crewmembers as an information source for
> awareness*.

**II. Demonstrate to crewmembers that crew situation awareness requires
a high level of flight knowledge about the specific flight (familiarity
with the details of the flight) among the crewmembers and this requires
involvement of all crewmembers**.

> *This establishes the importance of flight knowledge for gaining and
> maintaining situation awareness when combined with Guideline I above.
> It suggests three important activities for situation awareness:
> preparation, communication, and monitoring, (addressed below in
> Guidelines IV-VI). All help maintain a high level of flight knowledge
> for a particular flight. (See Appendix B Research Summary X)*
>
> [Example]{.underline}: *From the ASRS data base (Report Number 270923)
> the following is an example of problems with crew situation awareness
> resulting from a crew member's lack of specific flight knowledge and
> the second crew member's failure to monitor the flight progression*.
> This was the second leg of B767/757 supervised operating experience as
> a Captain. This was my first leg as a Captain on the B757. I was
> occupying the left seat and a line check airman was in the right seat.
> After some problems getting approach clearance from ATC we were
> cleared to land on Runway 24R at LAX. I was somewhat uptight with my
> first landing on the aircraft as Captain. After landing we turned off
> the runway onto Taxiway 65 and were following a B737. At some point
> after exit we were told to hold short of Runway 24L. I called for the
> after landing checklist and deliberately taxied while the check airman
> became involved in the checklists and radio calls to Ramp Control. It
> had been several years since my last time at this airport and I did
> not realize the close proximity of the 2 runways to each other. I was
> concentrating on taxiing on the centerline and soon realized we were
> approaching the runway. I brought the aircraft to an abrupt stop.
> Almost immediately after we stopped, the Tower called and asked if we
> were 'holding short'. At this time the check Captain looked up and
> then quickly stated that we 'have intruded'. Before Tower answered we
> heard an aircraft declare "aborting takeoff'. At this time we were
> still on Taxiway 65 and not on the runway. Shortly thereafter Tower
> cleared us to cross Runway 24L. At no time did our aircraft enter onto
> Runway 24L until we were cleared to cross by Tower. After crossing the
> runway, we were told by Ground Control to contact the Tower by
> telephone. We were told by phone that a deviation would be filed by
> Tower. I believe that several factors contributed to cause this
> occurrence. The chief factor is the extremely close proximity of the
> two runways and the complicated 'small print' taxiing procedures in
> use at LAX. **The next main factor was that I was both inexperienced
> on this aircraft and unfamiliar with the distances between runways. In
> retrospect, I should have waited to call for after landing checklist
> until after we were clear of all active runways rather than
> distracting the check airman whose experience with the aircraft and
> the airport could have aided me greatly at this point.** The Check
> Airman added that because the Captain was a line check airman on the
> B737, had been a FO on the B757 and had performed proficiently on the
> trip, that he had felt some complacency and had allowed himself to be
> distracted by FO duties.
>
> **Training Tip*: *** *This example illustrates that awareness was not
> based on **crew** flight knowledge but on the knowledge relevant to
> the flight of a single crewmember (and his knowledge was deficient).
> Even though the check airman had the knowledge, he was not actively
> monitoring the Captain and not using his knowledge to provide backup.*
>
> **Training Tip, Practice/Feedback**: *Develop an exercise that
> includes a need for unique knowledge about, for example, different
> airports or routes, that someone unfamiliar with that part of the
> flight needs to know and can obtain from another crew member.*

**III. Emphasize the active role that crewmembers need to take to
maintain situation awareness (situation assessment)**

> *Crewmembers often have strong beliefs that situation awareness is a
> trait of individuals, related to personality or intelligence. Most who
> study situation awareness agree that awareness is built by multiple
> situation assessments. That is, observing the situation as it unfolds,
> comparing each observation made with other observations and with
> expectations and plans, and seeking more information. Emphasizing the
> importance of actions for [assessing]{.underline} situations,
> especially among team members, may overcome some resistance to the
> idea that team situation awareness can be improved with training.
> (Appendix B, Research Summaries I, VI)*
>
> [Example]{.underline}: *An example of crew actions that help build
> awareness in one another can be found in Report Number 167579 in the
> ASRS database*. On this flight from GTF to BZN, the PF TT 3000 hours,
> 600 hours time in type, had been recently upgraded to Captain. I am
> the company Chief Pilot acting as copilot (PNF). Cruising at FL190 we
> were cleared to 15000 feet and began our descent. While I was
> completing the descent checklist we were cleared for the approach to
> the BZN airport. The PF asked me if we were cleared for the approach.
> I advised we were and completed the checklist. I noticed then that we
> were passing through 11000 and questioned why we were so low AGL 30
> DME from our destination. He replied we were cleared for the approach
> and he had a glideslope indication. We were VIS at this time. I told
> him not to descend any lower. He stopped the descent at 10800 feet in
> VIS conditions just as the controller called and asked us our
> altitude. I told the controller 10800 feet. He told us to return to
> 12000 feet. Which we did. I then asked the controller to repeat our
> clearance. It was, "Cleared for the approach to the BZN airport, cross
> the BZN VOR at 12000 feet." This time without distractions we both
> heard, understood the clearance and complied with it. What we both had
> missed previously was, "Cross the BZN at 12000 feet." The PF knew
> Runway 12 was the active runway at BZN and since we were heading 150
> degrees, he thought we were cleared to intercept the Runway 12 ILS.
> The glideslope indication he was receiving was erroneous and he
> decided to begin his descent prior to having a LOC capture. We were in
> VIS conditions, which led to my becoming complacent and not mentally
> flying the airplane as I should have been. After landing we discussed
> all problems, misconceptions, mistakes and potential ramifications
> that occurred during this flight. We are two professional pilots that
> attend recurrent training semi-annually. This consists of ground
> school and stage II simulator training. **It is obvious to me that we
> have to work even harder to improve cockpit management, eliminate
> complacency and assign priorities while in our flying environment.**
>
> **Training Tip:** *This example shows that experienced pilots, even
> when flying with reliable crew members, need to stay active and alert
> to remain aware of the situation. As an aside, it is important to note
> positive examples of situation awareness-related actions that can be
> found in this example. Crewmembers discussed the situation and tried
> to analyze what went wrong, after landing. By discussing the
> situation, they were trying to ensure that they could avert a similar
> occurrence and by waiting until the flight was over to do their
> analysis, they were not distracted from the important flight elements.
> Additional Tip: Ask crewmembers if they have ever lost awareness. Then
> ask what led to the loss. Get them to admit that they were not working
> on maintaining awareness, whether it was handling a problem or simply
> not attending.*
>
> **Training Tip, Practice/Feedback**: *In de-briefing performance in a
> scenario, point out good instances of the crew's active involvement in
> situation awareness. The actions can be found in Guidelines IV-A-D.*

**IV. Refresh crewmembers' knowledge with information about CRM related
actions that help them develop and maintain crew awareness. These
actions, explained below, are in the categories of Leadership,
Communications, Preparation and Planning, and Adaptability.**.

> *Recall that one of the definitions of crew situation awareness, based
> on research and observation, is that it is an interaction of
> individual situation awareness and team processes. There are specific
> actions that fit into each of the process categories (e.g.,
> leadership, communications, planning and preparation and adaptability)
> that have been found to be related to the situation awareness of the
> whole crew. They are addressed in A-D below. (Appendix B, Research
> Summary I).*
>
> [Example]{.underline}: *An illustrative example was found in the ASRS
> database*. We were on an FAA route check. As we Taxied and completed
> our checklists, I requested a reduced power setting as SOP for our
> operations. The FO pulled a figure incorrectly from our performance
> manual and we used it. The FAA inspector informed us of an error at
> our destination. The error was a simple oversight by the FO. It should
> have been 98 percent instead of 89 percent. I may have been able to
> prevent this by reviewing the reduced power procedure with the FO
> before our trip.
>
> **Training Tip***: This example combines both leadership action and
> preparation. The Captain was aware that it was not just an error on
> the part of the FO, but one on his part as well . Reviewing the
> procedure before the trip would have been good preparation for both of
> them. This action would have prevented the lapse in awareness and the
> resulting error of the FO.*
>
> **Training Tip, Practice and Feedback***: After providing the
> information, follow up the information with demonstrations of the
> actions and include opportunities in simulations for practice and
> feedback. Place a crew in the position of planning a flight. Give them
> a problem with equipment. Observe the Captain's brief and critique
> his/her preparation of the crew. Also note how well the brief is
> opened up for the comments and inputs of others.*

A. Provide specific examples of leadership actions that help lead to
improved transfer of information in the cockpit.

> *Leadership actions that support and encourage the exchange of
> information for awareness within the crew have been identified by
> experiments and by crewmembers themselves. (See Appendix B, Research
> Summaries I and II ).* *Leadership actions include those in which the
> Captain takes actions that help the crewmembers become (and remain) a
> functioning team. These include making crew introductions and
> accepting input and suggestions from team members. They also include
> assigning tasks and re-assigning tasks to ensure no crewmember is
> overloaded. Research has shown that crew members who are not flying,
> even junior first officers, are able to monitor the actions of the
> pilot flying, but they may hesitate to let that person know of a
> discrepancy, particularly if the person flying is the captain. One of
> the captain's important functions is to let other crewmembers know
> that their input is wanted (Appendix B, Research Summary I and Summary
> II), and this is done by telling them and by showing them (by
> accepting comments).*
>
> [Example]{.underline}: *An example of leadership actions effect on
> situation awareness is found in the ASRS data base (Number 237661).*
> Aircraft was enroute to Memphis flying off of the GRW 352 degree
> radial. We were given a crossing restriction of 6000 feet MSL at WALET
> Intersection, approximately 30 NM from the fix. At this time, the
> Captain initiated a 500 FPM descent. We were at 15000 feet. As we
> continued with the descent, appropriate checklists were run and
> completed. The descent continued and it became apparent that the
> current descent rate would not be sufficient to make the assigned
> restriction. **Although I was concerned, I made no mention of it to
> the Captain for reasons I will explain later.** After being handed off
> to Memphis approach, we were asked if were aware of the assigned
> restriction and we answered in the affirmative. We were then given a
> 10 degree turn to the left and told to intercept the 225 degree radial
> from Memphis. As we continued, it was obvious to me that we would
> cross the intersection well above our altitude restriction. As I was
> about to make this known to the Captain, Approach asked if we would
> make our restriction. The Captain told me that we would so I relayed
> the message to Approach. At this point, the Captain increased our
> descent rate to 3100 FPM. We were at approximately 11000 feet and 10
> miles from the fix. It was still clear that we would probably not make
> this restriction and with much irritation, Approach cleared us direct
> Memphis. We eventually crossed near WALET intersection at 7400 feet
> MSL. Nothing else was mentioned by ATC and flight ended normally. What
> I consider to be the factor contributing to the problem was a very
> solo oriented and belligerent Captain. This Captain did not use or
> believe in CRM. **Time and again, I tried to make suggestions and was
> told he would take care of it or to let him do it.**
>
> **Training Tip**: *In this example, the Captain lost the information
> that he could have received from his FO because he made it clear that
> he did not want to listen to him. He reduced his own awareness by
> essentially shutting out an important information source and as a
> result, the FO's awareness had no effect on the flight problem.
> Encourage crewmembers to provide other examples. Follow up with active
> practice for Captains in these actions.*
>
> **Training Tip, Practice and Feedback***: All crew members should be
> aware of the importance of leadership to situation awareness but
> emphasis on practice and feedback on these actions should be primarily
> in the specialized training for Captains.*
>
> **B. Discuss the communication actions below and how they may affect
> situation awareness. Emphasize the importance of [Verbalizing
> actions/intended actions]{.underline}.**
>
> *Communications: the kind of communications are more important for
> situation awareness than the amount of communications. These include
> letting others know what actions you are taking or are planning to
> take and giving them information that they need ("You're coming up
> on...", "One to go", "I've got a low oil pressure indication on Number
> three", "I'm going to start down now so we can make that crossing
> restriction"). By letting the other crew members know what you are
> doing and what you are planning to do, two things may be
> accomplished: 1) the other crew members are made aware of what is
> going on from your perspective; and 2) if your actions and intended
> actions are indicators that you are not "up" on the situation, it
> gives the other crew members an opportunity to help forestall an
> ill-advised action. Good communications for situation awareness
> include confessing to a loss of awareness*.*(Appendix B, Research
> Summaries I and IV)*
>
> [Example]{.underline}: *The first example given here combines both
> good and poor communication (ASRS Report Number 260266).* PF was the
> FO receiving initial operating experience. After routine river visual
> and landing to Runway 18 at DCA, the Captain (Line Check Airman) took
> control of the aircraft at approximately 80 KIAS. As he assumed
> control, DCA Tower instructed us to turn off at the next R turnoff or
> roll to the end. At the time we received the instruction we were
> passing F Taxiway and interpreted the next right turn as Runway 21.
> The FO read back the clearance as the Captain began a turn onto Runway
> 21. The Tower Controller recognized that we were turning onto 21 and
> immediately sent a landing aircraft Y around on Runway 21. I believe
> the controller misjudged the pace of events and should have instructed
> us to simply roll to the end. The entire event took place very quickly
> (a few seconds) and both crewmembers interpreted the Tower's
> instructions the same way. **Fortunately, the FO's readback included
> specifically what we intended to do and, fortunately, the Tower
> Controller recognized immediately that we had not interpreted his
> instruction correctly and initiated the GAR.**
>
> **Training Tip**: *The information received in the cockpit from the
> Tower Controller was confusing, given the position that the aircraft
> had achieved when the instruction was received. However, the complete
> communication to Tower from the FO was sufficient to alert the
> controller to a potential problem and quick action on his part averted
> that problem. Give a positive example of the impact of communication
> on situation awareness from experience and encourage examples from the
> class. Follow up with active practice.*
>
> **Training Tip, Practice and Feedback***: In the training scenario,
> attend to the communication actions (good and poor) that have been
> discussed so they may be used in the debrief.*
>
> *An example of a failure to communicate pertinent information for
> awareness that was reported in the NASA Technical Memorandum X-3596
> follows:*
>
> [Example]{.underline}: The aircraft was operating as a non-stop from
> Los Angeles to Atlanta. The trip left Los Angeles at 2200 PST and
> arrived Atlanta at 0433 EST. The weather from Los Angeles to Atlanta
> was cloudless with unrestricted visibilities for the entire distance.
> Cruise to Atlanta was at 37,000 ft. Approaching Atlanta, the center
> gave us a clearance to the ATL VOR to cross Dallas International at
> 12,000 ft. The descent had been started on a previous clearance. A
> couple of minutes later the clearance was modified to "direct REX VOR
> to cross the 30 DME arc of ATL VOR at 12,000 ft. maintain 12,000 ft.
> The descent seemed to be going well - balanced against time, distance,
> and rate of descent - and the ground speed indicated a very strong
> tailwind. The center called at what we thought was near the 30 DME of
> ATL and wanted to know what our distance was to the ATL VOR and our
> altitude. It was at this point that it was discovered that both or our
> VOR receivers were tuned to the REX VOR and that neither pilot was
> covering the ATL VOR to get the 30 mi. fix. As a consequence we wound
> up over the 30 mile fix of ATL 300 ft. higher than cleared. We
> admitted this to the center at which time they cleared us to ATL
> approach control and they worked us in routinely. The probable cause
> of this occurrence was lack of coordination on the part of the pilots
> in setting up proper VOR frequencies in order to obtain the right fix
> and **not advising the other which frequency he was on**. Because of
> this we wound up over the 30 mile fix of REX which is about 10 miles
> inside the 30 mi. fix off ATL.
>
> **Training Tip***: In this case, the problem is clear, crew members
> were not keeping one another informed of what they were doing.*
>
> **Training Tip, Practice and Feedback***: Be sure to look for good and
> poor examples of communications between the crewmembers in the
> scenario based training and debrief what was observed*.
>
> *The following is an example of the importance of verbalizing
> actions:*
>
> [Example]{.underline}: On the ground at PDX the F.O. called for ATC
> clearance to SEA; clearance was "as filed - maintain 9000 - expect
> flight level 200 3 min. after departure"; departure frequency and
> transponder code was also given. I missed the first portion of the
> clearance and picked it up as FL200 (the center stored ALT) and got
> the frequency and transponder code. I set 20,000 in the altitude alert
> unit and set the transponder code as the F.O. read back the clearance.
> Again, I did not hear the 9000 restriction and asked if we were
> cleared as filed - the F.O. answered yes. I thought I understood the
> entire clearance. The F.O. made the take-off and I changed to
> departure control and reported leaving 1000 ft. and climbing to FL200.
> The controller said 'roger' and gave us additional climb instructions
> which included a heading change at 2500 ft. At about 8000 ft. the F.O.
> asked if we had been cleared to 20,000 and I replied, yes - at 10,000
> ft. the controller asked what altitude we had been cleared to and
> again I replied 20,000. He said we should have been stopped at 9000
> then cleared us to FL200 and asked us to expedite through 11,000 which
> we did. The F.O. later said we had been cleared to 9000 originally,
> but thought we had been recleared to FL200 and he had missed the
> reclearance. The crew composition helped create this situation as the
> F.O. regularly flew this trip, the captain was a management pilot who
> hadn't flown the route recently and the S.O. was a reserve who was
> totally unfamiliar with the route. The F.O. on taxi-in at PDX,
> **unknown to the captain, had set 9000 ft. in altitude alert system in
> anticipation of what he knew was a normal altitude restriction for
> departure, when the captain thought he was given FL200 with the
> clearance he set in that altitude replacing the 9000 ft**. This was
> missed also by the F.O. as he read back the clearance and was not
> rechecked for a proper setting prior to take-off. The captain checked
> in with departure control after takeoff and stated he was climbing to
> FL200 and received no correction. When the F.O. asked if we had been
> cleared to 20,000 ft. and got a positive answer he assumed he had
> missed something and continued to climb through 9000 ft. *(NASA
> Technical Memorandum X-3546)*
>
> **Training Tip:** *Discuss where communications broke down and where
> they could have been "fixed". For example, if the FO had told the
> Captain that he was setting 9000 in preparation for their departure,
> the Captain would have been alerted to the restriction.*
>
> **Training Tip, Practice and Feedback**: *When the crew is interacting
> in a scenario, make note of communications about activities (or the
> lack of communication) and debrief crew members on the specific
> documented instances, both good and poor.*
>
> **C. Emphasize preparation and planning to ensure crew situation
> awareness**.
>
> *Preparation and Planning: This refers to personal preparation
> (knowing you are "up" for the flight), familiarization with taxi
> routes,, knowing something about the experience of other crewmembers
> (Have they flown recently? Have they flown this route before?) and
> even refreshing knowledge on the location of various items in the
> cockpit (particularly when equipment is not standardized). Planning
> the route and requirements is more specific to the particular flight,
> and has to do with the departure plan, the level of automation to be
> used, frequencies, and who will handle emergencies. (Appendix B,
> Research Summary I)*
>
> [Example]{.underline}: *An example from the ASRS data base (259694)
> follows*: Cleared for takeoff on Runway 13 at LGA. Final checklist
> items completed. Orally reviewed initial departure procedures. Captain
> applied thrust for takeoff. We heard a warning horn and observed a
> configuration message on EICAS. Captain pulled throttles to idle as we
> both simultaneously discovered the flaps still retracted. Captain
> called for flaps extended. **I hesitated, thinking we were going to
> pull off the runway**. Captain again said to extend flaps, that we
> would be ok. I complied, extending flaps to 5 degrees (our planned
> takeoff setting) and we resumed the takeoff roll. Liftoff and climb
> were normal except on climbout I realized to my horror that my initial
> error of letting myself become distracted and missing the flap
> extension after engine start had been compounded by our missing
> completely the initial items on the 'Before takeoff' checklist. This,
> of course, would have been our 'backup' safety check. We reviewed and
> discussed the events leading up to and our handling of the incident
> throughout much of the remainder of the flight. I have replayed in my
> mind over and over and over everything I did from the time I got up
> that morning all the way through the flight. Picking it up at the
> pushback: we had been delayed at the gate due to some other traffic
> out and inbound. We were pushed back onto the inner taxiway with
> clearance to start engines delayed until brakes were set. We received
> a salute and release from the mechanic as I was starting the left
> engine. Engine starts were normal and the Captain called for taxi
> clearance. Since we had been blocking the inner, I felt a bit rushed
> and called ground while starting my after start flow. That flow was
> then interrupted as we received taxi instructions. The Captain began
> to taxi, I checked for wing clearance, checked the taxi chart to
> understand the route. The Captain then said he was checking the
> rudders at which time I then checked aileron and elevator movement so
> as not to forget. I then noticed a flashing message indication on the
> ACARS. It was our updated weight manifest so while dividing time
> between inside and outside I checked passenger count, installed the
> new ZFW in the FMC and reset trim, checked the new weight against the
> card and advised the Captain of the new trim and V speeds. We were
> approaching our hold short point for crossing Runway 4. I had
> completely forgotten the rest of my after start flow, which of course
> included extending the flaps. As we were holding short of Runway 4, I
> thought of the 'Before takeoff' checklist but rejected the idea as I
> was afraid, ironically, that we would be interrupted by a clearance to
> cross the runway and thought it would be safer to wait until after we
> had crossed. After crossing we were more involved in watching the taxi
> route and with our attention diverted completely outside, neither of
> us remembered the checklist. Somehow in the later stages of taxi and
> waiting for the takeoff clearance**, we had failed to provide a cue to
> remind us about the checklist**. Curiously, as I mentioned before,
> when takeoff clearance was received, we were both cued to do the final
> items.
>
> **Training Tip**: *In this example, preparation for take-off was
> incomplete. The important task of the checklist was interrupted and
> never completed. Ask class members for cues that they use to remind
> them to return to an important, unfinished task. Point out a small
> problem in the example with planning. That was, when the Captain asked
> for extension of flaps, the FO hesitated because he did not know what
> the Captain was planning (to pull off runway or continue).*

D. Give examples of adaptability of crew members and encourage
crewmembers to discuss how that affects crew situation awareness. As
with the other CRM skill areas, ensure that adaptability actions are
observed in active practice and that they are debriefed.

> *Adaptability/Flexibility: Crewmembers have to be open to information
> that suggests their plans need to be changed. They need to be able to
> recognize when a plan is not working or not likely to work. This
> includes recognizing that they need to alter their own tasks to assist
> a crewmember who may be having a problem*. *(Appendix B, Research
> Summary I; Appendix C, Research Summary ).*
>
> [Example]{.underline}: *An example of the effect of a failure to adapt
> is shown in ASRS 266910*. My flight was perfect until I was cleared
> for a visual approach to Runway 24R at LAX. Prior to the turn I was
> approximately 7000 feet MSL and 250 KIAS. As I turned S. on base I
> could see the airport out the FO's window and then became preoccupied
> with getting the aircraft down and configured for the landing. I
> decided to 'square off' the turn to final to allow myself the time to
> get down. Still worried about getting down, I heard the tower
> controller call advisory traffic on approach to the S complex at LAX.
> The FO advised me to lower the gear. I agreed and asked for gear down.
> Later as we talked about it, that action was the first indication that
> we were thinking different things. He wanted me to slow down and start
> the turn. I was still worried about getting down. I commanded the
> autopilot to turn to a heading of 250 degrees and initiated VOR/LOC to
> intercept the localizer. The FO advised me to increase the turn rate.
> I selected 25 degrees of bank and went back to worrying about my
> altitude. The TCASII went off and I looked up and saw a DC-10 on final
> for the S complex. I was above and behind him and I still felt
> 'comfortable'. The female tower controller raised her voice level
> 20-30 decibels and continued to call traffic. The TCASII was going
> off. The tower was alarmed and the FO was now commanding a big turn.
> He visually found the Baron that was flying an approach to Runway 25R
> and told the tower that we would maintain visual separation. **I
> finally woke up**, disconnected the autopilot and rolled to 40 degrees
> of bank and quickly turned toward the N. complex. The tower controller
> stopped yelling. The TCASII stopped squawking and the rest of the
> approach went fine. (Overshooting helped me get down). What
> happened: 1) My comfort zone was too big. Flying a visual approach on
> a clear day is normally an easy, relaxed way to terminate a flight. A
> visual to LAX should not be considered that way with multiple runways
> and aircraft**. I became too preoccupied with the descent** and not
> enough with the Nav to the final. I relied on the autopilot to turn
> the aircraft to final rather than proceed manually, which would have
> been quicker and more positive. The 'square off' maneuver invited an
> overshoot and was ill advised. The TCASII, while alerting me
> initially, became a distraction after I determined that the DC-10 was
> not a threat, I did not immediately know of the smaller aircraft. The
> aural warning distracted me from hearing/understanding the FO. The FO
> was alarmed earlier. Then I thought he had communicated effectively
> with his advice to lower the gear. As we passed the N complex he saw
> everything clearly and later chastised himself for not taking positive
> action at that time. The difference in time when he could have acted
> and I finally did wake up was approximately 15-30 seconds, but it
> would have made a big difference.
>
> **Training Tip**: *This crewmember became involved in following a
> landing plan that he had made. He failed to notice much of what was
> occurring around him. Having committed to one plan, he had difficulty
> recognizing that the plan needed to be changed, he was not open for
> change.*
>
> **Training Tip, Practice and Feedback***: Adding an unexpected element
> to a scenario gives the instructor an opportunity to observe a crew's
> adaptability. For example, requiring them to do a missed approach at
> the last minute, the instructor can observe how they transition into
> climbing, cleaning up the aircraft, getting instructions, etc.*

**V. Expand on the CRM related actions listed above with the following
information.**

> *Additional information on planning, preparation, and adaptability is
> given here*
>
> *particularly for the training course that is designed for crew
> members with low*
>
> *experience levels, such as new FO's in a regional airline.*

A. Distinguish between the actions involved in planning and preparation.
Emphasize the importance of both for building and maintaining crew
situation awareness.

> *Planning includes specific actions and occurrences expected for the
> flight, including direction of take-off, initial climb altitude, use
> of engine anti-ice and other considerations, and who will do specified
> tasks. Preparation includes becoming familiar with airport layout,
> relevant geography enroute, knowing what kind of service the flight
> attendants will be providing. Preparation helps in planning, and
> planning in turn increases preparation.*

-   *both develop expectations for what will occur on the flight and
    makes crewmembers more alert to those occurrences. (We often do not
    perceive something that is unexpected).*

-   *both free crew members to be more alert to other aspects of the
    situation if they are already prepared for the routine occurrences*

> [Example]{.underline}: A crew, preparing for a flight, discovers that
> they have a particular piece of equipment that is inoperative. The MEL
> allows them to fly without it, but it is going to mean some
> alterations in procedures. The crewmembers bring up what those will
> be, and if it requires coordination, who will handle them.
>
> **Training Tip**: *By planning ahead of time, they will not need to
> take the time in flight to review what needs to be done. As a result,
> their attention can remain on the on-going situation in the flight*
>
> *Preparation includes such mundane things as knowing where items are
> located in your particular cockpit.*
>
> [Example]{.underline}: *A failure to be prepared leading to a failure
> to monitor is found in ASRS Report 295140.* The Captain was descending
> from cruise altitude to level at FL240. I, the FO, was off COM 1 to
> get ATIS and call company with ETA. The Captain has a hearing
> deficiency and because of this he requested that I use a headset to
> listen to COM 2. **I rarely use a headset and had my head down looking
> for the headset and jack**. While I was accomplishing this task the
> Captain descended to FL235. I looked up and said, "Altitude." The
> Captain climbed to FL240.
>
> **Training Tip**: *It should be pointed out that the FO, although he
> was not ensuring that the Captain was aware (while he was searching
> for the headset and jack), did become immediately aware when he looked
> up and gave that information to the Captain*.
>
> **B. Demonstrate that planning, with preparation for possible
> contingencies, should encourage adaptability during flight. Although
> developing a plan is crucial, not being ready to deviate from this
> plan may interfere with awareness**.
>
> *Adaptability is important in maintaining situation awareness. A crew
> committed to a pre-specified plan may not recognize information that
> suggests that the plan needs to be revised. (See Appendix B, Research
> Summary I and Appendix C, Summary ). From the NASA aviation Safety
> Reporting System Report No. 14: "Human error is clearly evident in the
> Altitude Deviations. Over half involve pilot perception as the
> enabling factor..."*
>
> *"The data show that the mind-set phenomenon is the dominant
> predisposing condition for the human errors leading to ADs." (p.11).
> "During an analysis of altitude deviation occurrences, the phenomenon
> of mind set emerged, wherein the pilot apparently selected an altitude
> from a preconditioning or subsequent intervening event other than that
> which was assigned..."*
>
> [Example]{.underline}: *An ASRS Report (267472) illustrates this
> situation*. During cockpit setup for flight SFO-DFW I set up the FMS
> for the filed departure which was the SFO6. When clearance was
> received from ATC we were assigned the CUIT1 instead. This was written
> down but we still missed the fact that it was not the SFO6 that was
> already set up. We briefed the SFO6 and took off with that in the FMS
> and started to fly the departure by initiating a turn to 030 degrees
> heading at 400 feet. Tower asked us what SID we were flying and we
> said the SFO6. They said we were supposed to be on the CUIT1 and to
> fly the CUIT1. I started to page through SID plates looking for the
> CUIT1 but realized it would take too long to find and read it and I
> should be looking outside at this point, so I requested a heading and
> was assigned 350 degrees and told to contact Departure. Departure
> assigned us another couple of turns to clear traffic before vectoring
> us on course. After departing the terminal area, I checked the
> clearance we copied and verified that we had indeed been assigned the
> CUIT1. I see 2 lessons from this: 1) compare the written clearance
> with the flight plan that is loaded in the FMS; 2) in the event an
> aircraft deviates from the assigned departure, ATC should give a
> heading and or an altitude instead of a named departure. Take-off is
> not a good time for the crew to be paging through SID plates and it
> will take valuable time to locate a plate and set up Nav for it. This
> time could cause a loss of separation. Supplemental Information: The
> SFO6 departure was the one we expected and filed and the Captain
> indicated it was a preferred departure for our flight. This was the
> final leg of a long day and this sequence had always ended in SFO, but
> this additional leg had just been added. After waiting 2 hours for the
> freight loading to be complete, we finished our checklists and
> received our clearance. I copied the CUIT1 SID along with the rest of
> our clearance and read it back, the Captain also copied the
> clearance**. The SFO6 SID was never replaced in the FMS, or apparently
> in our minds either**.
>
> **Training Tip**: *The crewmembers indicated that the SFO6 SID was
> both expected and preferred and it was made a part of their plan. Both
> had a difficult time recognizing that this was not the SID they were
> given and both adhered to their original plan. But, it is important to
> note that the Captain did show flexibility when adjusting to the new
> SID when he realized that it was not efficient or safe for him to find
> the necessary plate and asked ATC for vectors. Thus, he helped ensure
> that he would be able to maintain the necessary awareness*.
>
> **Training Tip, Practice and Feedback**: *Flexibility is enhanced by
> crew members running through, "What if's" during the pre-flight
> period*. *Encourage crew members, in their planning period for their
> training scenario to prepare themselves with some possible
> contingencies and to continue to do so whenever possible during
> flight.*

VI\. Introduce the specific actions that can be observed and that
indicate the level of a crew's situation awareness. Follow up with
practice and feedback.

> *Situation awareness has been identified as a state of the individual
> or crew. As a state, it is hard to measure. Fortunately, when a crew
> is interacting, there are some actions that indicate the level of the
> crew members awareness. Researchers have identified three general
> actions that can be observed in crewmembers that are related to
> situation awareness. These*
>
> *Continued on next page....*
>
> *actions are reflected in the communications of the crewmembers and in
> the way they react to*
>
> *the situation in the cockpit. (Appendix B, Research Summaries I, II,
> and III). Knowing these actions helps crewmembers to assess their own
> levels of awareness and the awareness of the other crewmembers. It
> also provides information on what can be done to improve awareness. By
> using a videotape of a crew that displays these actions, crewmembers
> can begin to see how those actions help keep one another alert to the
> situation.(See the specific actions listed below for guidance, A-C).
> When the crewmembers participate in their LOS training, point out to
> them instances of these behaviors.*

A. Discuss and demonstrate the importance of identifying
problems/potential problems to one another to help raise the situation
awareness of the entire crew.

> *As one researcher observed, crew members gain information from a
> variety of sources and they represent multiple eyes, ears, heads, and
> hands. This can lead to careful monitoring, backing up, reduced
> workload, and increased perspectives.*
>
> [Example]{.underline}: In a NASA-Ames research study, it was found
> that only those crews that recognized the potential implications of a
> report on turbulence at their destination sought additional
> information about cross winds and recognized early the high
> possibility of re-routing.
>
> **Training Tip, Practice and Feedback**: *Include a problem in the
> training scenario that is subtle in its initial cues. Note when
> crewmembers recognize these cues and how they respond to them (ignore
> them, wait for additional confirmation of the problem, seek more
> information). Debrief the crewmembers on their awareness as it related
> to the potential problem.*
>
> **B. Discuss and demonstrate the importance of showing alertness to
> the present flight status ("We've got 30 miles before we need to start
> down", "With this temperature, we'll need to put on the wing anti-ice
> after take-off").**

Crewmembers that do not continually monitor and anticipate, even during
the low workload times, are vulnerable to losing awareness or to
increasing their workload when they are required to accomplish more
tasks.

> [Example:]{.underline} *(ASRS Report 172596).* Incident occurred
> toward the end of a 6 hour 12 minute flight (Block to Block) going
> from Boston to LAX. We departed Boston following an 18 hour layover.
> As we approached the top of descent point, I felt we were already
> getting 'behind the task'. Being the PF I relied on the Captain to
> copy and post the ATIS. Instead he was involved in heavy conversation
> which was distracting. After being cleared for the profile descent, I
> didn't have the usual backup and coordination (e.g., PNF putting in
> the next altitude in the MCP window as we arrived over each successive
> fix), because by now, the PNF was scrambling to get ATIS, call the
> gate and respond to numerous ATC frequency changes which were coming a
> mile a minute. At approximately 30 NM out on the CIVET profile
> descent, we were instructed to maintain 8000'. Soon after we were
> handed off to the next controller. With workload now quite high, the
> Captain was late calling the next controller and we were now about 20
> NM out without clearance for the approach. Finally cleared for the
> approach, we were very high on the normal glide path
>
> Continued on next page....
>
> and my requests for gear and additional flaps were coming quickly
> while the PNF was busy programming the FMC with the last minute
> changed runway (from 25R to 24L). 24L had no ILS operating and with
> the haze it was difficult to get lined up on 24L centerline without
> the FMC generated 'line' to follow, so the PNF was visibly becoming
> overloaded and I felt behind. At approximately 1100 MSL, I called for
> flaps 30 degrees (final setting) and the PNF placed the right engine
> fuel control to cut off. I heard the click, saw the spar valve light
> illuminate, and placed the switch back to run (within five seconds).
> It relit immediately, the flaps were placed to 30 degrees and we
> landed normally*.*
>
> **Training Tip***: The Captain's withdrawal from the flight during
> cruise not only affected his own awareness later in the flight, but,
> because he was not supplying timely information to his FO, it affected
> the FO's awareness**.** Ask class members what could have been done to
> avert this situation.* *Point out the FO's quick action because of his
> awareness of the Captain's actions.*
>
> **C. Discuss and demonstrate the importance of showing alertness to
> the task performance of self/others.**
>
> *This is clearly demonstrated by providing information to another
> crewmember when it is needed but before the other crewmember has to
> ask for it.* *Crewmembers who notice when another is becoming
> overloaded and off-load some of his/her tasks (when possible) are not
> just demonstrating their awareness of the other person's performance
> but they are helping that person to re-gain situation awareness by
> reducing the task load. It is at the heart of back-up of other
> crewmembers. (Appendix B, Research Summaries I and II)*
>
> [Example]{.underline}: *An example of not being aware of the
> performance of self/others, from the ASRS database was submitted by a
> pilot*. During captain's P/A talk to passengers we received a change
> of headings and change of altitude from 9000 ft. to 14,000? 16,000?
> Captain started climb and changed power while talking. When I finished
> talking F.O. said left to 160. I turned aircraft and kept climbing -
> **16,000 was in altitude reminder window**. Leveled off at 16,000 ft.
> Called departure and requested higher, was advised we should be on
> center control - called center - he asked if this was our first call.
> Said yes. He asked for altitude, said 16,000. He said we should be at
> 14,000 but climb to FL230. We did**! We don't know who put 16,000 ft.
> in window of altitude reminder**. We could not recall any conversation
> about changing to center. Number 1 radio still on departure control.
> *(NASA Technical Memorandum X-3546).*
>
> **Training Tip, Practice and Feedback**: *In a training scenario, look
> for signs that crew members are aware of the performance of one
> another and point these out during the scenario debrief*

**VII.Present the information that crew situation awareness must be
maintained throughout the flight. Situation awareness in the routine
phases of flight is related to handling decisions that may be required
as the result of an emergency later in the flight.**

> *Monitoring the situation and checking to see if it is matching
> expectations leads a crew to detect an abnormal situation more rapidly
> than if they have not been actively monitoring and comparing. Having a
> high level of situation awareness when an abnormal occurs also allows
> the crewmembers more options in making decisions because they know
> what those options are. (See B, Research Summary V).*
>
> [Example]{.underline}: ASRS Report 154320. This sequence of trips had
> me paired with a very experienced, very capable, and very
> conscientious FAA designated check airman. Owing to his high diligence
> and capability displayed throughout the course of the trip series, I
> felt very secure flying with this individual. The final leg of the
> series ended with the CIVIT two profile descent to a landing on Runway
> 25L at LAX. Upon being cleared for the profile descent by a LA
> controller, the Captain/instructor directed me to dial in the Los
> Angeles ILS frequency used in this descent which he had tuned in his
> radio. I looked over at what he had tuned in his receiver and tuned
> the same frequency in mine, 109.7, without consulting my chart. I
> noticed we were high on the glide slope, with the localizer pegged to
> the R. I pulled the speed brakes, increasing speed and rate of descent
> in order to try to cross CIVIT as close to 14000' as possible. At the
> same time, I began correcting to the R in order to intercept the
> localizer. I crossed what I assumed to be CIVIT high at FL180 and
> continued down in order to cross ARNES at as close to 10000/250 knots
> as possible. I believe I was very close to 10000', still no movement
> on the localizer, when an urgent call from Controller directed me to
> "Climb immediately to 14000 feet". Without hesitation I did this. It
> was shortly after this that we discovered that the ILS frequency which
> we had tuned was for Ontario, Runway 26L, about 40 miles closer to us
> than LA. It is hard to point to determine exactly where we were or how
> close to the mountains we were. I consider this to be a major altitude
> bust. My assessment, for myself, of this incident is as follows: I am
> a professional, and this is a basic violation of procedure. I continue
> to learn not to trust/assume that anyone's performance will be perfect
> and it seems always that I relearn this lesson when I am flying with
> those in our profession who seem the most competent. A simple glance
> at the chart would have told me that 109.9 was the correct frequency,
> But I assumed that 109.7 was correct because it came from this
> individual who had performed so flawlessly up to this point.
>
> **Training Tip:** *This example illustrates how a crew member failed
> to ensure that an important piece of information was correct and how
> that led to a serious confusion about the plane's location at a later
> point. Perhaps a more familiar example to most is crew members who
> provide reminders during the flight (e.g., "We'll be at our next
> reporting point in about 5 minutes", "We've got another 8 minutes
> before we'll need to make a contamination check to see if we need to
> be de-iced again". If something happens in the interim, the whole crew
> is aware of where they are and what they need to do to keep up with
> important on-going requirements. Ask crew members to provide an
> example when something like this did happen to them. Have the
> instructor prepared with an example.*

**VII. Introduce crewmembers to the idea that the main purpose of
building a team in the cockpit is to reduce the consequences of human
error. The team must be structured with clear leadership and roles so
that it will not deteriorate into a group when under pressure, but not
be so structured that it cannot be flexible**.

> *Aviation accident reports dramatically demonstrate that putting two
> or three people together in a cockpit does not make them into a team.
> Research in other industries has shown that when an abnormal situation
> occurs, often the team, designed to catch and reduce human error,
> functions less like a team with interdependent tasks and a common
> overall goal and more like a group of collected individuals. However,
> care must be exercised to ensure that the team is not too tightly
> structured. A team that has a rigid structure may not be able to
> respond to an abnormal situation with new tasking and changed
> interchanges. (See References, Helmreich and Foushee; and Moray ).*
>
> [Example]{.underline}: *An example of lowered performance due to a
> crew member not acting as part of the team can be found in the ASRS
> data base (Report Number 274399).* I was more experienced on the 767
> than either the Captain or FO. We arrived at 265000 pounds heavy, due
> to over-fueling back at Gatwick. Maximum landing (normal) is 27800
> pounds. Captain was a 'strong/authoritative type. I concluded he was
> competent, that we were in 'good hands' but jet lag can muddle any
> flight's performance. ATC cleared us Runway 32, the shortest option.
> If I had been in front, I would've suggested we request Runway
> 28L/R..better choices at our weight. If I had been flying I would have
> insisted on one of the Runway 28's for safety sake. But I said nothing
> and the Captain accepted Runway 32 without comment. Slight mishandling
> by ATC put us close and high, demanding aggressive pilotage by the
> Captain. The Captain was behind the aircraft and slow to configure. By
> the time we passed 1000 feet, we were anything but stabilized: Idle, 2
> dots high, not fully deployed flaps, plus 35 Kts. Plus, recall we were
> heavy and heavy jets have lots of inertia...slow to slow down. Had I
> been flying I would've given up the approach and gone around. But
> there I sat back in the IRO seat, overpaid and under-employed, not
> saying "Boo". I remember thinking, "Boy, this is going to be
> interesting..." And it was..frighteningly so. (Also, I fell back on
> the old, "He must know something I don't...He didn't!). We zoomed past
> the threshold at 160 Kts., the Captain couldn't seem to get the wheels
> on the ground. We passed 3000 feet still 6 feet in the air, floating,
> still at 150-145 Kts. Finally, in desperation, the Captain raise the
> flaps to 20 degrees (actually a good piece of airmanship), we sank
> down to a touchdown at 3500-3000 feet remaining at 145 knots. The FO's
> mantra: "Binders! Binders! Binders!" Max braking...or close to
> it...brought us to an amazingly quick stop. Thank you, Boeing. Thank
> you, Maintenance. Thank you, Lord. We lost 100 Kts. in about 1500 feet
> and mercifully made the turnoff. If you've ever seen the N end of
> Runway 32 at Pit, well, it makes LaGuardia look like a nice option.
> Anyhow, while initially tempted to blame the Captain, I quickly
> realized I shared the blame. No CRM from me on this approach. The
> Captain needed my input (GAR for example) and I was silent.
>
> **Training Tip:** *This is a good example of a crewmember who acted as
> an observer even when his input could have been useful. His actions
> were not those of a team member, rather he acted as someone who just
> happened to be in this cockpit with the flight crew.*
>
> **Training Tip, Practice and Feedback***: In a training scenario, the
> crew is aware they are being observed, so it is unlikely that a
> crewmember will not participate. Still, the instructor/observer should
> be alert to look for such an occurrence and in the debrief, discuss
> what was observed and how that could be corrected.*

**IX. Emphasize that good team situation awareness helps reduce human
error caused accidents.**

> *Merely having more than one person in the cockpit does not ensure
> that errors and accidents will be reduced. Individuals need to be
> encouraged and taught to work as a team and to carry out team
> processes that ensure team situation awareness. (Appendix B, Summary
> VI)*
>
> *An example of two crew members in the cockpit not improving overall
> situation awareness and performance is found in the ASRS data base
> (Report Number 296666)*
>
> [Example]{.underline}: Hand flying a B757 on Runway 25L ILS LAX (Gear
> down, flaps 30). At approximately 1100 feet AGL in IMC, noticed a left
> yaw and roll, dramatic pitch up and airspeed increase. Initiated a two
> engine GAR. Because of our low gross weight, the GAR pitch attitude
> was approximately 22 degrees aircraft nose up. The Captain was overly
> concerned with the high pitch attitude and was distracted from
> determining that the right engine had accelerated to over 101 percent
> N1 with no throttle movement. When I retarded the throttles at the
> missed approach altitude, the aircraft again yawed and rolled left. I
> advanced both throttles to maintain control
>
> of the aircraft. I called out the right engine's condition to the
> Captain and advised we shut down the right engine. The engine was shut
> down per SOP. An emergency had been declared by the Captain during the
> early
>
> Continued on next page....
>
> part of the GAR. All appropriate checklists were completed. The cabin
> was prepared for evacuation and an uneventful single-engine approach
> and landing was accomplished. There were no injuries or damage to the
> passengers or the aircraft. I feel the Captain's unwarranted concern
> over the aircraft's pitch attitude during the GAR (he actually pushed
> forward on the yoke and fought me to reduce the pitch) (He never said,
> "I have the airplane") prevented him from accomplishing the PNF's
> duties of recognizing and calling out the problem with the engine.
> **The entire event\'s duration was prolonged because two people were
> flying the aircraft and the PNF was not monitoring the engine
> instruments per SOP.**
>
> **Training Tip**: *In this example, the Captain's momentary
> preoccupation with a problem that the PF was handling caused him to
> miss important engine indications that he should have picked up
> readily had he been monitoring as required. This was a classic case of
> team SA being reduced, a team breaking down, and the resulting error
> when both crewmembers became concerned with the same task.*
>
> **Training Tip, Practice and Feedback**: *The instructor should be
> prepared to observe crews in the training scenario for the tendency to
> ignore some of their tasks because of an unusual occurrence. This is
> important to be brought up in the debrief*.

X. Give crewmembers examples of basic problems in situation awareness
and how those problems can be connected to classes of errors.

> *Researchers have developed different classification systems that
> connect situation awareness and errors in the cockpit. Two different
> approaches are given here as examples of those connections. The first
> was described by Endsley (1995) and is a classification system built
> on her definition of situation awareness that includes three levels:
> perception, understanding and projection. At the first level,
> perception, an error may occur because the crewmember simply fails to
> perceive the information. This may come about through problems in the
> environment (the information is not available) or it may be because
> the individual does not look for information that is available to
> him/her. This latter case can be the result from such things as a
> failure to divide attention, high workload or inexperience (the
> individual does not know that the information is needed). At level two
> (understanding), the individual may perceive the information but does
> not understand its significance. A simple example would be when a
> novice pilot is given information about cross winds at his/her
> destination and does not understand that he/she cannot land there. At
> a higher level of experience, crewmembers may hear certain engine
> sounds and mis-interpret the problem. Finally, at the third level,
> projection, crewmembers may correctly interpret the problem that they
> have with an engine but be unable to visualize how that will affect
> their flight if certain conditions occur. In all of these cases,
> additional crewmembers who are well trained and are actively engaged
> in the flight should help in overcoming the problems.*
>
> *A second scheme, developed by Rouse and Rouse (1983) and presented in
> O'Hare and Roscoe's 1992 book, [Flightdeck Performance]{.underline},
> breaks the handling of a system problem into six stages: observation
> of the system; choice of hypothesis (for what could be wrong); testing
> the hypothesis; choice of a goal; choice of a procedure; and execution
> of the procedure. At each stage, they describe opportunities for
> errors. Those that are closely*
>
> *related to situation awareness are: mis-interpretation of readings;
> failure to observe; choice of hypothesis irrelevant to the situation;
> reaching the wrong conclusion; and required steps*
>
> *omitted. It should be clear that a crew that is functioning as a team
> should be able to avoid some of these errors in many situations. If
> one crewmember, through inattention, or a failure to perceive the
> information that is available, mis-interprets an instrument reading,
> another should be able to interpret it correctly.*
>
> [Example]{.underline}: *An example of an inability to perceive
> information that was available in the environment can be found in
> Report Number 167579 in the ASRS database (used with Guideline III)*.
> Cruising at FL190 we were cleared to 15000 feet and began our descent.
> While I was completing the descent checklist we were cleared for the
> approach to the BZN airport. The PF asked me if we were cleared for
> the approach. I advised we were and completed the checklist. I noticed
> then that we were passing through 11000 and questioned why we were so
> low AGL 30 DME from our destination. He replied we were cleared for
> the approach and he had a glideslope indication. We were VIS at this
> time. I told him not to descend any lower. He stopped the descent at
> 10800 feet in VIS conditions just as the controller called and asked
> us our altitude. I told the controller 10800 feet. He told us to
> return to 12000 feet. Which we did. I then asked the controller to
> repeat our clearance. It was, "Cleared for the approach to the BZN
> airport, cross the BZN VOR at 12000 feet." This time with out
> distractions we both heard, understood the clearance and complied with
> it. What we both had missed previously was, "Cross the BZN at 12000
> feet."
>
> [Example]{.underline}: *An illustrative example used in Guideline IV
> and found in the ASRS data base points out that an error can occur
> when there is a misreading of information* . We were on an FAA route
> check. As we Taxied and completed our checklists, I requested a
> reduced power setting as SOP for our operations. The FO pulled a
> figure incorrectly from our performance manual and we used it. The FAA
> inspector informed us of an error at our destination. The error was a
> simple oversight by the FO. It should have been 98 percent instead of
> 89 percent.
>
> [Example]{.underline}: *This example is an excerpt from Guideline IV C
> from the ASRS data base (259694) and illustrates how easily a required
> step can be omitted.* Cleared for takeoff on Runway 13 at LGA. Final
> checklist items completed. Orally reviewed initial departure
> procedures. Captain applied thrust for takeoff. We heard a warning
> horn and observed a configuration message on EICAS. Captain pulled
> throttles to idle as we both simultaneously discovered the flaps still
> retracted. Captain called for flaps extended. I hesitated, thinking we
> were going to pull off the runway. Captain again said to extend flaps,
> that we would be ok. I complied, extending flaps to 5 degrees (our
> planned takeoff setting) and we resumed the takeoff roll. Liftoff and
> climb were normal except on climbout I realized to my horror that my
> initial error of letting myself become distracted and missing the flap
> extension after engine start had been compounded by our missing
> completely the initial items on the 'Before takeoff' checklist.
>
> **Training Tip:** *Select one of the examples from these Guidelines,
> or an example of your own. Have crewmembers use either error
> classification scheme and apply it to the example. Lead a discussion
> on how the crew could have corrected the problem.*

XI\. Emphasize to crewmembers when pointing out a loss of situation
awareness to another crewmember, they need to consider the other person.

> *What appears to you as a loss of situation awareness on the part of
> another may actually be your own confusion. Even if the other has
> actually lost some awareness, by pointing this out with tact it
> reduces the chance of defensiveness or bad feelings and helps maintain
> the team. (Appendix B, Research Summary I)*
>
> [Example]{.underline}: *(ASRS Report 156735).* Our clearance was LGA 3
> Whitestone CLB, Runway 13,
>
> Radar vector ELLIOT, etc. The Captain and I had been discussing an
> area of stress developing between us (which has been increasing). It
> seems he has little or no respect for FO's in general and that we are
> here to do the Captain's wishes. I had pointed out the immaturity of
> this attitude and how easily threatened a 24 year Captain could be and
> my perception of the absurdity of this. Apparently no one had ever
> challenged him before and I could see truth and logic having their
> effect in a positive way. He appeared to be shaken to a degree, but
> not distracted from his responsibility as Captain. I read our
> clearance to him, not stating, "Whitestone CLB", but choosing to read
> the actual Whitestone CLB requirement. I read this as we started to
> aircraft checklist. All checklists were then completed. We pushed and
> started engines and taxied for take off. As we waited our turn, I
> again read, "175 degree turn on takeoff, left to 040 degrees at 1500
> feet". **Apparently, the Captain never realized what I was saying and
> still had his mind on our discussion of his cockpit demeanor**. As I
> climbed on 175 degree heading, reaching 1500', I began a 30 degree
> bank turn left to 040 degrees and he suddenly asked, "What are you
> doing? You must turn right to 275 degrees (Coney CLB)". He asked me to
> stop my turn until he could clarify our clearance. I assured him of
> the correctness of my actions. By that point, however, we had entered
> JFK's air space as pointed out by LGA departure ATC. We quickly
> maneuvered back on course and no problems were created. Moral: Keep it
> all business, regardless of personality clashes until in the crew room
> or parking lot.
>
> **Training Tip**: Although the Captain in this example had not been
> told that he had lost awareness, his professional behavior had been
> questioned. He became preoccupied with the criticism and therefore
> lost awareness as a result.

**XII**.**Inform crew members of the need to be alert** **to certain
situations that have been found to be particular threats to situation
awareness.**

> *Although crew situation awareness can be degraded at any time, there
> are some situations that make a crew particularly vulnerable to
> problems in maintaining their awareness. Situations identified in the
> ASRS data base as being particularly threatening to the maintenance of
> situation awareness are presented below. They include: high workload,
> breakdown in communication or coordination, occurrence of improper
> procedures, maintenance problem or equipment malfunction, unusual
> weather conditions, and fatigued crew members. (See Appendix B,
> Research Summary VII).*
>
> **Training Tip**: *Crew members adopt techniques from others and
> develop some on their own that help them maintain awareness in these
> situations. Ask them to share techniques that they use to ensure the
> crew's continual situation assessments in situations that they believe
> present a threat to situation awareness***.** *Introduce the following
> situations and provide an example of each situation's effect on the
> crew's awareness*

A. Make crew members aware that they are particularly vulnerable to
losing situation awareness in high workload situations.

> *ASRS data show that the largest number of reports about a loss of
> awareness occurred during high workload, where problems often
> distracted the crew members from their normal duties. (Appendix I,
> Research Summary VII). This can be helped by alert Captains, who,
> whenever possible, can re-assign duties to keep one crew member from*
>
> Continued on next page....
>
> *being overloaded. Other crew members, alert to one another's
> activities, can also step in and help. Crew members need to be alert
> to how automation can help or hinder in these situations. For example,
> the autopilot may handle some of the workload, whereas re-programming
> a computer during high workload may overload the crew members.*
>
> [Example]{.underline}: *An example of the effect of a high workload
> situation follows (ASRS Report 180082).* We were approaching DFW from
> the SW on the ACTON arrival. ATIS informed us to plan on a visual
> approach to 18R. When we turned N (on downwind) we were given descent
> and frequency change. The FMS and MCP were set for 18R. The new
> Controller told us to expect 17L and a short final. The time involved
> with reprogramming everything took me until base leg, while the
> copilot flew the aircraft (descending from 11000 to 3000). Upon
> turning base, the Controller asked if we had the field in sight. The
> copilot said he did and then the controller cleared us for a visual to
> 17L and switched us to Tower. At this point we began configuring the
> aircraft for landing and running the landing checklist. When we came
> upon the Tower\'' frequency, he offered us 17R and we accepted it. As
> we rolled out on final, I was busy reprogramming the radios and FMS
> for the third time and did not notice the copilot had lined up on 18R.
> Approximately one mile later I finished setting up the cockpit and saw
> on the NAV display that the aircraft was displaced to the right or W
> of course. After checking frequencies and courses very quickly, I
> looked outside and started to tell the copilot he had lined up on the
> wrong runway. At that moment the Tower called to confirm we had the
> airport in sight..we were approaching the outer marker... and we
> turned toward 17R. The remainder of the approach was uneventful.
> Contributing factors were the fading daylight and the copilot had been
> out of this cockpit for 2 months while transitioning to another
> aircraft, which left him a little behind and pre-occupied. There were
> not other aircraft on approach to either side of DFW. **Changing
> approaches and runways inside of 10 miles (even in VMC) in this high
> workload 2 person cockpit aircraft represents a challenge, even when
> both pilots are experienced and current...**
>
> **Training Tip**: *This information should be included in special
> training for Captains. It is one of the tasks of the Captain to try to
> assign and reassign tasks so that no one crew member has more than
> he/she can do. Careful planning and preparation also help the crew
> function efficiently so that workload does not increase. And, finally,
> if workload is high for all and cannot be helped, then it is up to the
> Captain to ensure that the crew's tasks are prioritized so that
> attention is placed on only those important tasks.*
>
> **Training Tip, Practice and Feedback***: Include a high workload
> section in a training scenario. Be sure that it is a realistic
> simulation. Be prepared to ask the crew members: 1) how workload
> affected their awareness and 2) what they did, or could have done to
> decrease the workload (e.g., prioritize, use automation, stop using
> automation, Captain reassigning tasks, a crew member confessing that
> he/she was over-loaded.)*
>
> **B. Emphasize to crew members that they need to maintain crew
> coordination and communication as a method of helping to maintain crew
> situation awareness.**
>
> *ASRS data show that a lack of crew coordination and/or communication
> is the second highest cause of a loss of situation awareness.
> (Appendix B, Research Summary VII)*
>
> [Example]{.underline}: *A brief example from the ASRS data base
> (Report 313640) illustrates this*. Columbus Approach assigned us 2500
> feet. While trying to locate the airport we descended to 2000 feet**.
> Contributing factors were poor crew coordination (i.e., both pilots
> looking for the airport instead of one looking and one flying).**
> Additionally, we thought we had the airport which caused an initial
> let down which was not fully recovered from when we realized that we
> did not have the actual airport. Descent to 2000 feet was realized
> when ATC advised us to maintain 2500 feet.
>
> **Training Tip**: *Class members should be able to supply instances of
> loss of situation awareness due to breakdowns in communication and
> coordination. Point out that when communication and coordination are
> gone, the team disintegrates into a group, that is, they go from
> working together for a common goal to being simply an aggregate of
> people. (See Additional Reading 1). Using two brief video segments of
> a situation where the Captain receives information from ATC that
> causes him/her to change plans could make the point. In one, the
> Captain relays the information, declares the change, and tells the
> crew what each of them will do. In the second, none of this
> information is conveyed until the First Officer asks what is going
> on.*
>
> **Training Tip, Practice and Feedback**: *A breakdown in coordination
> and communication cannot be designed into a scenario, but it is likely
> to happen spontaneously. The instructor needs to be alert to the
> possibility that this will happen and to know how to observe it.
> Showing a crew where this happened in their scenario and the result by
> playing the videotaped scenario should be part of the debrief.*
>
> **C. Inform crew members the occurrence of improper procedures are
> often followed by a problem with situation awareness.**
>
> *Analysis of ASRS data have shown a high incidence of problems with
> situation awareness following improper procedures*. *(Appendix B,
> Research Summary VII)*
>
> [Example]{.underline}: I was involved in an incident which occurred
> ...during my initial flight training as a FO with a Corporation, a
> regional airline based in ICT. I was acting as a new hire student
> under the instruction of a company flight instructor and check
> airman...The incident occurred on the third training flight. The
> incident flight began shortly after midnight and consisted of a series
> of maneuvers and approaches. On the last approach, a simulated single
> engine ILS to minimums, I was flying the aircraft while being radar
> vectored for the approach on one engine. The instructor explained that
> the gear can be extended at the marker or can remain retracted until
> breaking out and the landing is assured. He further explained that it
> was the Captain's decision and depended on a variety of circumstances.
> He said, "In this case we will leave the gear up, so at the marker the
> call will be, 'Landing check, hold gear'." I made this call at the
> marker and we continued the approach without any further mention of
> the gear. At decision height, the view limiting device was removed and
> things got a little frantic. I called, "Going Vis, full flaps," but
> the instructor said that we would just use half flaps. I then quickly
> glanced around for three green lights until my eyes caught the
> indicator and I realized that this was that strange mechanical
> indicator that I'd been flying around with all night. I saw the gear
> indicator markings and it was time to flare, so I concluded that the
> gear must be down. During the flare, it looked like I was finally
> going to have a nice gentle landing, but the wheels never touched.
> Instead, I heard a strange buzzing noise and the instructor
> immediately pulled it out, added the power and climbed out. He reached
> for the gear handle; it was up. I immediately realized the buzz was
> the props on the concrete I also remember that we had held the gear up
> and I should have called for it down sometime. We entered the pattern
> and landed. The aircraft seemed to perform and sound normal. However,
> after shutdown at the ramp, we saw that the outer 3" of the left prop
> blades were curled aft and the tips of right blades were
> scraped...Supplemental information from CAN 181978: We were executing
> a simulated single engine ILS approach and our company procedure is to
> do the landing checklist at the outer marker, but to hold the
> extension of the landing gear till the landing is assured. When the
> student saw the runway and said, "Going Vis", he also should have
> said, "Landing gear down". He did not. And I as the instructor should
> have put the landing gear down, I did not. In the flare, I realized
> the gear was not down and immediately applied full power and pulled
> up. (ASRS Summary 182635)
>
> **Training Tip**: *In this case, the instructor was familiar with the
> company procedure on extending the landing gear, but he was not alert
> to the PF's failure to accomplish this procedure. As a result, neither
> crew member was aware that the gear had not been extended. The crew
> came close to a gear-up landing*.
>
> **D. Demonstrate to crew members that a problem with maintenance or an
> equipment malfunction may result in a problem in situation awareness
> for the crew.**
>
> *An ASRS data analysis report showed this to be the cause of slightly
> less than one third of the reported incidents of a loss of situation
> awareness.*
>
> [Example]{.underline}: While climbing out of CID, Controller advised
> to "Climb to 15000 feet. Traffic at 16000 feet. Will have higher when
> clear." While passing through 12000 feet the Number 1 prop governor
> started to lose control (plus or minus 1.5%). This problem happened
> earlier in the day and was found to be within tolerances, so the
> Captain paid it no attention**. I, (PF) decided to try and adjust the
> condition levers (while hand flying the aircraft) and saw that I was
> passing through 13000 feet. Essentially, I had become preoccupied with
> the situation (the props) and was waiting to hear the altitude
> alerter, which of course malfunctioned,** so we naturally passed
> through our altitude by approximately 800 feet when I had noticed it.
> The alerter never signaled in either the pre- or post the feet alarm
> mode as it is designed to do. (In fact, as it turned out, it would
> only malfunction in climb inside as we after discovered). Additionally
> the Captain had gotten preoccupied with some company paperwork, so
> that he had missed his '1000 feet to go' call. Just as I had noticed
> the altitude and started to level off the controller came on and said
> that we can continue climb to 23000 feet. No mention was made of our
> altitude and no deviations of any kind were necessary. In fact, I
> don't believe the controller noticed at all. In any event, I feel the
> problem arose from my reliance on the equipment (namely the altitude
> alerter), and the Captain's attention being taken away to perform
> company business. *(ASRS Report Number 294359)*
>
> **Training Tip:** *In this example, two equipment problems contributed
> to the crew's lack of knowledge about their altitude. In the first,
> the PF was trying to adjust condition levers and diverted his
> attention from monitoring the gauges. The PNF was working on
> paperwork, so he was not monitoring. Because the altitude alerter
> failed, the crew was not given the altitude information they expected
> from the equipment. This illustrates that equipment can affect
> awareness by distracting, and by failing to give information***.**

E. Discuss the need of crew members to find out about the experience
level of others. This way, the crew members who are more experienced
will anticipate problems and be more alert to the awareness of the other
crew member. Unfamiliarity or lack of experience (with the route of
flight, the airport layout, the airplane, the company, the other crew
members) lowers the situation awareness of the crew. They should also be
encouraged to confess their own lack of experience in a situation.

> *Humans have limited attention capacity. As we become familiar with
> tasks and situations we have to spend less attention to them and can
> be alert to other things. Someone who is not familiar with the
> aircraft, the route, the equipment, or the airport layout or
> taxi-ways, for example, will have to attend closely to the unfamiliar
> and may not notice something as obvious as a call from Tower.
> (Appendix B, Summary I and Summary VII)*
>
> [Example]{.underline}: *(ASRS Report 243880)* Copilot is PF. Prior to
> descent I set up my approach charts for arrival. I look to the
> copilot's side and see that he has the ILS approach plate out and
> ready to brief the one involving a DME ARC and final approach course N
> of the field. The approach I expected to use was the VOR-A straight in
> from the S, the direction we were approaching from. The destination
> wind was out of the N at 7-10 knots, right down the runway. The
> visibility was excellent (30 miles) and the ceiling was high enough
> (6000-8000 feet) allowing for a nice efficient straight in descent
> over the VOR. However, descent must begin 50 NM sooner to allow us to
> descend into VMC, acquire the airport and runway visually and cancel
> IFR before the VOR passage. I go ahead and request lower altitude and
> controller asks which approach we would like, the ILS 11 or the VOR-A.
> I now realize the copilot is not aware of the VOR-A approach and is
> asking me why we wouldn't want to approach from the N on the ILS
> Circle to land N. I explain the inefficiency of his plan based on the
> wind and ceiling. Now I have allowed a dangerous situation to develop
> During cruise, I did not take the time to confirm that the copilot was
> aware of the ceiling, wind and visibility pointing to a straight in
> arrival from the S and that the copilot was aware of the VOR-A
> approach as the inst approach procedure used to let down from en-route
> to VMC and subsequent cancellation of IFR. At top of descent I realize
> that the copilot does not have the approach plate for the VOR-A handy,
> is not familiar with it and is unable to brief it to the crew prior to
> descent. At this point, I could've taken the airplane as PF and
> initiated descent while he caught up with me **(If my first mistake
> was not ensuring that everyone was familiar with the arrival that I
> was expecting so that there was plenty of time to discuss it and
> understand it, my second mistake was not initiating descent myself and
> assuming the role of PF to prevent a hurried high speed descent).**
> Instead, the copilot was frustrated and confused by his hurried
> descent. As we approached the VOR at 8000 feet in IMC, I told him we
> could not proceed beyond the VOR unless we were in VMC and that we
> must now turn L in a course reversal to intercept and use the
> procedure turn to continue descent. At this point, the copilot asked
> me to "Take the airplane" and I did, momentarily over-banking to 45
> degrees (max. 30 degrees). As I turned hastily outbound while he
> coordinated our procedure turn entry with the controller within
> moments we were in VMC. Visually acquired the runway, canceled IFR and
> contacted airport advisory service. I gave the airplane back to the
> copilot who completed the approach and landing. I apologized to the
> copilot for my not confirming with him earlier about his familiarity
> with the airport and NAVAIDS and thanked him for handing over the
> airplane when he did. This was the first leg of the duty day. I had
> flown with him twice before. We had flown into this airport 11 months
> ago at night using the ILS without incident**. In future, I will
> ensure my crew members familiarity with a particular operation before
> its too late.**
>
> **Training Tip**: *The example, with the Captain's stated intention,
> could be used to initiate a class discussion, where crew members are
> encouraged to share their own experience with a similar situation,
> either where they lost awareness themselves because of a lack of
> familiarity or were flying with someone else who did.*
>
> **Training Tip for Practice/Feedback**: *In the initial meeting* *of
> the crew members for the training scenario, the instructor should
> observe and make note of any questions the crew members ask of one
> another that give them information about their knowledge ("Been to LAX
> recently?") or about the personal readiness for the flight ("I
> understand you just got back from vacation"). During de-brief, ask
> crew members if there is any information that they could have had
> about the other that would have helped in the flight.*
>
> **F. Introduce unusual weather conditions as a cause of loss of
> situation awareness.**
>
> *The ASRS data base shows that crew members often lose situation
> awareness as a result of flying in bad weather conditions. It is
> likely that the weather adds to the workload and is a distraction from
> normal activities that help crew members maintain their situation
> awareness*. *When preparing to fly in bad weather, crew members need
> to consider how*
>
> *Continued on next page....*
>
> *this will change their flight and prepare for it. Is de-icing
> required? Consider what that means to the flight. Are there
> thunderstorms in the area of the destination airport? Who will monitor
> the weather? Will that mean possible holding? Preparation aids the
> crew members by reducing their workload during flight.*
>
> Example: I departed DEA on a routine scheduled flight to YKM. We were
> cleared to 5000 feet and inadvertently went through this altitude by
> 2000 feet. My FO was flying at the time. I was trying to identify a
> radar return for Weather avoidance. I gave him the 1000 feet call
> required in our Ops. Specs. But he was also involved in the radar
> tracking of a thunderstorm we were trying to avoid. Consequently, we
> flew right through our assigned altitude. The aircraft we fly are not
> equipped with altitude alerters which would have prevented this from
> happening. Recommend that aircraft like these require alerters to
> prevent this, which will continue to occur because we are only human.
> *(ASRS Report Number 145134)*
>
> **Training Tip**: *Present a video tape segment of a crew handling a
> fairly high workload in weather that is no problem. Ask the class to
> discuss how the tasking in the cockpit may change if the same
> situation is played out in severe weather conditions.*
>
> **Training Tip, Practice and Feedback***: In the training scenario,
> include some weather considerations, such as potential for icing on
> climb out, a moving line of thunderstorms, potentially dangerous
> weather over the destination air field. Be sure to provide feedback to
> the crew members on their handling of the weather and maintenance of
> situation awareness for the whole flight.*
>
> **G. Ensure that crew members discuss how to overcome some of the
> effects of fatigue so that they may maintain awareness as a crew.**
>
> *ASRS data show that fatigue is an issue in a loss of situation
> awareness. Communicating and coordinating as a crew will help overcome
> the decrements caused by fatigue*. *When crew members are fatigued
> they need to increase their efforts in communicating and coordinating
> with one another. (Appendix B, Research Summary VII)*
>
> [Example]{.underline}: *This example comes from ASRS Report 273363*.
> An altitude of 14000 feet was issued. Captain entered 15000 feet in
> the altitude window. This is usually done by PNF, but since this
> particular aircraft with a slightly different configuration had the
> altitude window closer to the Captain and out of the FO's scan it was
> not uncommon for it to be done by the captain, even though he might be
> flying. We climbed to 15000 feet and were corrected by ATC, then given
> a final altitude of 16000 feet. Fatigue was also a big factor. It was
> the third day of a 4 day trip and our fifth leg for that particular
> day. This might have been avoided if we had more rest and the altitude
> window had more central location in the cockpit.
>
> **Training Tip**: *The example above describes two problems, one, the
> incorrect setting of the altitude is clearly the reason for the bust.
> The second, fatigue, was not objectively determined to be a cause, but
> may well have been the underlying reason for the mis-setting. The ASRS
> reports often describe a mistake on the part of the crew members and
> add that this was the fourth day, or the 11^th^ hour for the crew. The
> crew members in the example were aware that the altitude window was
> not easy for both of them to see and they were aware that they were
> likely to be fatigued*. *Discuss what they could have done in that
> situation to overcome fatigue-caused mistakes.*
>
> **Training Tip, Practice and Feedback***. Fatigue cannot be built into
> a scenario. However, some crew members arrive for the training
> scenario fatigued (e.g., they have come from another time zone, have
> not slept well). Encourage them to talk about problems the fatigue may
> have caused and what they could have done about them.*
>
> **H. Remind crew members that none of these problems happens in
> isolation and that two or more are usually combined in a situation**.
>
> *Consider the examples presented throughout these guidelines and note
> that each one could be used to illustrate more than one problem.
> Often, high workload and an unusual problem combine to compel crew
> members' attention. Fatigue causes attention to be lowered and this
> can lead to a problem if anything out of the ordinary happens.*
>
> [Example]{.underline}: *(ASRS Report Number 265310*). Last leg of long
> flight day (7:50 scheduled, 4 legs, already logged 7:00 hours domestic
> flight time), 11^th^ hour of 12 hour duty day. Night**. Crew made
> comments of being tired.** PF: FO. Captain taxiing aircraft onto
> runway. Captain familiar with airport. **FO recently upgraded (6
> months earlier) after 6 years as FE, had been to CLE less than 6
> times** as FO. Cleared to follow 2 commuter aircraft to Runway 23 by
> Ground Control. Followed aircraft, determined takeoff would be in
> short period of time**. While taxiing, started 3^rd^ engine and
> completed taxi checklist simultaneously.** Somewhat rushed. Both
> commuter aircraft departed Runway 23L (the longer way). Captain's ear
> piece fell out **and Captain was fumbling with ear piece when Tower
> cleared aircraft** to "Cross Runway 23L, cleared for takeoff Runway
> 23R". Clearance was read back correctly by FO, but Captain was still
> fumbling and only heard clearance for Runway 23R, on cabin speaker. FO
> was doing duties and heard clearance for Runway 23R, (and was not
> familiar with airport), but heard another clearance given later, but
> not its contents (it was for a landing aircraft), and **assumed our
> clearance had been changed.** Captain taxied onto Runway 23L, lined
> up, and transferred control of aircraft to FO. FO knew what runway we
> had been given clearance for, and was actively trying to ascertain we
> were on the correct runway. **He did not say anything to the Captain,
> thinking he knew which runway we were on, and what we were cleared to
> depart**. FO did not realize Captain thought we were cleared to
> takeoff on Runway 23L. Takeoff roll was started after lining up on
> Runway 23L, Tower canceled takeoff clearance after approximately 50
> feet of roll. Problem from FO standpoint: Departure ends of CLE
> runways 23L, 23R and 28 are all very close together. Cleveland is
> notorious for this and FO was aware of that. However, this my have
> been the **FO's first takeoff from Runway 23 at night**, and he
> thought that Runway 28 was Runway 23L and the aircraft was on the
> correct runway (23R). He even checked the markers thinking he was on
> the correct runway and when the Captain lined up on Runway 23L he
> assumed the Captain knew where he was (as indeed he did). This was a
> classic case from start to finish. Long day. **Tired** crew. Crew
> member **distracted**. **New** FO. **Night**. **Hurrying** checklist.
> CLE Tower contributions to situation: Clearing both commuters to
> takeoff from Runway 23L then in 1 instruction clearing us to cross
> Runway 23L, cleared for takeoff Runway 23R.
>
> **Training Tip**: *Any example of a combination of threats to
> situation awareness of the crew would be worthwhile discussing. In the
> example above, are crew members able to pick out the threats to
> awareness? Discuss methods for overcoming the problems (e.g., thorough
> planning, confessing, communicating and checking information).*

**Evaluation**

The guidelines in this section address the training of instructors to
observe crew members in order to debrief them after active practice and
feedback and the training of crew members to observe examples of
situation awareness actions in crew members and in themselves**.** The
format for the guidelines in this section is:

I-V Guideline

Expansion of the guideline

**I. Train instructors or crew members or both to observe crew member
actions. These include those actions helpful for situation awareness and
those related to errors in situation awareness**.

> *Understanding and observing important crew actions must be trained.
> Although experience with the task is an aid to observing accurately,
> observation skills can be improved through training and guidance.
> (Appendix B, Research Summary VIII)*

II\. Consider the experience level of those being trained when making up
a class for observation training

> *People vary in their ability to recognize errors committed by a crew
> when watching their performance on videotape. More experienced pilots
> recognize more errors, overall. Although less experienced pilots are
> likely to recognize more tactical errors, more experienced pilots are
> more likely to recognize the strategic errors and CRM related errors.
> (Appendix B, Research Summary VIII)*

III\. When developing a videotape for classroom demonstration or for
observation training, include both good and poor actions on the part of
the crew members.

> *There are two reasons for this: 1) Crew members are more likely to be
> able to identify poor actions on the part of videotaped crews than
> they are to identify good situation awareness actions, thus, their
> ability to recognize poor actions does not necessarily indicate their
> ability to recognize good actions; and, 2) it is the good actions that
> you want to demonstrate for them to learn. (Appendix B, Research
> Summary VIII)*

IV\. Include re-enforcement of some of the actions of interest on the
training videotape (i.e., some consequence for the action is shown) when
developing a videotape for classroom demonstration or for observation
training.

> *Crew members are more likely to recognize good and poor behaviors if
> they are followed by consequences than if they are not. (Appendix I,
> Research Summary VIII).*
>
> **Training Tip:** *For example, a scenario could have the Captain
> thanking the First Officer for information that he/she provided or
> could include a failure to clarify an altitude assigned by ATC leading
> to an altitude bust.*

V. When developing a videotape for classroom demonstration or for
observation training, develop a specific list of behaviors to look for
in the videotape.

> *Crew members at all levels of experience are more likely to recognize
> and describe targeted behaviors if they are given a specific list of
> behaviors to look for. (Appendix B, Research Summary VIII)*

**Specialized Training**

A training program that has been developed and tested for a particular
level of crew member may not be as effective for crew members with a
different experience level. For example, a regional air line may want to
consider including separate situation awareness training for their new
hire FO's and adding training in relevant situation awareness actions to
Captain up-grade courses. This training for FO's, many of whom are new
to flying as a crew member, could help them recognize the often subtle
cues for a loss of situation awareness in an experienced pilot and for
new Captains, could re-enforce important leadership actions that they
need to make to help ensure their own awareness.

Guidelines in this section have the following format:

I-VIII Guidelines, followed by an expansion of the guideline

Example

Training Tips

**I. Emphasize the special responsibility of Captains to set up their
crew as team so it will be organized to give the information necessary
for him/her to maintain situation awareness**.

> *Captains need to have awareness of all aspects of the flight. Without
> the inputs of crew members in different positions who may have
> different information available to them this would be an impossible
> task. (Appendix I, Research Summary I)*
>
> An example of a Captain's failure to assess the status of his
> information gathering system in the crew is found in the ASRS data
> base (Report 135982)
>
> [Example:]{.underline} We were being vectored and descended for
> approach and landing at ORD. The weather was clear and the visibility
> was good. There was a great deal of visible traffic arriving and
> departing from ORD and it was layered above and below us. We were
> cleared out of 8000 feet to 7000 feet. As the FE, it is a primary duty
> to maintain a traffic watch below 10000 feet. The FO was flying the
> aircraft with use of the autopilot. There was an audible click of a
> switch movement, followed by a red flashing light that drew my
> attention inside the cockpit. I heard approach control ask were we
> were going. I looked inside and observed the Captain pulling back on
> the yoke while telling the FO to stop the descent. The descent stopped
> at approximately 6740 feet. The FO stabilized the aircraft at 7000
> feet. The autopilot had dropped off and the flight director remained
> engaged. The excursion occurred due to over reliance by the FO on the
> autopilot system. This is a product of current training doctrine,
> which is heavily weighed with emphasis on autoflight of the aircraft.
> More and more of the basic stick and rudder flying is being delegated
> to autopilot systems, which in turn is setting the stage for more of
> this type of incident. The workload could be somewhat eased by ATC
> through gentler descent and vector profiles. Reduce the rapid fire
> clearance data and slam dunk descent profiles that are becoming the
> rule rather than the exception. Supplemental information from CAN
> 135982: FO was fairly new on equipment. He said nothing about being
> uncomfortable on the widebody. Discussed incident with him at gate and
> suggested he ask for help when overwhelmed (not apparent to me). FO
> stated perhaps he spent too much time outside the cockpit. He'd stated
> to me he'd flown one trip. I'd assumed 5 or 6 legs...My mistake.

II\. Make Captains aware that crew members with different levels of
experience may see both risk and time pressure differently when they are
making their individual assessments of the situation.

> *An experienced Captain, flying with a junior First Officer should
> know that the FO is likely to overestimate the time available to
> correct an abnormal situation and to overestimate the risk. The
> difference in time estimation may make the First Officer delay in
> giving information on an abnormal to the Captain. The over-estimation
> of risk may make the junior First Officer assess the situation as
> needing dramatic measures and he/she may not consider less drastic
> corrections. This makes it more imperative that the Captain encourage
> the First Officer's inputs on what he/she sees, hears, or feels rather
> than having him/her remain silent and come to his/her own conclusions.
> (Appendix B, Research Summary XI.*
>
> *Results of research in the air defense domain demonstrated that
> inexperienced individuals in this domain were less likely to use the
> available time to try to determine the possible causes of a situation
> and were more likely to select the drastic solution and to stick with
> the solution during critical review than were more experienced
> individuals. (See References, Cohen et al.)*
>
> There is a written set of aviation situations that have been presented
> to crew members of different levels of experience. These are included
> at the end of this document with their estimations of risk and time
> available. (Appendix IV). Present a subset of these in classroom
> training and allow crew members to discuss and guess differences that
> experience may make in assessing the situation. This would heighten
> their attentiveness to possible differences in the way crew members
> assess the same situation.

**III. Refresh Captains' knowledge with information about CRM related
actions that help them develop and maintain crew awareness. These
actions, explained below, are in the categories of Leadership,
Communications, Preparation and Planning, and Adaptability. After
providing the information, follow up the information with demonstrations
of the actions and include opportunities in simulations for practice and
feedback**.

> *Recall that one of the definitions of crew situation awareness, based
> on research and observation, is that it involves individual situation
> awareness and team processes in a highly interactive relationship.
> There are specific actions that fit into each of the process
> categories (e.g., leadership, communications, planning and preparation
> and adaptability) that have been found to be related to the situation
> awareness of the whole crew. They are addressed in A-D below.
> (Appendix B, Research Summary I)*
>
> [Example]{.underline}: *An illustrative example was found in the ASRS
> data base*. We were on an FAA route check. As we Taxied and completed
> our checklists, I requested a reduced power setting as SOP for our
> operations. The FO pulled a figure incorrectly from our performance
> manual and we used it. The FAA inspector informed us of an error at
> our destination. The error was a simple oversight by the FO. It should
> have been 98 percent instead of 89 percent. I may have been able to
> prevent this by reviewing the reduced power procedure with the FO
> before our trip.
>
> **Training Tip***: This example combines both leadership action and
> preparation. The Captain was aware that it was not just an error on
> the part of the FO, but one on his part as well . Reviewing the
> procedure before the trip would have been good preparation for both of
> them. This action would have prevented the lapse in awareness and the
> resulting error of the FO.*
>
> **Training Tip, Practice and Feedback***: Place a crew in the position
> of planning a flight. Give them a problem with equipment. Observe the
> Captain's brief and critique his/her preparation of the crew. Also
> note how well the brief is opened up for the comments and inputs of
> others.*

IV\. Provide specific examples of leadership actions that help lead to
improved transfer of information in the cockpit.

> *Leadership actions that support and encourage the exchange of
> information for awareness within the crew have been identified by
> experiments and by crew members themselves. (See Appendix B, Research
> Summaries 1 and 2).* *Leadership actions include those in which the
> Captain takes actions that help the crew members become (and remain) a
> functioning team. These include making crew introductions and
> accepting input and suggestions from team members. They also include
> assigning tasks and re-assigning tasks to ensure no crew member is
> overloaded. Research has shown that crew members who are not flying,
> even junior first officers, are able to monitor the actions of the
> pilot flying, but they may hesitate to let that person know of a
> discrepancy, particularly if the person flying is the captain. One of
> the captain's important functions is to let other crew members know
> that their input is wanted (Appendix B, Research Summary I) and this
> is done by telling them and by showing them (by accepting comments).*
>
> [Example]{.underline}: *An example of leadership actions effect on
> situation awareness is found in the ASRS data base (Number 237661).*
> Aircraft was enroute to Memphis flying off of the GRW 352 degree
> radial. We were given a crossing restriction of 6000 feet MSL at WALET
> Intersection, approximately 30 NM from the fix. At this time, the
> Captain initiated a 500 FPM descent. We were at 15000 feet. As we
> continued with the descent, appropriate checklists were run and
> completed. The descent continued and it became apparent that the
> current descent rate would not be sufficient to make the assigned
> restriction. **Although I was concerned, I made no mention of it to
> the Captain for reasons I will explain later.** After being handed off
> to Memphis approach, we were asked if were aware of the assigned
> restriction and we answered in the affirmative. We were then given a
> 10 degree turn to the left and told to intercept the 225 degree radial
> from Memphis. As we continued, it was obvious to me that we would
> cross the intersection well above our altitude restriction. As I was
> about to make this known to the Captain, Approach asked if we would
> make our restriction. The Captain told me that we would so I relayed
> the message to Approach. At this point, the Captain increased our
> descent rate to 3100 FPM. We were at approximately 11000 feet and 10
> miles from the fix. It was still clear that we would probably not make
> this restriction and with much irritation, Approach cleared us direct
> Memphis. We eventually crossed near WALET intersection at 7400 feet
> MSL. Nothing else was mentioned by ATC and flight ended normally. What
> I consider to be the factor contributing to the problem was a very
> solo oriented and belligerent Captain. This Captain did not use or
> believe in CRM. **Time and again, I tried to make suggestions and was
> told he would take care of it or to let him do it.**
>
> **Training Tip**: *In this example, the Captain lost the information
> that he could have received from his FO because he made it clear that
> he did not want to listen to him. He reduced his own awareness by
> essentially shutting out an important information source and as a
> result, the FO's awareness had no effect on the flight problem.
> Encourage crew members to provide other examples. Follow with active
> practice.*
>
> **Training Tip, Practice and Feedback***: All crew members should be
> aware of the importance of leadership to situation awareness but
> emphasis on practice and feedback on these actions should be given to
> Captains.*

**V. Introduce Captains to the idea that the main purpose of building a
team in the cockpit is to reduce the consequences of human error. The
team must be structured with clear leadership and roles so that it will
not deteriorate into a group when under pressure, but not be so
structured that it cannot be flexible**.

> *Aviation accident reports dramatically demonstrate that putting two
> or three people together in a cockpit does not make them into a team.
> (Appendix B, Research Summary VI). Research in other industries has
> shown that when an abnormal situation occurs, often the team, designed
> to catch and reduce human error, functions less like a team with
> interdependent tasks and a common overall goal and more like a group
> of collected individuals. However, care must be exercised to ensure
> that the team is not too tightly structured. A team that has a rigid
> structure may not be able to respond to an abnormal situation with new
> tasking and changed interchanges. (See References, Helmreich and
> Foushee; Moray )*
>
> [Example]{.underline}: *An example of lowered performance due to a
> crew member not acting as part of the team can be found in the ASRS
> data base (Report Number 274399).* I was more experienced on the 767
> than either the Captain or FO. We arrived at 265000 pounds heavy, due
> to over-fueling back at Gatwick. Maximum landing (normal) is 27800
> pounds. Captain was a 'strong/authoritative type. I concluded he was
> competent, that we were in 'good hands' but jet lag can muddle any
> flight's performance. ATC cleared us Runway 32, the shortest option.
> If I had been in front, I would've suggested we request Runway
> 28L/R..better choices at our weight. If I had been flying I would have
> insisted on one of the Runway 28's for safety sake. But I said nothing
> and the Captain accepted Runway 32 without comment. Slight mishandling
> by ATC put us close and high, demanding aggressive pilotage by the
> Captain. The Captain was behind the aircraft and slow to configure. By
> the time we passed 1000 feet, we were anything but stabilized: Idle, 2
> dots high, not fully deployed flaps, plus 35 Kts. Plus, recall we were
> heavy and heavy jets have lots of inertia...slow to slow down. Had I
> been flying I would've given up the approach and gone around. But
> there I sat back in the IRO seat, overpaid and under-employed, not
> saying, "Boo". I remember thinking, "Boy, this is going to be
> interesting..." And it was..frighteningly so. (Also, I fell back on
> the old, "He must know something I don't...He didn't!). We zoomed past
> the threshold at 160 Kts., the Captain couldn't seem to get the wheels
> on the ground. We passed 3000 feet still 6 feet in the air, floating,
> still at 150-145 Kts. Finally, in desperation, the Captain raise the
> flaps to 20 degrees (actually a good piece of airmanship), we sank
> down to a touchdown at 3500-3000 feet remaining at 145 knots. The FO's
> mantra: "Binders! Binders! Binders!" Max braking...or close to
> it...brought us to an amazingly quick stop. Thank you, Boeing. Thank
> you, Maintenance. Thank you, Lord. We lost 100 Kts. in about 1500 feet
> and mercifully made the turnoff. If you've ever seen the N end of
> Runway 32 at Pit, well, it makes LaGuardia look like a nice option.
> Anyhow, while initially tempted to blame the Captain, I quickly
> realized I shared the blame. No CRM from me on this approach. The
> Captain needed my input (GAR for example) and I was silent.
>
> **Training Tip:** *This is a good example of a crew member who acted
> as an observer even when his input could have been useful. His actions
> were not those of a team member, rather he acted as someone who just
> happened to be in this cockpit with the flight crew.*
>
> **Training Tip, Practice and Feedback***: In a training scenario, the
> crew is aware they are being observed, so it is unlikely that a crew
> member will not participate. Still, the instructor/observer should be
> alert to look for such an occurrence and in the debrief, discuss what
> was observed and how that could be corrected.*

VI\. Emphasize to Captains when pointing out a loss of situation
awareness to another crew member, to do so with consideration for the
other person.

> *What appears to you as a loss of situation awareness on the part of
> another may actually be your own confusion. Even if the other has
> actually lost some awareness, by pointing this out with tact it
> reduces the chance of defensiveness or bad feelings and helps maintain
> the team. In an informal survey at four air carriers, Captains
> frequently stated that CRM had made them more aware of the need for
> consideration of others (Appendix B, Research Summary I)*
>
> [Example]{.underline}: *(ASRS Report 156735).* Our clearance was LGA 3
> Whitestone CLB, Runway 13, Radar vector ELLIOT, etc. The Captain and I
> had been discussing an area of stress developing between us (which has
> been increasing). It seems he has little or no respect for FO's in
> general and that we are here to do the Captain's wishes. I had pointed
> out the immaturity of this attitude and how easily threatened a 24
> year Captain could be and my perception of the absurdity of this.
> Apparently no one had ever challenged him before and I could see truth
> and logic having their effect in a positive way. He appeared to be
> shaken to a degree, but not distracted from his responsibility as
> Captain. I read our clearance to him, not stating, "Whitestone CLB",
> but choosing to read the actual Whitestone CLB requirement. I read
> this as we started to aircraft checklist. All checklists were then
> completed. We pushed and started engines and taxied for take off. As
> we waited our turn, I again read, "175 degree turn on takeoff, left to
> 040 degrees at 1500 feet". **Apparently, the Captain never realized
> what I was saying and still had his mind on our discussion of his
> cockpit demeanor**. As I climbed on 175 degree heading, reaching
> 1500', I began a 30 degree bank turn left to 040 degrees and he
> suddenly asked, "What are you doing? You must turn right to 275
> degrees (Coney CLB)". He asked me to stop my turn until he could
> clarify our clearance. I assured him of the correctness of my actions.
> By that point, however, we had entered JFK's air space as pointed out
> by LGA departure ATC. We quickly maneuvered back on course and no
> problems were created. Moral: Keep it all business, regardless of
> personality clashes until in the crew room or parking lot.
>
> **Training Tip**: Although the Captain in this example had not been
> told that he had lost awareness, his professional behavior had been
> questioned. He became preoccupied with the criticism and therefore
> lost awareness as a result.

VII.Discuss the need of Captains to find out about the experience level
of others. This way, the crew members who are more experienced will
anticipate problems and be more alert to the awareness of the other crew
member. Unfamiliarity or lack of experience (with the route of flight,
the airport layout, the airplane, the company, the other crew members)
lowers the situation awareness of the crew. They should also be
encouraged to confess their own lack of experience in a situation.

> *Humans have limited attention capacity. As we become familiar with
> tasks and situations we have to spend less attention to them and can
> be alert to other things. Someone who is not familiar with the
> aircraft, the route, the equipment, or the airport layout or
> taxi-ways, for example, will have to attend closely to the unfamiliar
> and may not notice something as obvious as a call from Tower.
> (Appendix B, Summaries I and II)*
>
> [Example]{.underline}: *(ASRS Report 243880)* Copilot is PF. Prior to
> descent I set up my approach charts for arrival. I look to the
> copilot's side and see that he has the ILS approach plate out and
> ready to brief the one involving a DME ARC and final approach course N
> of the field. The approach I expected to use was the VOR-A straight in
> from the S, the direction we were approaching from. The destination
> wind was out of the N at 7-10 knots, right down the runway. The
> visibility was excellent (30 miles) and the ceiling was high enough
> (6000-8000 feet) allowing for a nice efficient straight in descent
> over the VOR. However, descent must begin 50 NM sooner to allow us to
> descend into VMC, acquire the airport and runway visually and cancel
> IFR before the VOR passage. I go ahead and request lower altitude and
> controller asks which approach we would like, the ILS 11 or the VOR-A.
> I now realize the copilot is not aware of the VOR-A approach and is
> asking me why we wouldn't want to approach from the N on the ILS
> Circle to land N. I explain the inefficiency of his plan based on the
> wind and ceiling. Now I have allowed a dangerous situation to develop
> During cruise, I did not take the time to confirm that the copilot was
> aware of the ceiling, wind and visibility pointing to a straight in
> arrival from the S and that the copilot was aware of the VOR-A
> approach as the inst approach procedure used to let down from en-route
> to VMC and subsequent cancellation of IFR. At top of descent I realize
> that the copilot does not have the approach plate for the VOR-A handy,
> is not familiar with it and is unable to brief it to the crew prior to
> descent. At this point, I could've taken the airplane as PF and
> initiated descent while he caught up with me **(If my first mistake
> was not ensuring that everyone was familiar with the arrival that I
> was expecting so that there was plenty of time to discuss it and
> understand it, my second mistake was not initiating descent myself and
> assuming the role of PF to prevent a hurried high speed descent).**
> Instead, the copilot was frustrated and confused by his hurried
> descent. As we approached the VOR at 8000 feet in IMC, I told him we
> could not proceed beyond the VOR unless we were in VMC and that we
> must now turn L in a course reversal to intercept and use the
> procedure turn to continue descent. At this point, the copilot asked
> me to "Take the airplane" and I did, momentarily over-banking to 45
> degrees (max. 30 degrees). As I turned hastily outbound while he
> coordinated our procedure turn entry with the controller within
> moments we were in VMC. Visually acquired the runway, canceled IFR and
> contacted airport advisory service. I gave the airplane back to the
> copilot who completed the approach and landing. I apologized to the
> copilot for my not confirming with him earlier about his familiarity
> with the airport and NAVAIDS and thanked him for handing over the
> airplane when he did. This was the first leg of the duty day. I had
> flown with him twice before. We had flown into this airport 11 months
> ago at night using the ILS without incident**. In future, I will
> ensure my crew members familiarity with a particular operation before
> its too late.**
>
> **Training Tip**: *The example, with the Captain's stated intention,
> could be used to initiate a class discussion, where crew members are
> encouraged to share their own experience with a similar situation,
> either where they lost awareness themselves because of a lack of
> familiarity or were flying with someone else who did.*
>
> **Training Tip for Practice/Feedback**: *In the initial meeting* *of
> the crew members for the training scenario, the instructor should
> observe and make note of any questions the crew members ask of one
> another that give them information about their knowledge ("Been to LAX
> recently?") or about the personal readiness for the flight ("I
> understand you just got back from vacation"). During de-brief, ask
> crew members if there is any information that they could have had
> about the other that would have helped in the flight.*

VIII.Make crew members aware that they are particularly vulnerable to
losing situation awareness in high workload situations.

> *ASRS data show that the largest number of reports about a loss of
> awareness occurred during high workload, where problems often
> distracted the crew members from their normal duties. (Appendix I,
> Research Summary VII). This can be helped by alert Captains, who,
> whenever possible, can re-assign duties to keep one crew member from
> being overloaded.*
>
> *Other crew members, alert to one another's activities, can also step
> in and help. Crew members need to be alert to how automation can help
> or hinder in these situations. For example, the autopilot may handle
> some of the workload, whereas re-programming a computer during high
> workload may overload the crew members.*
>
> [Example]{.underline}: *An example of the effect of a high workload
> situation follows (ASRS Report 180082).* We were approaching DFW from
> the SW on the ACTON arrival. ATIS informed us to plan on a visual
> approach to 18R. When we turned N (on downwind) we were given descent
> and frequency change. The FMS and MCP were set for 18R. The new
> Controller told us to expect 17L and a short final. The time involved
> with reprogramming everything took me until base leg, while the
> copilot flew the aircraft (descending from 11000 to 3000). Upon
> turning base, the Controller asked if we had the field in sight. The
> copilot said he did and then the controller cleared us for a visual to
> 17L and switched us to Tower. At this point we began configuring the
> aircraft for landing and running the landing checklist. When we came
> upon the Tower\'' frequency, he offered us 17R and we accepted it. As
> we rolled out on final, I was busy reprogramming the radios and FMS
> for the third time and did not notice the copilot had lined up on 18R.
> Approximately one mile later I finished setting up the cockpit and saw
> on the NAV display that the aircraft was displaced to the right or W
> of course. After checking frequencies and courses very quickly, I
> looked outside and started to tell the copilot he had lined up on the
> wrong runway. At that moment the Tower called to confirm we had the
> airport in sight..we were approaching the outer marker... and we
> turned toward 17R. The remainder of the approach was uneventful.
> Contributing factors were the fading daylight and the copilot had been
> out of this cockpit for 2 months while transitioning to another
> aircraft which left him a little behind and pre-occupied. There were
> not other aircraft on approach to either side of DFW. **Changing
> approaches and runways inside of 10 miles (even in VMC) in this high
> workload 2 person cockpit aircraft represents a challenge, even when
> both pilots are experienced and current...**
>
> **Training Tip**: *This information should be included in special
> training for Captains. It is one of the tasks of the Captain to try to
> assign and reassign tasks so that no one crew member has more than
> he/she can do. Careful planning and preparation also help the crew
> function efficiently so that workload does not increase. And, finally,
> if workload is high for all and cannot be helped, then it is up to the
> Captain to ensure that the crew's tasks are prioritized so that
> attention is placed on only those important tasks.*
>
> **Training Tip, Practice and Feedback***: Include a high workload
> section in a training scenario. Be sure that it is a realistic
> simulation. Be prepared to ask the crew members: 1) how workload
> affected their awareness and 2) what they did, or could have done to
> decrease the workload (e.g., prioritize, use automation, stop using
> automation, Captain reassigning tasks, a crew member confessing that
> he/she was over-loaded.)*

**Appendix A**

**Training Tips**

**Training Tips**

**Classroom Training**

**I. Use examples of either real or potential situation awareness
problems to demonstrate a point**.

> *This helps make the connection between what is learned in the
> classroom and the cockpit. It often serves as a reminder of a
> situation that crew members have experienced and helps drive the point
> home.*

**II. Determine if the crew members are already familiar with the
specific actions that contribute to individual situation assessment and
awareness.**

> *If they are, more time can be spent focusing on the team actions and
> processes.*

**III. Select examples to use during training that are relevant to the
crew members' experience**.

> *Examples provided in the Guidelines are from a variety of aircraft
> types; a few even specify type of aircraft. Although they are
> illustrative of problems crew members have in general, it is sometimes
> difficult to make a point if crew members become concerned about
> differences in their aircraft. If training crew members who fly a
> glass cockpit, examples should be from a similar aircraft. If no
> appropriate examples can be found from their fleet, present an example
> and ask how that would relate to situation requirements in their
> aircraft.*

**Scenario Based Training**

Scenario based training (e.g., LOFT, SPOT, LOS) is a well established
method of training in aviation and it is considered to be the method of
choice for training CRM related actions. Although most scenario based
training includes the use of high fidelity simulators, it can be
presented as a problem solving situation or on a table-top training
device. These additional methods can be used to enhance the training
that follows in the simulator*.*

*Active training, with practice and feedback has been addressed with a
number of the guidelines for developing the training content. The
guidelines in this section are aimed at general considerations when
using scenario based training and were arrived at through their use in
research.*

**I. Consider the difficulty level of the scenario when building a
scenario for training or evaluating situation awareness**.

> *Easy scenarios may not be demanding enough to allow enough
> discrimination among crews to diagnose problems with situation
> awareness. If crew members who have not developed ways to ensure their
> situation awareness is high are not distracted by problems or events
> in their environment, they may be able to maintain situation awareness
> as well as anyone else. When, however, unexpected events occur that
> make demands on crew members' attention and memory, then there may be
> dramatic differences in the observed awareness of the* crews. *(See
> Appendix B, Research Summary II)*
>
> [Example]{.underline}: Crews made up of either experienced crew
> members or inexperienced crew members flew a relatively easy scenario.
> There was no difference in their ratings on situation awareness. When
> the same crews flew a more demanding scenario, experienced crew
> members performed more actions that are related to high situation
> awareness*.*

II\. Use a scenario on a low fidelity training device for training
specific situation awareness actions that you want to see a crew
demonstrate in a LOS scenario

> *Low fidelity training devices do not require crew members to perform
> technical tasks as they would in a high fidelity simulator or the
> aircraft, but with careful scenario development and presentation, they
> do provide an adequate environment to elicit some crew member actions.
> Practice and feedback with such an instrument can lead to training
> some situation awareness actions. This allows concentration on other
> elements in the LOS scenarios. . (See Appendix B, Research Summary
> IV). In addition, such a device helps crew members overcome an
> aversion to the artificial nature of role-play by providing tasks to
> accomplish that are similar to many flying tasks (e.g., maintain
> altitude, navigate, handle ATC communications, be alert to traffic,
> communicate with one another).*

III\. Use a simple table-top training device to have crew members fly a
scenario. The scenario can be stopped to ask questions of the crew
members and to direct their attention to important elements for
awareness

> *Immediate feedback on actions is considered to be important for
> training. With a low fidelity system, intrusion in the scenario for
> questions and for feedback has been found not to interfere with crew
> members interest in pursuing the scenario. (See Appendix B, Research
> Summary III)*
>
> **Training Tip***: Develop a realistic scenario, taking into account
> the device's limitations. For example, develop a scenario that does
> not call for equipment malfunctions that cannot be simulated on the
> device. Include in the scenario questions that could legitimately come
> from outside agencies, such as ATC requesting a PIREP. Responses to
> the questions often give some indication of the crew's awareness of
> the situation. It is also possible to stop the scenario periodically,
> blank the displays, and ask crew members questions relating to where
> they are, where they are going, and what they expect to occur.*

IV\. Introduce scenarios in the classroom setting. Form crews with the
class members and have several crews fly different short scenarios while
class members watch and take notes for de-briefing the crews.

> *This technique has been found to be successful for training in other
> settings as well. Two elements are responsible for the technique
> working, 1) the crew members know that they are going to have to
> perform in front of the class and 2) the use of checklists that
> re-enforce the learning of actions that have been introduced in the
> section of the class dedicated to information and demonstration of the
> concepts (Appendix B, Research Summary VIII).*
>
> [Example:]{.underline} One organization, where class make-up could be
> controlled, ensured that they had classes made up of four people each
> in designated crew positions (e.g., Captain, FO, SO). Classes were
> then separated into four crews. Each crew flew a different scenario
> lasting about 15 minutes while the other class members watched.
> Scenarios were developed to be realistic and to illustrate a point
> made earlier in the training. It was found that the experience of
> watching others and critiquing their performance was useful for
> training. In all classes where this was done, the last crew of the day
> flew their scenario with the highest level of CRM skill.
>
> **Training Tip:** *This technique works best when there are at least
> two facilitators in the class, since one person will be involved in
> running the scenario and playing ATC and other necessary roles (e.g.,
> dispatch, ground, other aircraft).*

V. Include situations that are a threat to crew situation awareness when
developing scenarios for training. These were listed in the section,
Guidelines for Training Program Content.

> *Because situations that cause problems for crew members in
> maintaining situation awareness have been identified, it is useful to
> include some of these situations in training scenarios. This gives
> crew members an opportunity to apply new techniques discussed in
> training in the situation, or, it gives them the opportunity to see
> where they could have improved their own performance if they had used
> these techniques***.** *(Appendix B, Research Summary VII)*

VI\. Develop a debrief form that can be used by both class members and
instructors for observing crews in the scenarios used for training.

> *It has been found that crew members have difficulty in recognizing
> actions that are related to situation awareness and other CRM skill
> areas. This can be improved by supplying observers with a list of the
> actions they need to identify. (Appendix B, Research Summary VIII)*
>
> **Training Tip***: This de-brief form should be simple and tailored to
> both the scenario and the general information about situation
> awareness provided in your program. It should include actions to look
> for that will be un-ambiguous and easy to observe.*

VII.Develop a debrief guide to refresh instructors on the scenario's
purpose and what they need to look for in the scenario beyond, "Did it",
"Didn't do it".

> *Consider that it is possible that the instructor is not very familiar
> with the scenarios. Make his/her preparation easy and increase
> effectiveness by giving guidance on the scenarios. Include information
> about: the purpose of the scenario, what will occur in the situation,
> expected actions of the crew members and how those actions relate to
> the information given in the training on situation awareness. Also
> include suggestions for alternative actions that the instructor may be
> able to provide to crew members. This guide is to help ensure that the
> instructor who is not very familiar with the scenario will be as
> effective as one who knows the scenario well.*

**VIII.Select examples to de-brief that are explicit and demonstrate
specific concepts**.

> *This makes the debriefing easier and more acceptable to the crew
> members. If the examples are subtle, they may be difficult for crew
> members to see their relationships to concepts that have been
> presented.*

IX\. De-brief exercises to make connections for crew members on major
concepts. Don't assume they understand, make sure they do.

> *It is often the case that observers see actions that people in the
> situation are unaware they have taken. Be sure the observers know this
> and that they also know how the actions they observe link to the
> classroom materials. It is important that they make that link.*

**X. Use one training method to convey both individual and team
situation assessment/awareness concepts**.

> *For example, if debriefing a role-play ask individual role play
> members about their experience, ask the role play team about their
> experience as a team, and finally, open the discussion to the larger
> group for their input on individual and team actions*

**Appendix B**

**Appendix B**

**Contents**

Research Summary I Interviews

Research Summary II Research on CRM Skills and Performance

Research Summary III Two Measures of Situation Awareness

Research Summary IV Training Transfer Using Low Fidelity Trainer

Research Summary V Situation Awareness and Its Impact on Decision Making

Research Summary VI CRM Skill Training Evaluation

Research Summary VII An Analysis of the ASRS Data Base for Situation

> Awareness

Research Summary VIII Observation of Videotapes

Research Summary IX Pilot Not Flying Skills

Research Summary X Flight Knowledge, Situation Awareness and Performance

Research Summary XI Situations That Elicit Crew Member Comments

**Research Summary I**

The purpose of the interviews was to gain crew members' perspective on
the interactions and knowledge required for CRM skills, particularly for
situation awareness. Two different sets of interviews with crew members
were conducted.

The first set of interviews were critical incident interviews, in which
crew members were asked to describe their best, and worst, experience
flying. Crew members interviewed were from military, air/sea rescue, and
transport communities. Their answers to questions were transcribed and
the transcriptions were examined for the contribution of the crew to the
situations described. For example, one individual told of a harrowing
experience flying in a violent storm with a mechanical problem. He
related how he and his co-pilot would trade off tasks, almost
automatically, to give one another some relief. At the same time, both
crew members would take on the new task almost seamlessly. He described
the interactions between the two during this situation. Descriptions
from over two hundred crew members revealed a number of actions that
were considered important for team situation awareness and other CRM
skills across a variety of situations. One cluster of these actions were
classified as important to either gaining or maintaining crew situation
awareness. **These actions include: identifying problems, potential
problems; demonstrating knowledge of the actions of other crew members;
keeping up with the details of the flight; and verbalizing actions and
intended actions.**

A second set of interviews were conducted in the air transport
community. These interviews were specifically aimed at gaining an
understanding of situation awareness from the crew members' perspective.
Three experience levels of pilots were interviewed: those with under
1000 hours of flight experience who were working as flight instructors
until they could be hired by an airline; line pilots (primarily First
and Second Officers) with an average of 6000 hours experience; and
instructors/check airmen with an average of over 12000 hours experience.
The same set of questions were asked each respondent and the answers to
the questions were transcribed. The answers revealed a developmental
trend in some aspects of knowledge about situation awareness. For
example, when asked to define situation awareness, low flight time
pilots tended to apply it only to what they could see in the cockpit and
immediately around them. As experience level increased, pilots included
more aspects of the flight and more detail and they recognized that they
have an active role in the situation and they need to be alert to how
their actions are going to affect the situation. **All pilots expressed
a belief that preparation and planning and communication were essential
aides to crew situation awareness. The more experienced pilots added
adaptability and leadership actions to those they considered important
to maintaining awareness as a crew.**

References: Prince, C. and Salas, E. (1993) Training and research for
the military aircrew**.** In E. Wiener, B. Kanki and R. Helmreich (Eds.)
Cockpit Resource Management. San Diego

Prince, C. and Salas, E. (in press). Situation assessment for routine
flight and decision making. International Journal of Cognitive
Ergonomics**.**

**Research Summary II**

This research was designed to determine if the actions of crew members
that relate to situation awareness (as described in Research Summary I)
can be observed and measured in crew members, if there is a difference
between awareness of different experience level aviators, and if these
actions can be related to technical performance.

One hundred crew members volunteered to take part in this research. They
were separated into two groups, those who were low experience level
aviators and those with high experience levels. Within each group, the
pilots were randomly assigned to fly in two person crews. Each crew flew
two scenarios. Half of the crews flew scenario A first, half flew
scenario B first. Each crew briefed their flight, flew the scenario, had
a short break and then briefed and flew the second scenario. Crewmembers
were allowed to de-brief following each flight but they were not given
any evaluation by the instructors who were observing them. After
completing the second scenario, each pilot was asked to rate the
difficulty level of the two scenarios. Scenario B was considered to be
more difficult than scenario A. Technical performance was rated for each
crew using a form that contained the important elements in each scenario
and the expected crew actions. Actions of the crew members related to
CRM that were derived from the critical incident interviews described in
Research Summary I were also observed and rated. This research yielded
several important results:

1\. Performance of the CRM actions were related to technical
performance, that is, crews that performed better technically were also
higher performers in the CRM skills. Thus, the actions identified in the
interviews are related to better performance.

2\. In the easier of the two scenarios, less experienced crew members'
situation awareness scores were similar to those of more experienced
crew members, but, in the more difficult scenario, where workload was
higher, experienced crew members maintained awareness actions better
than inexperienced crew members. Thus, experience does aid crew members
in gaining and maintaining situation awareness when their attention is
demanded for other tasks.

3\. Merely flying a scenario did not ensure that the crew's performance
would improve in the second scenario. Some crews did better, some worse,
some remained about the same. Recall that the crews were not given any
feedback on their performance after the first scenario. It did not make
any difference whether the crews flew the easier or the more difficult
scenario first. Thus, practice in a LOFT scenario without focusing on
the performance does not necessarily constitute training.

Reference: Prince, A., Prince, C., Brannick, M. and Salas, E. (1997).
Measurement of team process behaviors. In M. Brannick, E. Salas, and C.
Prince (Eds.) Team Performance Assessment and Measurement. Mahwah, NJ:
Lawerence Erlbaum Associates, Publishers

**Research Summary III**

Since crew situation awareness is made up of individual awareness as
well as crew processes, this experiment was done to investigate the
potential for questioning during a scenario to indicate the level of a
crew's situation awareness. A second goal was to compare situation
awareness as measured through the questioning technique with a measure
of awareness based on the actions of the crew members. (Those actions
were identified in Research Summary I and verified as important in
Research Summary II).

Situation awareness, particularly individual situation awareness, has
been difficult to measure since much of what is involved in situation
assessment is something that goes on in the pilot's head. There are some
researchers (e.g., Endsley, 1995) who believe that stopping pilots
during a scenario and questioning them about important elements in the
scenario reveals their level of situation awareness. Endsley has used
this technique to assess situation awareness of military pilots.

Eighty pilots volunteered to take part in the research. All crews were
videotaped as they flew a simulator scenario from one central Florida
airport to another. The scenario took place in the summer and there were
a number of thunderstorms in the area. In addition, there was
considerable traffic along the route. No emergencies were included in
the scenario. The air traffic controller's role was scripted. Three
stopping points in the scenario were selected and a question set
developed for each. All pilots were stopped at the same points in the
scenario and were given the same questions. At each stop, they were
given two minutes to answer the questions. Questions were written on
paper and pilots wrote out the answers. Pilots answered the questions
individually.

The questions were developed by three experienced aviators/researchers
to include those elements that crew members should know at that
particular point in the flight. When the experiment was completed, the
questions were graded and each individual was given a grade that was a
simple sum of his/her scores on the question sets. The two pilots'
scores in each crew were then added together to obtain a crew score.
Three crews scored two standard deviations above the mean of all the
scores and three crews scored close to two standard deviations below the
mean. The videotapes of these six crews (representing the highest
scorers and the lowest scorers) as they flew the scenario were then
observed by two researchers who did not know what the team scores were.
Observers rated the crews on their actions as had been defined from the
interviews (i.e., identify problems/potential problems; verbalize
actions/intended actions; show alertness to flight details and
performance of other crew member). The frequency of these actions was
documented to be higher in those teams that were high scorers than in
those teams that were low scorers**. In other words, the actions that
had been identified in the interviews (discussed in Research Summary I)
and confirmed to be related to technical performance (Summary II) were
related to the situation knowledge displayed by the crew members as
measured by their answers to the question sets.**

Reference: Prince, C., Salas, E., and Stout, R. J. (1995, November).
*Situation Awareness: Team measures, training methods*. Paper presented
at the International Conference on Experimental Analysis and Measurement
of Situation Awareness, Daytona Beach, FL

**Research Summary IV**

In this research, the goal was to determine if a low fidelity training
device could be used to train some crew member interactions and if that
training would transfer to a high fidelity simulator. The result of such
training transfer would mean that some active training could occur
before a LOFT scenario, so that the LOFT could be used to integrate
simpler actions (demonstrated and learned in a low fidelity device) in
the more complex technical situations that a high fidelity simulator can
present.

100 pilots volunteered to fly a LOFT scenario in a high fidelity
simulator. They were assigned, at random, to 50 crews, made up of 2 crew
members each, Twenty five of the crews first flew a scenario on a
table-top training device and twenty five crews played team/management
"games", similar to those used in some CRM awareness phase courses. This
was done to give crew members in this group an opportunity to interact
with one another. Crew members who flew the table-top device were
debriefed about their CRM actions after the scenario they flew on that
device, prior to flying the LOFT scenario in the high fidelity
simulator. The de-briefs were conducted by experienced instructor pilots
who observed the crews as they flew the scenario. There were specific
learning points in the training scenario, and the LOFT scenario was
constructed to test some of these points. For specific CRM actions that
were de-briefed, there was a strong training effect. That is, those
crews who flew the scenario on the table top trainer demonstrated by
their performance in the LOFT that they had learned from specific
training points that had been included in the training scenario and in
the de-brief. **The researchers concluded that even though the training
device had low physical fidelity to the aircraft, its CRM demands were
sufficient to provide training for the crew members in those CRM
activities, and that this training could be seen in the actions of crew
members in a LOFT scenario.**

Reference: Brannick, M., Prince, C., Salas, E. and Stout, R. (in
preparation). Development and evaluation of a team training tool.

## Research Summary V

The purpose of this research was to determine if situation awareness of
crew members, as displayed by their actions in the cockpit during
routine portions of the flight, is related to crew decision making. The
decision making investigated related to abnormal, unexpected situations.

In this research, videotapes of 100 crew members (50 crews) were
analyzed. These crew members had flown two different scenarios, each
with distinct decisions included. Two different decision points were
selected, one from each scenario. One decision point represented a clear
problem that all crew members were aware of but that required a
"creative" solution. The second was a subtle problem situation that,
once the crew members were aware it existed, could be solved by the
application of a procedure. Observers searched each videotape for the
beginning of the problem situation. They then backed up the tape to a
point five minutes before the onset of the problem could be observed.
During the five minute period before the problem situation occurred,
observers counted the actions that each crew member took that could be
classified under the four general actions that had been identified
through the interviews (see Research Summary I). Those crews who more
frequently performed these actions were found to come to a solution to
the problem **faster** for the creative problem and to recognize a
problem situation **sooner** for the subtle problem than those crew
members who showed few of these actions. **Because time is often a
critical element in flight, these results indicate that crew members who
are maintaining a high level of crew situation awareness, even in
routine flight, are likely to be quicker in identifying and solving
problem situations when they occur.**

Reference**:** Jentsch, F., Bowers, C., Settin-Wolters, and Salas, E.
(1995). Crew coordination behaviors as predictors of problem detection
and decision making. Proceedings of the 39^th^ Meeting of the Human
Factors and Ergonomics Society. Santa Monica, CA

**Research Summary VI**

The research described here was done to evaluate the training of CRM
skills. Evaluation was done on three levels; attitude change, knowledge
change, and action change. Only the evaluation related to actions is
described here.

A sample of crews from three different fleets of aviators flew a LOFT
type scenario, were trained in team process skills (including team
situation awareness) and flew a second (and different) LOFT type
scenario. The performance of trained crew members in the second scenario
was compared in two different ways. First, their performance in the team
process skills was compared with their pre-training performance in the
first scenario. Second, their performance in the second scenario was
compared with crews who had not been trained and who flew both
scenarios. It was found in all three fleets that the training increased
the use of good team skills. In all cases, crew members increased their
CRM skill actions over their performance prior to training and, in all
cases, trained crew members performed more of the CRM skill actions than
did untrained crew members.

**This research indicates that team process skills can be trained, even
in experienced crew members. This training makes a difference in the use
of those skills in a simulated aircraft flight.**

Reference: Stout, R., Salas, E., and Kraiger, K. (1997). The role of
trainee knowledge structures in aviation team environments.
International Journal of Aviation Psychology, Vol. 7 (pp.235-250).

**Research Summary VII**

In this research, the ASRS data base was searched for situation
awareness incidents. This search was done to increase information about
situation awareness incidents and to determine who is most likely to
lose awareness and what conditions are most to result in a loss of
awareness.

Over 300 reports were included in the analysis. They were the result of
four different searches of the data base where 100 incidents were
requested for each of the following circumstances: 1) for those that
happened when the Captain was the pilot flying; 2) for those where the
First Officer was the pilot flying; 3) in regional airlines and 4) in
crews where one crew member was described as inexperienced or junior.
This analysis was done to try to replicate the findings of an earlier
analysis where 159 reports were analyzed, all of which were obtained
from a search for incidents that included words such as, situation
awareness, judgment, risk, safe, and CRM (Gibson, Orasanu, Vileda, and
Nygren, 1997).

The analysis of the collected incidents showed that:

1\. Approximately 85% of all the incident reports included a mention of
loss of awareness.

2\. In 60% of the reports, the entire crew was described as having lost
awareness; in 25%, it was the Captain only; and in 17% it was the First
or Second Officer.

3\. Across incident reports, some common factors seem to consistently
contribute to loss of awareness. The four most prominent of these
factors are: high workload, distractions; lack of crew communication,
coordination; execution of an improper procedure by a crew member; and
maintenance factors and technical problems.

4\. The remaining factors, in order of occurrence, were: lack of
experience; weather; fatigue; and emotional crew member.

5\. These data confirmed the findings of the earlier analysis by Gibson
et al. even though a different set of raters were used and a different
set of incidents was selected. This replication increases the confidence
that can be placed in the findings.

Reference: Jentsch, F., Barnett, J. and Bowers, C. (1997). Loss of
aircrew situation awareness: A cross validation. Poster proposal
submitted to the HFES 1997 Annual Meeting.

**Research Summary VIII**

The experiment described in this summary was conducted to gather
information about the ability of observers to discriminate important
actions of crew members. It also investigated the value of specific
instructions and an aid to observation on the observers' ability to see
and evaluate crew member actions.

Sixty pilots with different levels of flight experience were assigned to
one of three groups. They watched a videotaped flight simulation with
positive and negative examples of selected crew actions. In some cases,
consequences for the actions were shown (a crew member was either
rewarded or punished for action taken) and in others they were not. Each
group was given different instructions for observing videotapes. The
first group was told to watch for positive and negative crew actions.
The second group was told to focus on the actions of the Pilot Not
Flying (PNF). Instructions for the third group were to focus on the PNF
and to look for certain specific actions (provided on a list to each
pilot in Group Three to use while observing the videotape. All pilots
watched the same videotapes and recorded and rated their observations..

The results of analyzing the observation forms of the pilots were:

1.  More experienced observers recognized more actions than less
    experienced observers.

    2\. The amount and contents of the instructions given to each group
    of pilots affected the quality and quantity of observations. Those
    pilots who were given instructions to observe the PNF (Group 2)
    recorded more actions than did those who were simply told to look
    for the actions within the crew (Group 1). Pilots who were told to
    observe the PNF and who were given a list of actions (Group 3) were
    able to document more actions than those who were merely instructed
    to observe the PNF (Group 2).

    3\. Negative examples of actions were recognized more often than
    positive actions.

    4\. Those examples with a consequence shown were recognized more
    readily

    5\. Negative examples profited more from having their results shown
    than did positive examples of actions.

    6\. Positive examples of actions were found to be somewhat more
    distinctive for the observers than negative examples.

    **The authors of the study make the following training
    recommendations:**

    1\. Observers need to have a level of task relevant knowledge and
    experience or they will not be able to recognize actions.

2.  **Observers need instructions on whom to observe as well as what to
    observe.**

3.  **Observers are more likely to recognize negative actions and that
    adds to their learning. However, they should also be exposed to
    examples of desirable and acceptable actions.**

4.  **Observed actions should show a consequence. This aids learning.**

Reference: Jentsch, F., Irvin, J., Bowers, C., Smith-Jentsch, K. and
Salas, E. (1997). What determines whether observers recognize targeted
behaviors in modeling displays? Unpublished Manuscript**.**

**Research Summary IX**

This research was conducted to test training approaches for training
junior First Officers in monitoring and challenging actions when they
are the pilot not flying (PNF). Specifically, a meta-cognitive training
intervention was tested to determine if it would help First Officers
assess their understanding of a situation before committing to a course
of action, teach them to monitor and control a systematic assessment of
risk, and to use available time more effectively.

Forty nine pilots, all commercial pilots/flight instructors with
multi-engine and instrument ratings who aspired to enter the airlines as
a First Officer, volunteered for this research. Pilots were randomly
assigned to one of three different training programs, meta-cognitive
training, turbine engine training, or behavioral training (primarily
assertiveness training). Each training program was of equal length. The
turbine engine program was considered the control group, since no crew
interactions were discussed; pilots received technical training on a
subject that was of interest to them. The meta-cognitive training
addressed why it is important to know when to apply the back-up actions
and how to assess and prioritize in situations. Four different
techniques were introduced to the pilots and they were given an
opportunity to practice the techniques. In the behavioral training,
pilots were introduced to assertive techniques and observed crew members
using (or failing to use) those techniques on a videotape. Finally, they
were given an opportunity to practice the use of the techniques.

In the meta-cognitive training, one of the training tools used was a set
of practice cards. These cards contained descriptions of flight
situations (one situation on each card) that were described from the
perspective of the First Officer. Crew members were asked to read each
card, to determine which back-up behaviors applied, and to use the four
techniques that had been introduced in the course to prioritize which
behavior they would choose. Pilots were guided through this procedure by
a worksheet. An instructor provided feedback on their choices and
justifications. On the reverse side of each card was a description for
what happened in the situation described. See Appendix C.

Training evaluation was on three levels: attitude; knowledge; and
actions in a realistic scenario. It was found that all pilots liked the
training, thought it was useful, and learned from the training (as
indicated by written test performance). Each pilot was scheduled to fly
a scenario as PNF with a Captain, who was one of the researchers and
kept his actions the same for all pilots (and who did not know which
training program each subject had been through). It was found that the
kind of training did make a difference in the actions of the crew
members. **Those trainees who had received the meta-cognitive or the
behavioral training overall performed the role of the PNF significantly
better than those in the control group. Both training programs
significantly improved general pro-active backup (communication and
coordination).**

Reference: Jentsch, F.G. (1997). Metacognitive training for junior team
members solving the copilot's Catch 22. Unpublished doctoral
dissertation. Orlando, FL: University of Central Florida

**Research Summary X**

In this experiment, the purpose was to determine if there is a
relationship between the way that a crew keeps up with current flight
details and their CRM related actions and their performance. It combines
information gained from Summary I and from Summary III.

Forty crews of aviators flew two different scenarios. In the first, they
flew a demanding LOFT type scenario in a full mission simulator. The
scenario contained icing conditions, incomplete information, a boost
pump failure, and an electrical fire. They were videotaped while
completing this flight and two experienced instructor pilots observed
the tapes and rated the crew members' technical performance and their
CRM skill actions. While flying the second scenario, the crews were
stopped three times to answer questions on flight details in the
scenario. These details had been selected by experienced aviators as
being important information for crew members to notice and maintain.
(See Summary III). The scores for the crews' answers to the questions
were computed. These scores were then analyzed for their correlation
with the CRM skill ratings and performance ratings the same crews had
received in the first scenario. Significant correlations were found,
particularly for the skill areas of planning, adaptability, and
communications. Significant correlations were also found between the
technical performance scores and the knowledge question scores.

**This research result suggests that maintaining a current knowledge of
flight details is related to CRM skill actions and technical
performance. In addition, it suggests that crew members may be
consistent in their performance of these actions. That is, crew members
who keep up with flight details in one scenario tend to do so in other
scenarios.**

**Moving Guidelines into Programs**

**Using the Guidelines: Two Suggested Courses**

Because including all the guidelines in a program is not practical,
given time constraints and attention limitations, it is recommended that
the training designer select the guidelines that are most relevant to
the organization's training needs. These needs can be determined by
investigating company incident reports, recent FAA and NTSB findings,
and ASRS reports that indicate potential problems relating to situation
awareness. This section contains two general training programs using a
selection of the guidelines.

**Course on Situation Awareness**

**for Recurrent Training**

**Description**

2 Hour Classroom Course

SPOT/LOFT/LOS or low fidelity simulation to provide practice in Crew SA
actions

Written guidance for instructors to debrief Crew SA

**Introduce Course Objectives**

-   Crew members will be aware that maintaining their own situation
    awareness is necessary but not sufficient

-   Crew members will be familiar with situations that are a particular
    threat to crew situation awareness

-   Crew members will know actions that are important to maintaining
    crew situation awareness

-   Crew members will demonstrate their ability to use the behaviors
    necessary to maintain crew situation awareness

    **Definitions and Guideline I, Training Content:** Emphasize that
    **c**rew situation awareness is different from individual situation
    awareness; individual situation awareness is not sufficient in all
    cases when flying as a crew.

    **Guideline X, Training Content:** There are situations that are
    particular threats to situation awareness. Crew members should be
    aware of the threat and be prepared with actions that counter the
    threat.

    **Guidelines IV-VI, Training Content**: Crew members should be
    reminded of the actions that are important to crew situation
    awareness. In class, emphasize how each of these would work in the
    threat situations identified earlier.

    **Active Practice**: In the training scenario, introduce at least
    one of the situations that is a threat to situation awareness.
    Observe crew member behaviors that indicate they are acting to
    maintain awareness as a crew (e.g., verbalizing actions, showing
    alertness to another's awareness). Be specific about observed
    actions in the de-brief.

    **Course for Training Situation Awareness**

    **to New Hires**

    **Description**

    3 hour classroom course as part of CRM training

    SPOT/LOS/LOFT that includes learning points to enhance classroom
    training

    Instructor de-brief form to ensure situation awareness learning
    points are debriefed

    **Introduce Course Objectives:**

-   Crew members will know that it is important to build and maintain
    awareness as a crew and that this requires more than maintaining
    individual awareness.

-   Crew members will be aware that awareness as a crew can vary and
    that they must act to maintain awareness

-   Crew members will know that situation awareness is dependent upon
    detailed flight knowledge (among other things) and that general
    flight preparation is not enough

-   Crew members will be able to recognize actions that are related to
    crew situation awareness and suggest ways to maintain it

-   Crew members will demonstrate the ability to use the actions
    important for situation awareness in a realistic scenario

    **Definitions and Guideline I, Training Content**: Crew situation
    awareness and individual situation awareness are not the same.
    Individual situation awareness is important to maintain but is not
    enough when flying as a crew

    **Guidelines II and VII, Training Content**: Situation awareness can
    be high in a crew and within a short time, can be low; maintaining
    it requires constant vigilance. Situation awareness for a flight is
    dependent on many things and it always includes specific, detailed
    current flight knowledge.

    **Guidelines IV, V, and VI, Training Content**: Introduce, discuss,
    and practice, in class, the actions important to maintaining crew
    situation awareness. For example, use a videotape of a crew in a
    situation where the Captain's awareness is not high, stop the tape,
    ask crew members if the Captain's awareness is high, what cues
    indicate a problem with awareness, what actions should be taken to
    restore a high level of awareness in the crew.

    **Guideline X, Training Content**: If there is time, simply
    introduce the situations where there is a particular threat to
    situation awareness. New crew members should be particularly alert
    to these situations and maintaining the actions the ensure crew
    awareness.

    **Active Practice:** Crew members who are new to flying revenue
    flights as a crew should be given practice in various scenarios up
    to and including a SPOT or LOFT session. The use of low fidelity
    simulation is particularly useful for introducing actions and
    providing repeated practice and feedback. Having a facilitator
    scripted to play a role as a Captain who shows subtle signs of a
    loss of awareness could be a good exercise for these crew members,
    especially if the scenario requires that they, themselves be busy.
    Such scenarios can be done effectively in a short period and can be
    conducted in the classroom. Be sure to include guidance for the
    instructor to use in the de-brief that follows the course
    objectives.

    **Appendix C**

    **Appendix C**

    **Training Tools**

    **Situations to Compare Crew Member Assessments of Time and
    Criticality**This Appendix contains 15 problem situations and
    possible solutions that can be used as a training tool. Each
    incident is presented in four parts. It is first described and is
    followed by the estimates for risk and time available that were made
    by two different groups of pilots, those who are experienced and
    those who are prospective first officers at a regional airline.
    Third, suggested actions are given for each situation and last, what
    actually occurred in the situation is given. One use of these
    situations is described in Research Summary IX. They can also be
    used in a classroom exercise to illustrate differences in
    estimations of time and risk.

    The description of the problem can be given to a class of crew
    members. They can be asked to assess the situation and come up with
    possible actions of the crew. The instructor can then give them what
    happened in the situation. If the class members consist of newly
    hired First Officers who have little experience in airline
    operations, it may be useful to let them assign risk and time
    available estimates to the situation and them show them how they
    compare with more experienced crew members. If the class members are
    Captains who are being upgraded, it may be useful to demonstrate to
    them that their estimates of time and risk differ from less
    experienced crew members. This helps demonstrate some differences in
    situation assessment due to experience level.

    For more information on these situations and for additional
    materials, see the report on training for the Pilot Not Flying
    issued by the FAA's office of the Chief Scientist.

    **Situations**

    1\. It is the captain's leg, landing in Denver, ILS RWY 36 approach,
    circle to land on RWY 8L. RWY 8L is in sight, 3 to 5 miles out.
    Cleared to land on RWY 8L. You observe that the captain does not
    widen the base leg, as you had suggested to him when you were
    cleared to land.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Low (-0.7) Low (-0.5)

    Time Pressure: Medium (-0.2) Medium (-0.1)

    Options: Few (-0.3) Few (-0.2)

    [Possible Steps to Take]{.underline}:

-   You are still 3 miles out. You could ask the captain again to widen
    the turn.

-   If captain does not positively react, prepare yourself to call for a
    go-around

    If captain does not react to go-around call, initiate go-around
    yourself

    [What Happened in this Case?]{.underline} This incident was
    described by an FO in an ASRS report. The captain, trying to land at
    the approach end of the runway, so as to make an early turnoff, flew
    at low altitude over another aircraft on an adjacent taxiway. On
    short final, at about 50 feet altitude, the aircraft was in a 20-30
    degree bank - with the low wing's tip barely clearing the surface.
    The captain landed the aircraft hard, off centerline, with the stall
    warning horn on. The captain admitted that he should not have
    attempted the landing from the position he was in. In retrospect,
    the FO concluded that he should have initiated a go-around, first by
    verbal command, then in person.

    2\. The controller turns you from HDG 300 to HDG 360 to gain spacing
    behind a company B737 which is also on approach to RWY 27L. You are
    told to descend to 2,500 feet and are now descending through 4,000
    feet to 3,500 feet. You are vectored north, and a Saab 340 is being
    vectored onto final for the parallel runway, RWY 26R. The Saab is
    controlled on a different frequency. You notice that he is also at
    3,500 feet and turning belly up to your plane. The captain is
    slightly behind the plane and does not appear to notice this.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (0.0) Medium (0.0)

    Time Pressure: High (0.5) High (0.8)

    Options: Few (-0.2) Few (0.2)

    [Possible Steps to Take]{.underline}:

-   Inform the captain immediately, yet calmly and precisely ("Captain,
    aircraft eleven o'clock, one mile, same altitude. A Saab 340 - he's
    turning left to right - belly up to us - for the parallel runway")

-   Assess evasive maneuvers, but keep in mind that the other aircraft
    is most likely not seeing you (it is turning belly up to your plane)

-   If needed, advise ATC of your actions

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. Upon
    noticing the problem, the FO advised the captain immediately. Based
    on estimation of distance, relative bearing, and closing speed, the
    crew decided not to initiate a turning maneuver, in order to avoid
    losing visual contact. Instead, the crew made only small corrections
    that allowed them to keep the traffic in sight. An uneventful
    landing followed. Factors that complicated this situation were (a)
    that the captain had low time in the aircraft, (b) that the air
    traffic controller had delayed the turn, and (c) that the
    conflicting traffic was controlled on a different frequency.

    3\. The captain is lined up on the LOC, and the runway is in sight.
    You are approximately 3-4 miles out on final and at 1300 feet when
    the TCASII announces "TFC." You observe a target on the display at 1
    o'clock, 2 mi., converging, indicating 500 feet below you (you are
    descending on the GS). You and the captain both search for the TFC
    but cannot see it. The TCASII announces a "CLB" resolution advisory.
    You look quickly at the display and say to the captain, "You better
    climb." The captain responds by saying , "There is nobody out
    there."

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: High (0.6) High (0.7)

    Time Pressure: High (1.0) High (0.9)

    Options: Only one (0.5) Only one (0.6)

    [Possible Steps to Take]{.underline}:

-   This situation is a tough one: The captain has control of the
    aircraft, the indications from various sources are ambiguous
    (traffic not in sight, yet TCASII-indication of a conflict), and you
    do not agree on a possible solution. Without knowing where the
    traffic actually is, any action (staying on course or maneuvering)
    can be dangerous

-   In general, however, following the TCASII resolution advisory is
    suggested

-   In this particular case, the other aircraft is most likely hidden by
    your aircraft's nose - a climb or climbing turn may thus indeed be
    the best solution

-   You might also want to call the tower controller who may have both
    aircraft in sight (see below) and might be able to assist you in
    keeping separation

    [What Happened in this Case?]{.underline}

    This incident was described by an jump-seat observer in an ASRS
    report. As it turned out, the local TWR controller was indeed aware
    of both aircraft and thought they were sufficiently separated. The
    controller, however, failed to inform the flightcrews of either
    aircraft about the other aircraft's presence. Luckily, no collision
    occurred.

    4\. After landing at HOU, GND instructs you to proceed north on Mike
    and hold short of runway 12L at Oscar. You acknowledge these
    instructions. While on Mike, a DC-9 lands on the runway you just
    exited, passes you, exiting the runway at the high speed taxiway.
    You are told to give way to the DC-9. The DC-9 is then told to
    proceed north on your taxiway and to cross RWYs 12L and 4/22 at
    Charlie. The DC-9 acknowledges and proceeds. As the captain turns
    right onto Oscar, you tell him that you have not been cleared across
    runway 12L. He replies that you were cleared across behind the
    preceding aircraft.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (0.4) High (0.6)

    Time Pressure: High (0.9) High (0.7)

    Options: Few (0.4) Only one (0.5)

    [Possible Steps to Take]{.underline}:

-   In this case of ambiguity and high risk, you must advocate and, if
    there is no positive reaction from the captain, initiate the safest
    option: To call ATC and ask for clarification on the clearance
    before taxiing onto the active runway

-   In addition, you might want to make it a practice to monitor the
    tower frequency on the second radio, to increase your chances of
    detecting conflicting traffic that is either taking off or landing

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. In this
    case, the captain incorrectly assumed that he was cleared to follow
    the preceding aircraft when he was not. Instead of stopping before
    crossing the runway and asking for clarification on the clearance,
    however, the crew continued the taxi. This resulted in a runway
    incursion that required a small aircraft to abort its takeoff from
    runway 12L.

    5\. The captain is flying the LOC BC RWY 24 into ABE. A note on the
    approach chart indicates the use of ETX DME when on IABE LOC course.
    However, FJC VOR is also depicted on the approach chart. The captain
    inadvertently holds the DME from FJC VOR instead of ETX VOR, using
    it for the stepdown altitudes while on the LOC BC. After being
    cleared for the approach, the captain thinks he is inside the FAF
    (Wiley INTXN) and descends to MDA of 760 feet MSL. Suddenly, you
    notice that the captain has been using FJC VOR instead of ETX DME.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: High (0.6) High (1.0)

    Time Pressure: High (1.0) High (0.9)

    Options: Only one (0.6) Only one (0.7)

    [Possible Steps to Take]{.underline}:

-   Immediately inform the captain of the problem ("Captain, we have
    used the wrong DME for our letdown - setting correct one now -
    suggest level off or climb!")

-   Immediately initiate measures to stop the descent

-   If over mountainous terrain or near obstacles, initiate climb back
    to MSA

-   As you inform the captain, dial in the proper DME to identify your
    correct position

-   Inform ATC of your actions to avoid further problems (e.g., traffic
    conflicts)

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. According to
    the report, as soon as the FO noticed the problem, ATC informed the
    flight that they had a "low altitude" alert on their radar. The crew
    climbed back to the intermediate altitude, changed to the
    appropriate DME, and subsequently landed uneventfully. Factors
    contributing to the problem were (a) that the FO was a new hire with
    little experience on the route and (b) that the crew was in IMC.

    6\. You are cleared to descend to 1,800 feet on a heading to
    intercept the LOC. You leave the ATC frequency after acknowledging
    the heading and altitude to speak to the company OPS (you set 1,800
    feet in the altitude alert before leaving the frequency). Just as
    you come back on the frequency, you notice the captain descending
    through 1,800 feet at about 800-1,000 FPM.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (0.4) High (0.8)

    Time Pressure: High (1.0) High (1.0)

    Options: Few (0.5) Only one (0.8)

    [Possible Steps to Take]{.underline}:

-   Cross-check instruments to confirm situation and identify position

-   Immediate call out current altitude and required action ("1,800
    feet, level off!")

-   On the ground, discuss division of tasks with captain to avoid
    reoccurrence

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. Upon
    noticing the problem, the FO cross-checked the instruments for
    full-scale localizer and glide-slope indications, checked the
    altitude alerter, and called out "Level 1,800." The captain
    immediately corrected - at the same time, the altitude alerter came
    on, and ATC announced a "low altitude" alert. On the ground, the
    captain reportedly said that he had expected a "500 feet to go" call
    from the FO - a call that never came because the FO was on the radio
    with company operations. This case again emphasizes the importance
    of clear division of tasks before crewmembers make calls on separate
    frequencies. A contributing factor was fatigue.

    7\. You contact ground control and are cleared to taxi to RWY 7.
    Your route would eventually cross RWY 7. You glance up from reading
    the checklist "heads down" and see that the captain is approaching
    RWY 7 midfield from your ramp. However, you are not cleared across.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (-0.4) Medium (0.2)

    Time Pressure: High (0.8) Medium (0.3)

    Options: Few (0.3) Few (0.2)

    [Possible Steps to Take]{.underline}:

-   Question taxi, initiate stop (if possible)

-   If in doubt or if taxi speed does not allow stopping in time, call
    ATC immediately to inform and clarify

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. Upon
    noticing that the captain was taxiing towards RWY 7, the FO said to
    the captain "Stop, you are not cleared across." The captain,
    although unfamiliar with the airport, however, had not only believed
    that he was cleared across the runway, he had also taxied faster
    than normal. As a result, the airplane was unable to stop before
    causing a runway incursion. Luckily, no other aircraft was using the
    runway at the time. In his report, the FO noticed that he believed
    he could have prevented the incursion had he relied less upon the
    captain and had he spent less time "heads-down" reading the
    checklists. Also, the FO later stated that he should have made his
    presence felt earlier when he noticed other minor errors by the
    captain.

    8\. The captain has the take off from STX, enroute to STT. A Twin
    Otter is in front of you. San Juan CTR give you a 150 KT speed
    restriction not to exceed. The captain refuses to slow down as
    requested.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (-0.18) Medium (0.1)

    Time Pressure: High (0.55) High (0.5)

    Options: Few (-0.09) Few (0.3)

    [Possible Steps to Take]{.underline}:

-   This is a tough situation: The captain has positive control over the
    aircraft and refuses to follow ATC instructions. While in the air,
    you must decide whether the danger imposed by the captain's behavior
    is sufficient to warrant taking control (speed, traffic, climb
    rate) - especially considering the dangers of wrestling for control
    in the air

-   If the problem persists and the captain is known for such behavior,
    however, you should consider reporting him to the airline and/or the
    authorities after landing

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. The FO
    reported that he repeatedly advised the captain to slow down, but
    that the captain refused. ATC, upon noticing the non-compliance,
    gave vectors to avoid further problems. According to the FO, the
    captain was an older pilot with a habit of flying at full throttle
    to arrive at the destination early. In fact, as the FO alleged, the
    captain regularly flew beyond red line, pulled circuit breakers to
    disable the overspeed warning, and generally did not like to work
    with a crew.

    9\. You and the captain are about to depart on a flight from St.
    Louis to Birmingham. After reviewing the convective sigmets and
    terminal forecasts for the route of flight, you realize that you
    don't have enough fuel for the weather you will have to fly around.
    You inform the captain that you will need more fuel but he
    disagrees.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (-0.3) Medium (0.3)

    Time Pressure: Medium (0.1) Medium (0.1)

    Options: Few (0.0) Few (0.0)

    [Possible Steps to Take]{.underline}:

-   This is another situation where you should focus not on "Who's
    right?," but rather on "What's right?" You could, for example,
    suggest to actually work out the fuel for both situations. This will
    allow you and the captain to discuss your respective assumptions

-   You might also want to look into alternatives available along the
    route of flight

-   Further, you might want to suggest involving dispatch/flight
    following, especially if the captain and you cannot reach a
    conclusion \-- dispatchers are a resource for precisely a case like
    this one (you are still on the ground and have some time!)

-   Finally, if you feel that safety is greatly compromised, you should
    refuse the flight

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. After the
    captain refused to get more fuel, the FO backed down, and the crew
    began the flight. In the air, the FO noted that the captain had
    forgotten to plan for an alternate airport. About one hour later,
    the FO was able to convince the captain to divert to Memphis, where
    the crew had to land in a thunder-storm because of low fuel. The
    crew refueled and continued without further incident.

    10\. You are issued an altitude of 14,000 feet. The captain enters
    15,000 feet in the altitude window. You climb to 15,000 feet and are
    corrected by ATC. You are then given a final altitude of 16,000
    feet.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (-0.45) Low (-0.5)

    Time Pressure: Low (-0.55) Medium (-0.3)

    Options: Medium (0.18) Few (0.2)

    [Possible Steps to Take]{.underline}:

-   Climb to the assigned altitude

-   Review the problem and its causes

-   Check whether anything else is amiss

-   When possible, you and the captain should discuss ways to avoid a
    reoccurrence

-   Report problem to the airline, so that SOPs, checklists, role
    assignments, etc. can be changed

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. According to
    the FO, the assigned altitude is normally set into the altitude
    window by the PNF. In this aircraft, however, the altitude window
    and control knob were located on the captain's side, so the captain,
    despite the fact that he was flying, set the altitude. In addition
    to this unusual role assignment, the FO also attributed the problem
    to fatigue: The error occurred during the fifth flight on the third
    day of their trip.

    11\. You receive the clearance "Taxi to the ramp via Papa, hold
    short of November (Outer)." You are occupied with after landing
    chores, including crew log entries, ACARS entries, after landing
    flow/checklist, etc. After an aircraft taxis by (from left to
    right), the captain begins to turn left from Papa, parallel to
    runway 9R. You think he has received clearance to the ramp, but you
    have been busy and the frequency has been congested.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Low (-0.6) Medium (-0.4)

    Time Pressure: High (0.7) Medium (0.4)

    Options: Few (0.0) Few (0.2)

    [Possible Steps to Take]{.underline}:

-   Question the captain about taxi - do not rely on trust alone

-   If situation is ambiguous, initiate stop (if possible)

-   In any case where there is a question regarding a taxi clearance,
    call for clarification from ATC

    [What Happened in this Case?]{.underline}

    This incident was described by an FO in an ASRS report. According to
    the report, the FO did not question whether the captain had received
    clearance to the ramp - the FO trusted the captain that he was
    following a clearance when, in fact, he was not. Upon noticing the
    error, ATC informed the crew who took corrective action. Luckily, no
    incursion occurred. The FO attributed part of the problem to the
    fact that the company procedures regarding after-landing items were
    overloading him at a critical moment.

    12\. 15 miles out from the final approach fix to the ILS35 at MCO,
    the captain asks you to call the airline's station at MCO for your
    "inbound call" (i.e., to report ETA, number of PAX, fuel and
    maintenance requirements, and any special needs for PAX assistance).

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Low (-0.73) Low (-0.9)

    Time Pressure: Medium (0.09) Low (-0.7)

    Options: Few (-0.09) Few (0.0)

    [Possible Steps to Take]{.underline}:

-   Make sure that the captain will listen and respond to the primary
    radio

-   Keep the call as short as possible

-   After finishing the call, make sure to get up to speed on what you
    may have missed

-   For the next flight, anticipate the call earlier than 15 miles (4-5
    min.) out

-   If you believe not enough time is available to make the call before
    the approach, or if you are inside the 'sterile cockpit' window as
    specified by FARs or company procedures, suggest to make the call as
    soon as you have landed

    [What Happened in this Case?]{.underline}

    This situation is based on numerous ASRS reports and on input from
    active captains. Time and again, incidents have occurred while the
    FO was talking to company operations (or other secondary
    facilities). Calls were not answered, clearances not written down or
    misunderstood, etc. It is therefore extremely important (a) to make
    these calls before high-workload periods, (b) to assign who is
    responsible for the primary radio during the calls, and (c) to
    inform the other crewmember about changes after they complete the
    calls.

    13\. You enter the briefing room late from a flight, just as the
    captain leaves. On his way out, he says to you "Sorry, FO, need to
    call my wife, she's pregnant and due tomorrow. I left the flight
    plan on the table, together with the load manifest. I'll see you out
    at the plane. Make sure all the cockpit checks are done so we can
    start up as soon as I get there." As you go over the paperwork, you
    notice that the dispatcher appears to have made a mistake in the
    fuel calculations: You think you are about 300 lbs. \[or 10 min.\]
    below legal limits. The fuel truck has pulled away from the plane,
    and you don't see the captain anywhere. Your start-up time is in 15
    min.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Low (-0.82) Medium (-0.3)

    Time Pressure: Medium (0.36) Medium (0.3)

    Options: Few (-.18) Few (0.2)

    [Possible Steps to Take]{.underline}:

-   You could call the fuel truck back to the aircraft and, if it turns
    out that you do not need more fuel, cancel the order

-   You could also check back with dispatch - most flight plans today
    are created by computer, and dispatchers can quickly recalculate the
    routes, times, weights, etc.

-   Finally, make a mental note that the captain may be distracted by
    his wife's pregnancy. You might want to watch his performance extra
    carefully today

    [What Happened in this Case?]{.underline}

    This situation is not based on an actual ASRS report, but it
    contains many elements commonly found in accident and incident
    reports. The description was included to illustrate a situation with
    high ambiguity and many complicating factors. In particular, we
    wanted to emphasize the importance of taking the initiative when
    time is limited and the captain is not available for a decision.
    Also, we wanted to stress the importance of the preflight phase. All
    too often, accidents and incidents originate on the ground. Finally,
    we wanted to illustrate the potential impact of personal life events
    on crew performance.

    14\. You are at the departure end of RWY12, the tower has just
    cleared you to line up on the runway for TO, when you notice that
    several items of the before-takeoff checklist are not completed. You
    still have to complete the line-up checks as well.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (-0.3) Medium (0.2)

    Time Pressure: High (0.6) Medium (0.4)

    Options: Few (0.2) Few (0.0)

    [Possible Steps to Take]{.underline}:

-   Inform the captain immediately that you still need to complete a
    number of items on the before-takeoff and line-up checklists

-   If you have not crossed onto the runway yet, suggest calling ATC and
    asking them for a brief delay to finish up the checklist items

-   If you have crossed onto the runway, suggest calling ATC and
    informing them that you still have to complete a number of checklist
    items

    [What Happened in this Case?]{.underline}

    This situation was based on a number of incidents and accidents that
    have occurred over the years because crews did not complete critical
    pre-takeoff checklist items. Reasons for such omissions are
    numerous, and only one of them is the feeling of being rushed by

    Continued on next page....

    ATC. Other occurrences that tend to distract the flightcrew from
    completing checklist items include interruptions by cabin
    crewmembers and/or passengers, aircraft abnormalities, and outside
    distractions by ATC, other aircraft, etc. In some cases, however,
    these omissions can have fatal consequences (e.g., when the airplane
    is not properly configured for takeoff).

    15\. In your departure clearance, you were cleared to 3,500 ft. You
    are at 3,200 ft, the climb rate is still 1,000 ft/min, and the
    captain shows no indication that he will reduce the climb angle.

    [Situation]{.underline} [Expert Rating]{.underline} [Prospective FO
    Rating]{.underline}

    Potential Risk: Medium (-0.36) Medium (-0.3)

    Time Pressure: High (0.82) High (0.7)

    Options: Few (0.36) Few (0.2)

    [Possible Steps to Take]{.underline}:

-   Call out "300 feet to go - Assigned altitude is 3,500 feet"

-   At 3,300 ft and 3,400 ft, call out "200 feet to go" and "100 feet to
    go," respectively

-   If the captain does not begin his level off at 3,400 feet and does
    not acknowledge any of the call-outs, challenge a second time:
    "Captain, 100 feet to go, begin level off to the assigned altitude
    of 3,500 feet"

-   If captain still does not respond, assume that he/she is
    incapacitated and take control of the aircraft - calling out
    "Captain, I have control of the aircraft - leveling off to the
    assigned altitude of 3,500 feet"

[What Happened in this Case?]{.underline}

This incident is based on several situations involving involuntary pilot
incapacitation. It also shows the utility of call-outs during climb and
demonstrates an application of the "two-challenge-rule" that is standard
operating procedure in many airlines and in the military: If the PF does
not respond to two challenges from a crewmember, that crewmember has the
obligation to take positive control of the aircraft.

**References**

**The references listed in this section are provided for those
interested in additional readings on situation awareness. Reference that
are in bold type are those mentioned in the text of the Guidelines.**

Adams, M., Tenney, Y., and Pew, R. (1995). Situation awareness and the
cognitive management of complex systems. *Human Factors*, Vol. 37,
85-104

Bowers, C.A., Braun, C., and Kline, P.B. (1994). Communication and team
situational awareness. In R.D. Gilson, D.J., Garland, and J.M. Koonce
(Eds.), Situational awareness in complex systems: Proceedings of the
CAHFA Conference (pp. 305-311). Daytona Beach, FL: Embry Riddle
Aeronautical University Press.

Bunecke, J.L., Povenmire, H.K., Rockway, M.R., and Patton, M.W. (1990,
April). The situation awareness component of cockpit resource
management. In Proceedings of the Psychology in the Department of
Defense Twelfth Symposium (Tech. Report No. USAFA-TR-()-!, pp. 259-263).
Colorado Springs, CO: US Air Force Academy. (AD-A240113).

**Cohen, M., Freeman, J. and Wolf, S. (1996). Metacognition in time
stressed decision making: Recognizing, critiquing, and correcting. Human
Factors, 38(2). (pp. 206-209)**

Endsley, M.R. (1990). Predictive utility of an objective measure of
situation awareness. In Proceedings of the Human Factors Society 34th
Annual Meeting (Volume 1, pp. 41-45). Santa Monica, CA: Human Factors
Society.

Endsley, M. R. (1993a). Situation awareness and workload: Flip sides of
the same coin. In Proceedings of the Seventh International Symposium on
Aviation Psychology (Vol. 2, pp. 906-911). Columbus: The Ohio State
University.

**Endsley, M. (1995). Situation awareness theory. *Human Factors*,
Vol.37, pp. 32-64.**

Flach, J.M. (1995). Situation awareness: Proceed with caution. *Human
Factors, 37*, pp.149-157.

Fracker, M. L. (1988b). A theory of situation assessment: Implications
for measuring situation awareness. In *Proceedings of the Human Factors
Society 32nd Annual Meeting* (Vol. 1, pp. 102-106). Santa Monica, CA:
Human Factors Society.

Fracker, M. L. (1989). Attention allocation in situation awareness. In
*Proceedings of the Human Factors Society 33rd Annual Meeting* (Volume
2, pp. 1396-1400).

**Helmreich, R. and Foushee, C. (1993). Why crew resource management?
Empirical and theoretical bases of human factors training in aviation.
In E. Wiener, B. Kanki and R. Helmreich (Eds.) Cockpit Resource
Management. San Diego, CA: Academic Press, Inc**

Jentsch, F., Bowers, C., Settin-Wolters, and Salas, E. (1995). Crew
coordination behaviors as predictors of problem detection and decision
making. Proceedings of the 39^th^ Meeting of the Human Factors and
Ergonomics Society. Santa Monica, CA.

Jentsch, F., Irvin, J., Bowers, C., Smith-Jentsch, K. and Salas, E.
(1997). What determines whether observers recognize targeted behaviors
in videotaped modeling displays. Paper presented at the Society of
Industrial Organizational Psychology. Saint Louis, MO.

**Moray, N. (1994). Error reduction as a systems problem. In M. S.
Bogner (Ed.), *Human Error in Medicine.* Hillsdale, NJ: Lawrence
Earlbaum Associates, Publishers**

**O'Hare, D. and Roscoe, S. (1990). Flightdeck Performance: The Human
Factor. Ames, IO: Iowa State University Press.**

**Orasanu, J. (1990). Shared mental models and crew decision making (CSI
Report 46). Princeton NJ: Princeton University, Cognitive Science
Laboratory**

Pew, R.A. (1994). An introduction to the concept of situation awareness.
In R.D. Gilson, D.J. Garland, and J.M. Koonce (Eds.), Situational
awareness in complex systems: Proceedings of a CAHFA Conference
(pp.17-23). Daytona Beach, FL: Embry-Riddle Aeronautical University
Press.

Prince, C. and Salas, E. (in press). Situation assessment in the conduct
of flight and in decision making. International Journal of Cognitive
Ergonomics.

Prince, C. and Stout, R.J. (1995, April). Situation awareness from the
team perspective. Paper presented at the 10th International Symposium on
Aviation Psychology. Columbus, OH.

Prince, C., Salas, E., and Stout, R. J. (1995, November). *Situation
Awareness: Team measures, training methods*. Paper presented at the
International Conference on Experimental Analysis and Measurement of
Situation Awareness, Daytona Beach, FL

Salas, E., Prince, C., Baker, D. and Shrestha, L. (1995). Situation
awareness in team performance: Implications for measurement and
training. *Human Factors*, 37(1) 123-136.

Sarter, N.B. (Panel Organizer) and Woods, D.D. (Panel Chair) (1990).
Situation awareness in the advanced commercial aircraft cockpit. In
Proceedings of the Human Factors Society 34th Annual Meeting (Volume 1,
pp.21-25). Santa Monica, CA: Human Factors Society.

Sarter, N.B. and Woods, D.D. (1991). Situation awareness: A critical but
ill-defined phenomenon. The International Journal of Aviation
Psychology, 1, 45-57.

Sarter, N.B. and Woods, D.D. (1992). Pilot interaction with cockpit
automation: Operational experiences with the flight management system.
The International Journal of Aviation Psychology, 2, 303-321.

Sarter, N.B. and Woods, D.D. (1994). Pilot interaction with cockpit
automation II: An experimental study of pilots' model and awareness of
the flight management system. The International Journal of Aviation
Psychology, 4, 1-28.

Sarter, N.B. and Woods, D.D. (1995). How in the world did we ever get
into that mode? Mode error and awareness in supervisory control. Human
Factors, 37, pp. 5-19.

Smith, K. and Hancock, P. (1995). Situation awareness is adaptive,
externally directed consciousness. Human Factors, 37, pp. 137-148.

Tenney, Y.J., Adams, M.J., Pew, R.W., Huggins, A.W.F., and Rogers, W.H.
(1992, July, 1992). A principled approach to the measurement of
situation awareness in commercial aviation (NASA Contractor Report
4551). Washington, DC: National Aeronautics and Space Administration.