![](media/image1.emf)

[\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}

HUMAN PERFORMANCE\
CONSIDERATIONS\
IN THE USE AND DESIGN OF\
AIRCRAFT CHECKLISTS

JANUARY 1995

[\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_]{.underline}

> Assistant Administrator for System Safety
>
> Office of Safety Services
>
> Safety Analysis Division

CONTENTS

Purpose\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...1

Related Federal Aviation Regulations \...\...\...\...\...\...1

Background\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...1

Definitions\...\...\...\...\...\...\...\...\...\...\...\...\...\.....2

Analysis of Checklist Error Incident Data\...\...\...\.....5

The Need For Checklists\...\...\...\...\...\...\...\...\...\.....7

Human Factors Considerations: How They

Affect Checklist Performance\...\...\...\...\...\...\...\...\...8

Obtaining Checklist Acceptance/Approval:

The Process \...\...\...\...\...\...\...\...\...\...\...\...\...\...16

Checklist Design Considerations\...\...\...\...\...\...\.....17

Methods Of Checklist Design\...\...\...\...\...\...\...\...\...21

FAA Policies and Recommendations For

Managing the Accomplishment of Checklists\...\...\...\....24

Checklist Format\...\...\...\...\...\...\...\...\...\...\...\.....31

Presenting the Checklist\...\...\...\...\...\...\...\...\...\...32

The Cumulative Effect of Improper

Typography \...\...\...\...\...\...\...\...\...\...\...\...\...\....42

Appendix I \...\...\...\...\...\...\...\...\...\...\...\...(5 Pages)

Appendix II \...\...\...\...\...\...\...\...\...\...\.....(2 Pages)

Appendix III \...\...\...\...\...\...\...\...\...\...\....(3 Pages)

List of Figures

Figure 1.(Sample Checklist)\...\...\...\...\...\...\...\...\...38

Figure 2.(Sample Checklist)\...\...\...\...\...\...\...\...\...39

HUMAN PERFORMANCE CONSIDERATIONS IN THE USE

AND DESIGN OF AIRCRAFT CHECKLISTS

1\. [PURPOSE]{.underline}

This report is intended to assist Part 121 and Part 135, operators in
the design, development, and use of aircraft flightdeck checklists, and
to increase awareness of the impact of human performance as it relates
to the use of checklists. In addition, the report presents guidelines
for checklist design.

2\. [RELATED FEDERAL AVIATION REGULATIONS SECTIONS]{.underline}

Part 21.5, Airplane or Rotorcraft Flight Manual

Part 25.1585, Operating Procedures

Part 121.315, Cockpit Check Procedure

Part 121.542, Flight Crewmember Duties

Part 135.83, Operating Information Required

Part 135.100, Flight Crewmember Duties

3\. [BACKGROUND]{.underline}

The small number of accidents that occur each year, when compared to
hours flown, reflects a strong dedication by operators and pilots to
provide an accident free environment. However, as the overall accident
rates have declined over the years and flight safety has shown
significant improvement, the percentage of accidents that result from
flightcrew action or inaction has remained relatively constant in the
majority of fatal air carrier accidents.

A National Transportation Safety Board (NTSB) Safety Study of 37 major
accidents of U.S. air carriers between 1978 and 1990, identified a need
to increase human factors awareness related to checklist usage and
design. The study cited the improper use or the failure to use a
checklist as the probable cause or contributing factor in several
aircraft accidents during the pre-departure and departure phases of
operation.[^1]

Ten of the thirty-seven accidents studied by the NTSB occurred during
the takeoff phase of operation. Their analyses indicate that a
significant number of errors contributing to these accidents were made
during the

preceding taxi phase. Eight (80 percent) of the accidents that occurred
during takeoff involved errors that were classified by the NTSB as
causal; six (60 percent) included procedural errors that were classified
as causal: uninitiated or inadequately performed checklists. These
checklist-related errors resulted in attempted takeoffs with mis-trimmed
control surfaces, flaps and slats not extended for takeoff, incorrect
use of engine anti-ice systems, and locked controls.

An additional review of NTSB accident data, conducted by the FAA\'s
Office of Integrated Safety Analysis, revealed that during the period
1983 to 1993, approximately 279 aircraft accidents occurred where the
checklist was not used or followed during CFR Part 91, 121, and 135
operations. In addition, a small number of accidents involved checklists
that were inadequate for the aircraft or failed to include critical
steps required for safe operation. The 279 accidents were responsible
for approximately 215 fatalities and over 260 injuries.

Analysis of incident reports submitted by flightcrews to the Aviation
Safety Reporting System (ASRS), also indicate that misuse and/or failure
to use the normal checklist is a continuing problem, both for air
carriers and commuter operators during ground operations. Several of the
errors identified by the National Transportation Safety Board (NTSB) as
causal or contributing in previous accidents continue to be identified
by flightcrews as still occurring.

The final analysis of accident and crew reported incidents suggests that
human performance, and its potential to create error, should be given
full consideration throughout the checklist design phase and emphasized
in crew training.

4\. [DEFINITIONS]{.underline}

*Abbreviated Procedure:* A list of sequential procedural steps without
an amplified description or amplified set of instructions.

*Accepted:* Used to describe a document, manual, or checklist which does
not have, or is not required to have, FAA approval.

*Airplane Flight Manual (AFM):* An approved airplane flight manual
prepared by the manufacturer and approved by the FAA Aircraft
Certification Office (ACO) under the provisions of FAR Part 21.5.

*Alternate:* Used to described a procedure or checklist, it refers to a
procedure which may be employed instead of another procedure. Alternate
procedures may either be normal, non normal or abnormal procedures.

*Amplified Procedure:* A description of sequential procedural steps with
detailed explanatory descriptions and/or instructions accompanying each
step.

*Approved:* Used to describe a document, manual, or checklist, it means
that a regulation requires FAA approval and that the FAA has evaluated
and specifically approved the document, manual, or checklist.

*Caution:* An instruction concerning a hazard that if ignored could
result in damage to an aircraft component or system - which would make
continued safe flight improbable.

*Checklist:* A formal list used to identify, schedule, compare, or
verify a group of elements or actions. A checklist is used as a visual
or oral aid that enables the user to enhance short-term human memory.

*Company Flight Manual (CFM):* An approved aircraft flight manual
developed by, or for, a specific operator for a specific aircraft type
and which is approved by the FAA, in accordance with the provisions of
FAR Parts 121.141(b) or 135.81(c).

*Emergency:* When emergency is used to describe a procedure or
checklist, it refers to a non-routine operation in which certain
procedures or actions must be taken to protect the crew and the
passengers, or the aircraft, from a serious hazard or potential hazard.

*High Workload Environment:* Any environment in which multiple demands
on the flightcrew necessitate the prioritizing of work functions. For
example, IFR operations below 10,000 feet during arrival or departure
from a terminal area (including taxiing) are considered to be high
workload environments.

*Immediate Action:* An action that must be taken in response to a
non-routine event so quickly that reference to a checklist is not
practical because of a potential loss of aircraft control,
incapacitation of a crewmember, damage to or loss of an aircraft
component or system, which would make continued safe flight improbable.

*Non normal or Abnormal:* Used to describe a procedure or checklist in
reference to a non-routine operation in which certain procedures or
actions must be taken to maintain an acceptable level of systems
integrity or airworthiness.

*Normal Checklist:* A checklist comprised of all of the phase checklists
used sequentially in routine flight operations.

*Normal:* Used to describe a procedure or checklist in reference to a
routine operation.

*Phase Checklist:* A checklist used to establish and/or verify aircraft
configuration during a specific phase of flight. An example of a phase
checklist is an \"after-takeoff checklist.\"

*Pilot-Flying (PF):* The pilot who is controlling the path of the
aircraft at any given time, whether or not the aircraft is in flight or
on the ground.

*Pilot-Not-Flying (PNF):* The pilot who is not controlling the path of
the aircraft.

*Policy:* A written requirement established by an operator\'s management
which is expected to be complied with by appropriate employee personnel.
A policy may be stated within a procedure or stated separately. A
written requirement such as, \"No aircraft may depart on a cross-country
flight without a spare case of oil\" is an example of policy.

*Procedure:* A logical progression of actions and/or decisions in a
fixed sequence which is prescribed by an operator to achieve a specified
objective.

*Recommendation:* A preferred technique or action described by the
operator which employees are expected to follow whenever practical. A
recommendation is not a policy statement.

*Rotorcraft Flight Manual (RFM):* An approved rotorcraft flight manual
prepared by the manufacturer and approved by the FAA Aircraft
Certification Office (ACO) under the provisions of FAR Part 21.5.

*Supplemental:* Used to describe a procedure or checklist with reference
to a procedure which may be employed in addition to a normal, non
normal, or abnormal procedure. Supplemental procedures may either be
normal or non normal procedures.

*Systems Management:* The management of those systems which sustain the
mechanical functions of the aircraft as opposed to the management of the
aircraft\'s thrust, flight path, or aerodynamic configuration.

*Technique:* A method of accomplishing a procedural step or maneuver.

*Warning:* An instruction about a hazard that, if ignored, could result
in injury, loss of aircraft control, or loss of life.

5\. [ANALYSIS OF CHECKLIST ERROR INCIDENT DATA]{.underline}

The National Transportation Safety Board (NTSB), based on accident
studies, has recommended that more emphasis be placed on checklist
development and use.[^2] A review of incident reports provided by
flightcrews to the Aviation Safety Reporting System (ASRS) operated by
NASA for the FAA, also suggests that more emphasis should to be placed
on the use of checklists.

Other than information obtained during the investigation of an aircraft
accident in which the aircraft was equipped with recording devices, it
is difficult to obtain specific data concerning the commission of human
error. The ASRS has proven to be a valuable source of information that
is being reported by crewmembers that have experienced an incident
involving checklist error. Although the ASRS reports do not provide for
follow-up on specifics, they do provide trend information that is
invaluable in identifying potential problem areas.

A review of approximately 300 randomly selected ASRS \"checklist\"
related reports suggests that many of the same errors that have been
identified by the NTSB as causal or contributing to past accidents
continue to be reported as occurring on the flightdeck.[^3] The
significant areas in which checklist errors have been, and continue to
be identified by flightcrews reporting to ASRS, include the following:

\(a\) Crew failed to use the checklist.

\(b\) Crew overlooked item(s) on the checklist.

\(c\) Crew failed to verify settings visually.

\(d\) Checklist flow was interrupted by outside sources.

\(e\) Operator\'s or aircraft manufacturer\'s checklist contained
error(s) or was incomplete.

It is important to note that the top three items listed previously are
the same items that have been identified by the NTSB as causal or
contributing in several major accidents.

Additional analysis was conducted to determine the extent of checklist
error and its impact on ground operations. Two-hundred randomly selected
ASRS checklist error reports revealed several safety concerns. The
reports were selected based on the use of key words by the reporter and
divided

into two specific groups. One contained reports in which the reporting
crewmember made reference to \"flaps and/or slats\" and \"checklist\",
and the other group of reports contained only general references to
\"checklists\".[^4]

In the \"flap and/or slat\" group, twenty-seven reports provided
specific references to crew checklist performance failure. Eighteen of
these incidents (66 percent), occurred during the taxi/takeoff phase of
operation in which the crew failed to select the correct flap and/or
slat setting required for takeoff. In each report the crew was alerted
through the activation of a flight deck alarm system that the aircraft
was not configured properly. In each reported case, the takeoff was
aborted or the crew delayed on the runway until the aircraft was
properly configured.

In over fifty percent of these incidents, the reporters suggested that
they became complacent. Each reported that the crew could have caught
the error if they had performed a visual verification of flap and/or
slat indicators as part of a last minute scan of critical flight
components before takeoff.

The general reference group identified sixty-one occurrences of failure
to monitor and cross check flight deck activity, misuse or failure to
use checklists, and missed or overlooked items on the checklist
following distraction or interruption. Analysis of the reports indicate
that these occurrences were most common when:

a\. The crew was nearing the end of the work day;

b\. The crew was rushing to make a scheduled departure time; or,

c\. The crew had not completed all checklist items, and did not decline
an air traffic control takeoff clearance because departure traffic was
backed up behind them and they did not want to cause delays.

In several cases, reporters stated that when the crew was rushed, the
checklist was either done from memory; was given a cursory effort; or,
as reflected in nearly 15 percent of the reports, was initiated but
never completed.

In many situations the reporter indicated that they are strong advocates
of checklists and for some unknown reason had strayed from a personal
policy or technique that they had developed to remind them to conduct a
certain function or conduct a visual check for verification. In most of
these reports, the individual was of the opinion that the incident was a
rare occurrence. However, each reporter had something in common with the
previous. All were impacted by one or more human factor(s) and failed to
recognize a deterioration in personal performance.

6\. [THE NEED FOR CHECKLISTS]{.underline}

The majority of the information contained in the following sections are
a compilation of information obtained from reports prepared by the
National Aeronautics and Space Administration (NASA), the FAA, and other
research efforts.[^5] A complete listing of references and other
materials that pertain to the preparation of this document are contained
in Appendix II.

The complexity of today\'s aircraft requires a systematic approach to
operation. The pilot and crew in fact, are an intregal part of an
aircraft system. Like any other complex system, when a system component
fails, the entire system may be subject to failure.

Checklists have been the foundation of pilot standardization and cockpit
safety for years. Such procedures, when applied in a disciplined and
standard manner, are intended to support human performance by providing
a firm foundation for the task, one which the pilot and crew can depend
on during a \"low\" in performance. The checklist is an aid to the
memory and helps to ensure that critical items necessary for the safe
operation of aircraft are not overlooked or forgotten.

However, checklists are of no value if the pilot is not committed to its
use. Without discipline and dedication to using the checklist at the
appropriate times, the odds are on the side of error. Crewmembers who
fail to take the checklist seriously become complacent and the only
thing they can rely on is memory and the fact that not all errors
resulting from poor checklist discipline result in accidents.

Pilots who develop strong cockpit discipline, foster team work, and make
a concerted effort to comply with tried and tested operational
procedures are seldom surprised by an occurrence that was not
anticipated. From a human factors

point of view, the checklist is an important interface between the human
and the aircraft. In addition to assisting the crew to configure and
operate the aircraft properly, the checklist provides a method and a
sequence for verifying the overall system operation. It is an important
aid in helping the crew to remain focused to the task at hand by
eliminating guesswork that often accompanies periods when crew attention
is divided especially during periods of stress or fatigue. The checklist
is an important and necessary backup for the pilot and crew.

A positive attitude must be promoted toward the use of checklists and
each crewmember should consider its importance. The procedures that are
used on the flight deck today are the result of experience, research,
and unfortunately, the findings of causal or contributing factors
gathered from previous accidents or incidents. Many of the procedures
used today were developed and implemented to avoid recurrence of
undesirable events.

Many training programs have been developed and implemented by the FAA
and aircraft operators to continue to reduce the number of aircraft
accidents and incidents. Airline conducted training, e.g., Crew Resource
Management (CRM), Line-Oriented Flight Training (LOFT), training
conducted by professional flight training sources, and others have
played a significant role in increasing the efficiency of flightcrew
personnel\'s performance.

While these programs have contributed to an overall increase in the
level of safety that currently exists in aviation, accident and ASRS
trend information indicate a special need for continued attention to
human factors and their impact on flight operations.

7\. [HUMAN FACTORS CONSIDERATIONS]{.underline}: HOW THEY AFFECT

CHECKLIST PERFORMANCE

In the past, accidents and incidents were frequently attribited to
\"pilot error\" when nothing else could be identified as the cause.
Today, \"pilot error\" is not a sufficiently descriptive explanation of
cause. Humans are very flexible in adapting to changing conditions.
Human performance is variable; it fluctuates and may produce errors
which should be clearly identified and understood.

To reduce human error in aircraft operations, a move was made in the
aviation industry to standardize aircraft controls, instruments,
charting, and other areas as much as possible. This effort, intended to
reduce variance in human behavior also addressed the use of checklists
in aircraft.

While the NTSB and ASRS reports suggest that flightcrews need to place
more emphasis on checklist usage, they also suggest other areas that
require attention with due consideration for human factors. Such factors
as fatigue, crew reliance on working or short term memory, crew
interruption or distraction, and complacency or failure to visually
verify aircraft configuration, are factors that may affect crew
performance and have the potential to cause checklist error.

a\. Fatigue/Stress

The terms \"fatigue\" and \"stress\" are often used interchangeably and
are difficult to define and measure. The degree or extent of
fatigue/stress on the human is individually determined because they
affect each person differently. Within this document fatigue is defined
as a physical condition that is a normal result of the lack of physical
fitness or conditioning; or sleep loss; or missed meals that may cause a
low blood sugar level; or any combination thereof. A condition that can
be minimized by adequate rest and sleep, regular exercise, and
maintaining a proper diet.

Fatigue continues to be one of the prime concerns to flight safety, as
it may not be apparent to a pilot until errors are committed. Many of
the decision errors made in the performance of flying tasks may be
attributed to fatigue.

Although a normal occurrence, fatigue can reduce the coordination and
alertness vital to safe pilot performance. Fatigue is the result of many
influences. Regardless of its cause, the sensation of fatigue should be
a warning of impending task overload and the need for compensatory
measures to avoid that overload. The most common of these compensations
is load shedding or load reduction; a fatigued individual tends to rank
tasks to be accomplished according to their percieved importance and
sheds or deletes those of a lower priority. The individual may also
refuse to accept new tasks or inputs or may devote less time to each of
the present tasks. This can directly impact the successful completion of
checklist tasking.

Stress is a concern as it can impair pilot performance in very subtle
ways. Difficulties in one\'s personal life or at work can occupy the
thought process enough to markedly decrease alertness. Certain
emotionally upsetting events, including a serious argument, death of a
family member or friend, divorce or separation, or financial problems,
can render a pilot unable to operate an aircraft safely.

The emotions of anger, depression, and anxiety from such events not only
decrease alertness, but may also lead to taking risks. Any pilot who
experiences such an emotionally

upsetting event should recognize that the risk of error is increased if
he or she elects to fly under these conditions.

Job related stress can be reduced through proper training and the
development of good pilot operating skill and technique. Aggressive
training programs can increase confidence and decrease on-the-job
stress.

Fatigue and mental stress can become an extremely hazardous combination.
Emotional stress, normal workload requirements, noise, mechanical or ATC
delays, and crew duty limitations, can create an environment conducive
to the load shedding process. This can lead to \"getting behind the
aircraft.\"

Pilots that fail to recognize the onset of fatigue or stress may expose
themselves and others to human error. During these periods, performance
can fall off, judgment can become impaired, and there is an increased
risk that short-cutting procedures, e.g., checklists, may occur.

b\. Interruption/Distraction: Task Management

Reconstruction of arrival and departure scenarios suggest that certain
phases of ground operation and flight have a higher potential for
checklist error than others.

Unless the weather has created a special challenge, the arrival phase is
generally less demanding on the crew. The task loading associated with
the arrival sequence to an airport is generally spread over a longer
time period. It may begin as much as 75 miles or more from the airport
and the cues for initiating the checklist are normally more pronounced,
e.g., descending out of cruise altitude, perform \"DESCENT\" checklist;
at or near the final approach fix complete the \"BEFORE LANDING\"
checks.

Based on a review of NTSB accident reports and ASRS incident reports,
the flight deck crew is most vulnerable to interruption and/or
distraction from the \"BEFORE START\" phase through \"PUSH BACK\",
\"START\", \"TAXI\", and \"BEFORE TAKEOFF\" phases of operation. These
phases of operation can be the most hurried. Often the events that take
place during the pre departure phase of operation do not occur in a
logical sequence, requiring a greater sense of situational awareness and
team work and presenting opportunities for checklist interruption.

Before taxi, numerous activities are in progress in and around the
aircraft. The crew can be distracted or interrupted by boarding
passengers, baggage loading, fuelers, push-back crews, mechanics, and
flight attendants. All are necessary functions that have the potential
to cause distraction or interruption, but are necessary activities

before the aircraft can depart. The crew has little control over these
activities and occasionally the checklist process is dependent on
external activity being completed, e.g., waiting for cargo doors to be
closed or the arrival of fuel, etc.

While the crew is waiting for these activities to be completed, a
\"hold\" may be placed on the checklist process. When this occurs the
crew should have a provision for retaining the hold point\--a memory
jogger to remind them that the checklist has not been completed.

It is recommended that anytime a checklist flow has been interrupted or
an item placed on hold, the checklist should not be stowed. It should be
kept in hand. If it must be laid down to allow the crewmember to perform
other functions it should be placed in a conspicuous area visible to all
crewmembers, e.g., in the throttle/power quadrant area or clipped to the
control yoke as a reminder that the list has not been completed. Other
methods, e.g., hand written notes placed in the power quadrant area, the
placing of a clean, empty styrofoam coffee cup inverted over a power
lever or something as simple as placing a paper clip along the border of
the list at the appropriate spot can serve as effective reminders.
Reportedly, each of these methods has been effective as reminders and
memory joggers with other flightcrew personnel. Whatever method works
for the individual crewmember is acceptable as long as that person has a
method for memory recall. If there is doubt as to where the flow was
interrupted the chance of error is reduced if the crew returns to the
beginning of the task list being performed.

A key to help reduce being side tracked from checklist duties following
a checklist hold, interruption, or distraction is not to stow the
checklist until all items in the flow are complete. Several researchers
are in agreement, from a human factors perspective, that if the
checklist is interrupted or placed on hold, returning it to its normal
storage place increases the possibility that the crewmember may forget
to resume that portion of the list.

Many of the distractions or interruptions occurring on the ramp area can
be reduced to a minimum by the aircraft operators through training of
support personnel. Operators should ensure that company ground support
personnel who communicate directly with flightcrews are familiar with
the procedures used on the flight deck and the need to avoid
interrupting the crew during a checklist flow. Persons entering the
flight deck to talk to the crew should make their presence known and
unless an emergency exists, refrain from interrupting any flight deck
activity or talking to the crew until the crew indicates that they have
completed their task and acknowledges their presence.

Additionally, all cabin crewmembers should be fully briefed on the
contents of FAR, Part 121.542, Flight Crewmember Duties, or Part
135.100, Flight Crewmember Duties, (as appropriate) more commonly
referred to as the \"sterile cockpit\" rule. As a minimum the briefing
should include the reason for the rule, and the potential impact on
safety should the cabin crew interrupt or distract any member of the
flight deck crew during checklist or other essential activity. At the
same time the operator should exercise care to ensure that the cabin
crew members are not reluctant to talk to the cockpit when issues that
they believe may be related to the safe operation of the aircraft are
detected.

As the crew begins the \"Push Back\", \"Engine Start\" and \"Taxi\"
phases a new set of potential distractions and interruptions enter the
picture, particularly as the aircraft enters the airport operational
area. The cockpit crew members may be forced to divide their attention
among completing cockpit duties (checklists), the monitor of air traffic
control, observing and avoiding other taxiing aircraft, ground vehicles,
and listening for the the company dispatch who may be providing new load
or weather data.

Throughout the \"TAXI\" phase of operation the potential for error can
be significant. Depending on when and where it is performed, the
checklist flow can interrupt or distract the crew from other cockpit
duties, i.e., the monitoring and attention required for external
operations. Simultaneously, these outside activities can interrupt
checklist use. This can easily force the flight deck crew into
functioning as individuals in the cockpit instead of a team. Crews
should be cautioned that when the continuity of performing monitor and
cross checks is interrupted, the safety margin may also be reduced.

A recent report published by NASA, concerning checklists suggests that
\"TAXI\" checklists should be completed as close as possible to the gate
and as far away as possible from the active runways and adjacent
taxiways.[^6]

c\. Working Memory

Research indicates that the capacity of working memory is limited.
Unaided, working memory can retain approximately seven (plus or minus
two) unrelated items. Unless actively rehearsed, or aided by some
external form of reminder or memory jogger, information contained in the
working memory will generally be forgotten in 10 to 20 seconds.[^7]

Depending on how incoming information is received, (tactile, visual,
etc.) some information may be retained for a longer duration than other
information.

Interference is the principal cause of loss of information from the
working memory.[^8] Interference can be defined as noise, incoming
verbal messages or other information, e.g., communication with ATC or
company sources, and an interruption and/or distraction. Due to
interference, information that has been stored in the working memory
either becomes forgotten or is replaced by new information. In addition,
an individual\'s emotional state can negatively impact the ability to
retain information. Many of the stress related emotions, e.g., panic,
anxiety, confusion, or frustration, can negatively impact an
individual\'s ability to maintain information in the short term memory.
Because of working memory\'s short duration and limited capacity, pilots
should develop their own system of memory joggers.

It is recommended that anytime the crew is not clear as to their
progress through the checklist the captain or pilot-in-command should,
without hesitation, direct that the appropriate section of the checklist
be re-accomplished from the beginning.

d\. Pressure On-The-Job

Pressure for \"on time performance\" can be a factor that may carry over
into the cockpit and affect flight deck activity. The NTSB reported in
its January 1994, safety study that seventeen or 55 percent of 31 air
carrier accidents for which flight time information was available had
departed late or were operating behind schedule before the accident. In
contrast, between 17 and 35 percent of an illustrative sample of
non-accident flights were running late.

Pilots should be cautioned that as pressure mounts and they allow the
pace to quicken to make an on time departure they may be increasing the
possibility of error. Laboratory research has demonstrated that there is
a very definable relationship between response-time and error-rate[^9].
This may have an effect on checklist performance and the relationship
between the speed of performing the checklist and the quality (accuracy)
of the check. For example: if the pilot scans the appropriate control
and instrument panels rapidly because of time pressure, the accuracy of
his or her perception may suffer and the probability of error will
increase.

Overall, the study of crew reported incidents and NTSB accident data
suggest that ground operations between the

parking area/terminal and the runway areas require strict attention to
duties. Even taxiing to the parking area after landing has been
identified as an area in which the checklist can distract from crew
duties. Several incidents have been reported in which the crew taxied
across an active runway or initiated a wrong turn while other
crewmembers were performing \"After Landing\" checklist tasks.

e\. Cues For Initiating Checklists

Although it is normally the Captain or Pilot-in-command\'s
responsibility to call for the checklist, in the absence of specific
operator guidance, each crewmember should develop their own internal and
external cues as to when the checklist should be initiated. For example;
the \"Before Start\" checklist can be cued by the completion of
passenger loading (internal cue); the \"Taxi\" checklist can be cued by
receipt of a taxi clearance and the \"Before Takeoff\" checklist can be
cued by reaching the hold line associated with the departure runway.

For economic considerations and to ensure minimum fuel consumption
before take off, aircraft operators or pilots may elect to delay
starting all engines until the aircraft has reached a point on the taxi
route. Delayed engine starts may be cued based on taxi progress
(external cue) to the runway.

In the absence of specific company policy or procedure, cueing the
checklist is a personal technique. It can fail if a crewmembers is
preoccupied with other tasks. However, if each crewmember develops his
or her own cues for checklist initiation, the chances that the cues will
be identical for each crewmember is remote. Such a system provides a
means of backing each other up in the cockpit.

Of equal importance and directly related to \"cueing\" is the timing of
tasks performed on the flight deck. Researchers have referred to this
period as the \"Window of Opportunity\", indicating the time period
which a task can take place. For example, the window of opportunity for
the DESCENT checklist may be defined as the time period between leaving
the cruise altitude and arriving at 10,000 feet, allowing for variations
based on vectors, restrictions, etc.

Although a given task can be effectively accomplished at any time within
the window, it appears that there is an advantage to conducting the task
early. Researchers conducted a study to evaluate task-scheduling
strategies of airline pilots flying DC-9 and MD-88 aircraft in a full
mission simulation. They reported that crews who scheduled their task
early within the window tended to be rated as high performing crews.
Conversely, crews who scheduled their tasks late within the window
tended to be low performing crews.

The research concluded that \"scheduling a task early in the window of
opportunity is the optimal task scheduling strategy\".

In summary, a well managed crew schedules the required tasks within a
window of opportunity in a way that it will not be done too early or too
late. For example, if one wishes to obtain the arrival ATIS, there is no
point in doing this task too early; the information may change by the
time the aircraft will start the approach, particularly during rapid
changing weather. On the other hand, there may be penalties for
obtaining the ATIS information so late, since it is required for
planning purposes.[^10]

f\. Silent Checklists

Although small in number, several ASRS reports address the use of silent
checklist procedures in which the pilot-not-flying (PNF) performed the
checklist improperly or not at all.

Silent checklists quite often are performed during heavy workload
periods, e.g., during the after takeoff climb or during the after
landing taxi-in. The use of silent checklist procedures has both
advantages and disadvantages. The procedure when performed by the PNF
reduces the amount of activity on the flight deck that the pilot-flying
(PF) normally has to contend with and allows the pilot to concentrate
more on flying the aircraft. Conversely, silent checklists do not
provide for the cross check and monitor that should take place between
crewmembers.

In all cases, the crewmember calling the checklist, normally the PNF,
should announce when all checklist items have been accomplished, e.g.,
\"\_\_\_\_\_\_\_ Checklist complete.\" This informs other crewmembers
that the task has been completed and provides the opportunity for them
to perform a visual cross check.

g\. Aircraft Maintenance

Several reports have been received by the ASRS in which the reporter
indicated that while on a stop-over, maintenance was performed on the
aircraft that required hands-on work on the flight deck. The reporters
revealed that work performed, normally while trouble shooting, required
the changing of switch positions, instrument control settings,
deactivation of circuit breakers etc., without the crew\'s knowledge. In
each reported instance the flightcrew was not present in the cockpit
when the changes were made and ,reportedly the crew was not advised of
any changes.

In all cases, the items affected were contained on the aircraft
acceptance checklist for the first flight of the day, but they were not
listed on the normal stop-over checklist. Consequently, the discrepancy
was not detected until later in the flight.

To avoid the possibility of a mishap resulting from hands-on work
performed on the flight deck while the crew is not present, each
operator should develop a procedure requiring maintenance personnel to
notify the crew that work has been performed on the flightdeck.
Notification can be either through direct communication with the crew or
by using some type of highly visible posting method affixed directly to
the

control yoke or over the face of a primary cockpit instrument.

h\. Personalizing the Checklist

Clear and concise communication in the cockpit is essential and reduces
the chance of mis-communication between crewmembers. Every effort should
be made to avoid substitution of self devised terms for checklist terms,
e.g., calling for \"Boost Pumps\" when the checklist calls for \"Fuel
Pumps\". The use of non-standard terms can be the cause for another
crewmember\'s failure to detect a checklist error or, may cause another
crewmember to not be able to follow the checklist sequence, or cause the
checklist callout to be confused with other intra-cockpit
communication.[^11]

Such communication circumvents standardization and has been linked in
studies as one of the behavioral attributes frequently found in
association with information transfer problems.

Strict use of the terms presented on the checklist reduces the chance
for misunderstanding of the task to be performed and its status. Any
attempt on the part of a crewmember to personalize the checklist erodes
the safety margin established by the procedure.

8\. [OBTAINING CHECKLIST ACCEPTANCE/APPROVAL]{.underline}: THE PROCESS

The process for developing a crew checklist begins with the manufacturer
who designs, builds, and tests the aircraft. Recommended operating
procedures and related checklists are developed and tested by the
manufacturer and published in the Aircraft Flight Manual (AFM) under the
provisions of Title 14 Code of Federal Regulations (CFR), Part 25.
Certification of the aircraft under Part 25, only

allows that the aircraft was manufactured and tested to certain
specifications and is ready to be operated.

Before the aircraft can be operated under either 14 CFR

Part 121 or Part 135, the operator must comply with all of the
applicable sections of the appropriate regulation. The development of a
flightcrew checklist is a provision that must be complied with by all
Part 121 and 135 operators. In addition to development of crew
checklists the operator must obtain approval or acceptance from the
FAA\'s Principal Operations Inspector (POI) before it can be used.

The POI is a representative of the FAA assigned to each operator. The
POI and staff are available to assist and advise the operator on matters
concerning the safe operation of aircraft. In addition, they provide
regulatory oversight from the initial operator certification process
through all phases of operation, including flightcrew training,
compliance with company operational policies, maintenance and inspection
of aircraft, and the development or approval of all flightcrew
checklists.

Although the manufacturer may have provided a checklist in the AFM, it
was not approved for use during the Part 25 certification process. When
a Part 121 operator proposes to use an AFM checklist, the POI must
review and approve that checklist. The same applies when a Part 135
operator proposes to use an AFM checklist, however; the POI need only
review the checklist and determine that it is acceptable for that
operators use.

Only after the operator has demonstrated compliance with the
regulations, will operating approval be issued by the FAA.

The final approval for an operator developed crew checklist and future
changes or modifications to that checklist require the approval or
acceptance by the FAA.

9\. [CHECKLIST DESIGN CONSIDERATIONS]{.underline}

Although it may be published in a manual, a checklist is designed for
independent use so that the user does not have to reference a manual.
Checklists are used to ensure that a particular series of specified
actions or procedures are accomplished in correct sequence and to verify
that the correct configuration has been established in specified phases
of flight.

If the its design presents the crew with a challenge or obstacle to
complete, and becomes work intensive, the checklist can set the stage
for error. When the checklist is lengthy, there is a tendency to perform
the items while reading the checklist in an effort to overcome a
time-consuming

procedure.[^12] This method of short-cutting can cause the crew to lose
the redundancy imbedded in the checklist. While such short-cutting may
not always be related to configuration items, it can easily migrate to
items that are critical to the safe operation of the aircraft.

Other pilots may deviate from accepted methods (primarily
challenge-and-response) of conducting checklists to what they consider a
faster method. A technique observed by researchers has been for one
pilot to call several challenge items together while the other pilot
replies with a series of chunked responses.[^13] This undermines the
concept behind the step-by-step process set in the challenge-response
method. This method is dependent on the pilot\'s short and long term
memory as to the order and completion of the checklist; this is exactly
what the checklist is supposed to prevent.

In addition, if the established flow patterns are not logical and the
checklist itself correct and consistent with procedures prescribed in
related manuals, the probability is very high that the crew may, when
pressed for time, revert to their own methods, cut corners, omit items,
or even worse, ignore the checklist entirely.

A new checklist design alone will not elimate the problems associated
with checklist error. Proper consideration must be given to the task,
the environment in which it is conducted, crew workload at the time the
action is called for, and human performance capability. When all of
these factors are properly taken into consideration along with the
technical and operational issues, the checklist can be an effective tool
and, under certain conditions, can reduce pilot workload.

When commissions of error in checklist usage are detected, it can be of
value for the company and the crew to analyze the error. If the error
rate is to be reduced, it is important to identify the causal or
contributing factors that led to the commission of the error. It is
appropriate to review the crew\'s activities at the time the error was
committed. In addition, review the company policies and/or procedures
requiring the task, and the overall design and placement of items in the
checklist. The objective of the review would be to reevaluate the
procedural requirement to determine if it contributed to the error.

Checklists conducted during periods of heavy workload are more subject
to error. A company required procedure, though needed, may be ill
placed, ill timed, or be so cumbersome that when interjected into a

heavy workload environment, e.g., during ground taxi operation, it may
potentially become a distraction.

Checklists that are easy to read and use are more resistant to error and
will contribute less to cockpit workload than those that are not. A
correction to a specific checklist problem may lie somewhere in the way
that the company develops and designs a checklist and/or provides
operational information in manuals.

b\. Checklist Content

Traditionally, aircraft checklists have been divided into three
categories. These are referred to as \"Normal\",

\"Nonnormal\", or \"Abnormal\", and \"Emergency\". In some instances,
these categories may be further divided into subcategories such as
\"Alternate\" and \"Supplemental\".

The \"Normal\" checklist is typically a listing of action items to be
performed and verified at a particular point in a routine operation
(without malfunctions). It is comprised of all the phases of flight and
used sequentially in routine flight operations.

The \"Nonnormal\" or \"Abnormal\" checklist is developed and used for
non-routine operations in which certain procedures or actions must be
taken to maintain an acceptable level of systems integrity or
airworthiness;

The \"Emergency\" checklist is developed and used for non-routine
operations in which certain procedures or actions must be taken to
protect the crew and the passengers, or the aircraft, from a serious or
potential hazard.

Operators should ensure aircraft checklists are limited to action items
or verification items. The checklist should not contain elaboration or
explanation of procedures. During the development of a checklist the
operator should consider the following:

\(1\) A normal checklist is typically a listing of action items to be
performed and verified at a particular point in the flight. Normal
checklist items do not necessarily represent a procedural step and may
even represent completion of a entire procedure. For example, the item
\"Gear; Up and Locked\" could indicate the gear handle has been raised,
the gear indications checked, the gear handle has been placed in the
neutral position to check the up-locks, and that the handle has been
returned to the up position.

\(2\) Non normal and emergency checklists should contain each sequential
step of a procedure.

c\. Criticality of Checklist Items

Very little data are available concerning placement priority of items on
the checklist. However, some researchers agree that the more critical an
item, the closer it should be to the top of the list. Their rationale is
that critical checks should be completed earlier in the ground phase in
order to decouple the critical items from the takeoff segment as well as
to allow enough time (buffers) for the crew to detect and recover from a
configuration failure.[^14] In most cases, the captain or
pilot-in-command will call for the checklist when the workload is less
than peak. The further the crew progresses through the checklist, the
greater the possibility becomes for interruption.

When possible, checklist items should be ranked in criticality according
to the potential effect of a crewmember\'s failing to perform the
action. Critical items are those items which, if not correctly
performed, have a direct, adverse effect on safety.

An item may be considered \"critical\" on one checklist but
\"non-critical\" on another. For example, a flightcrews\' failure to set
the flaps while accomplishing the \"Before Takeoff\" checklist was the
cause of a major accident. However, a crews failure to retract the flaps
while performing the \"After Landing\" checklist may have little effect
on safety.

The operator should analyze each phase of flight to identify critical
items for that phase and to ensure that all critical items are included
on the checklist.

d\. Diversion of the Flightcrew\'s Attention

The flightcrew\'s attention is often diverted from other tasks when
performing a checklist. In order to minimize a head down posture and
diversion time in the cockpit, the checklist should be kept as short as
practical.

As items are added to the checklist the potential for interruption or
diversion increase. Operators should compare the benefit of adding
additional items to the possible adverse effects this may have.

Normally, the FAA will withhold approval or acceptance of a checklist
that contains items not associated with aircraft operations.

e\. Aircraft Sophistication and Checklists

The degree of technological sophistication in the design of aircraft
directly affects the checklist. In older aircraft, the flightcrew must
manually select and monitor most items. In more advanced aircraft the
same items are accomplished and monitored by automatic systems which
relieve the flightcrew of these tasks. Checklists for these aircraft
tend to be shorter and simpler and may require a more careful task
analysis for the operational requirements of the specific aircraft.

f\. Fleet Standardization

Operators should standardize checklist items and the sequence of items
to the extent allowed by individual aircraft differences across all
aircraft in the fleet. Checklists for technologically sophisticated
aircraft are typically shorter and simpler than those for older
aircraft. The items on checklists for advanced aircraft, however, are
normally present on checklists for aircraft with older technology.

It is FAA policy to require operators to evaluate the feasibility of
placing common checklist items on checklists with standard titles for
all aircraft (such as before start, before takeoff, or before landing
checklists). Items should appear in a standard sequence to the degree
possible. The FAA will not normally approve placing an item on a
checklist which is not required for a specific aircraft solely because
the item is required in other aircraft of the fleet. Exceptions may be
made if the operator provides adequate justification.

10\. [METHODS OF CHECKLIST DESIGN]{.underline}

The most frequently used method of designing checklists for aircraft
with two or more persons assigned to the crew are the
\"challenge-do-verify\" (CDV). CDV is often referred to as the
challenge-response method. Another method is the \"do-verify\" (DV)
method,

The CDV method consists of a crewmember making a challenge before an
action is initiated, taking the action, and then visually and verbally,
verifying that the action has been accomplished. The CDV method is most
effective when one crewmember issues the challenge and the second
crewmember takes the action and responds to the first crewmember, who
monitors the action and verifies that the correct action was taken. This
method requires that the checklist be accomplished methodically, one
item at a time, in an unvarying sequence.

The primary advantage of the CDV method is the deliberate and systematic
manner in which each action item must be accomplished. The CDV method
keeps all crewmembers involved (in the loop), provides for concurrence
from a second crewmember before an action is taken, and provides
positive confirmation that the action was accomplished.

The DV method consists of the checklist being accomplished in a variable
sequence without a preliminary challenge. After all the items on the
checklist have been completed, the checklist is then read again while
each item is verified. The DV method allows the flightcrew to use flow
patterns from memory to accomplish a series of actions quickly. Each
individual crewmember can work independently which helps balance the
workload between crewmembers. However, this method has a higher inherent
risk of an item on the checklist being missed than does the CDV method
and is not recommended over the CVD method.

a\. Selection of Design Method

Both the CDV and the DV methods are currently being used for normal
checklists. Traditionally, operators have preferred the DV method for
normal checklists and the CDV method for non normal and emergency
checklists.

In most circumstances non normal and emergency checklists are more
effective when the CDV method is used. The correct accomplishment of the
actions and procedures incorporated in the non normal and emergency
checklist categories is critical and warrants an error free approach.
Since these checklists are seldom used, crewmembers are usually not as
familiar with the procedures contained in them. In addition, most non
normal and emergency checklists do not lend themselves to developing
flow patterns which crewmembers can readily recall. The CDV method
enforces crew coordination, cross-checking, and verification, all of
which aid the crewmember in overcoming the effects of stress.

Generally, the FAA will not approve or accept the DV method for non
normal or emergency procedures unless the operator can provide
substantial evidence that the method is effective for this application.

b\. Mechanical or Electronic Checklists

These devices differ in format from paper, hand-held checklists, but not
in the design method or use. The CDV and DV methods addressed in this
document can be applied to any type of checklist. Operators are
encouraged to use such devices.

c\. Verification

During the design of the checklist it is important to keep in mind that
all checklist designs are subject to human error. Crewmembers may omit
and skip checklist items or erroneously respond to a checklist at times
believing that an item or a task was accomplished when it was not. At
other times, crewmembers may see what they expect to see rather than
what is actually accomplished or indicated.

One strategy that helps to overcome error is to develop policies for
using checklists which require stringent cross-checking and verification
and reinforce those policies through crew training programs. Again, the
procedures intended for checklist use should be clearly written and
placed in the company\'s operating manual. This should provide clear
direction to each crewmember as to who is responsible for the completion
of specific tasks contained on the checklist, who initiates the
challenges and who responds. The operator\'s policy concerning
verification must be compatible with the operator\'s crew resource
management philosophy.

One of several recognized methods for reducing error and enhancing
verification during a checklist flow is a procedure that requires the
use of aural, visual, and tactile sensors. Announcing the checklist item
out loud (the challenge) stimulates the sense of hearing and helps focus
attention on the task. The pilot-in-command responds by visually
checking each item then actually touching (visual and tactile),
operating, or setting the control or device and announcing (the
response) the instrument reading or prescribed control position in
question. The crewmember calling the challenge monitors and verifies the
actions.

Touching the controls and displays is an effective enhancement for the
verification process. The use of the hand to guide the eye while using
the flow pattern can substantially aid the checklist procedure by
combining the mental sequencing process with motor movements.
Furthermore, the use of the hand and finger to direct the eye to an
alphanumeric display or control can aid in fixating the eyes on the
specific item and prevent the eyes from wandering away from that
indicator.[^15]

11\. [FAA POLICIES AND RECOMMENDATIONS FOR MANAGING THE]{.underline}
[ACCOMPLISHMENT OF CHECKLISTS]{.underline}[^16]

The FAA\'s policy requires each operator to provide specific crewmember
responsibility for monitoring, verifying, and managing the
accomplishment of checklists. These responsibilities may appear as
policy statements or as specific directives and must be contained in the
appropriate operator\'s manual.

a\. Objective of Policy Statements and Directives

The primary objective of the operator\'s policy statements or directives
is to assist in the standardization of crewmember interaction. These
statements should include, but not be limited to, the following items:

\(1\) Flightcrew responsibilities for maintaining aircraft control,
analyzing situations, and for requesting the appropriate checklist in
normal and emergency situations.

\(2\) The specified crewmember responsible for initiating each
checklist.

\(3\) The specified time when each checklist is to be initiated.

\(4\) The specified crewmember responsible for accomplishing each item
on the checklist.

\(5\) The specified crewmember responsible for ensuring that each
checklist is completed and for reporting that completion to the crew.

\(6\) Crewmember responsibilities for bringing to the attention of the
pilot-in-command and the rest of the crew any observed deviation from
prescribed procedures.

b\. Methods for Managing Checklist Accomplishment

The information contained in this section provides a discussion of
recommended methods that may be used for managing checklist
accomplishment. These methods are not all-inclusive nor should they be
interpreted as the only methods acceptable to the FAA.

\(1\) For single-pilot aircraft it is recommended that operators mount
the before-takeoff and the before landing checklists on the instrument
panel by means of a placard. When aircraft characteristics allow, the
operator should develop touch-verification procedures which contain a
requirement that the pilot touch each control to verify it is in the
correct position.

\(2\) For two-pilot aircraft in which only the pilot-in-command has
ground steering control, the recommended method for accomplishing
checklists is for the second-in-command (SIC) to read all checklists.
The recommended method for those aircraft in which either pilot can
steer on the ground is for the pilot-not-flying (PNF) to read all
checklists. In all two-pilot aircraft, the PNF should read all
checklists.

\(3\) For three-crewmember aircraft the recommended method is for the
SIC to read the flight engineer (FE) portion of the before-engine-start
checklist, so that the PIC can observe and verify the configuration of
the flight engineer panel as the FE responds to each item on the
checklist. Since the PNF is the crewmember most subject to interruptions
from radio communications, it is recommended that the FE should read all
normal checklists and verify that each pilot action has been taken when
the aircraft is in motion. The FE should have the explicit task of
verifying that critical items have been performed by the pilots, whether
or not the FE has verbal responses for those items. In those non normal
or emergency situations which involve significant activity by the FE, it
is recommended that the PNF read the checklist and verify FE actions
while the FE performs and responds to the items.

\(4\) For all aircraft, the crewmember responsible for reading the
checklist should be responsible for ensuring that the checklist is
completed systematically and expeditiously. This crewmember should be
responsible for managing interruptions, cross-checking controls and
indicators to ensure that the required actions have been accomplished,
and for reporting that the checklist has been completed.

\(5\) The pilot-flying (PF) should not be distracted from controlling
the aircraft to perform a checklist items that another crewmember can
accomplish. The PF should activate only those switches or controls
(other than the manual or automatic flight controls, throttles, and nose
wheel steering) that are not within practical reach of another
crewmember. Only one pilot should be in a head-down posture at any time.

\(6\) In the prestart phase, flight guidance and navigation checklist
items are considered critical items. A response should be required from
both pilots (and FE if applicable) when the same setting is required for
more than one device (such as computers, flight instruments, and
altimeters).

Inertial platform alignment and computer programming should be
accomplished by one crewmember and independently confirmed by another.
As many of these checklist items as possible should be accomplished and
verified before the aircraft is moved.

\(7\) In the taxi and pre takeoff phases, aircraft configuration (such
as flaps, trim, and speed brakes) and flight guidance items (such as
heading, flight-director, altitude select panel settings, and airspeed
bugs) are critical. All flightcrew members should be required to respond
to applicable checklist items.

\(8\) On approach, flight guidance checklist items are critical. At
least two crewmembers should be required to confirm and respond to these
items. A response should be required when the same setting is required
on two separate devices (such as computers, flight instruments, or
altimeters).

\(9\) All checklist items that are critical in the before-landing phase
vary with the type aircraft involved. The landing gear and flaps are
critical items and should require a confirmation and response by both
pilots.

\(10\) All checklists, except the after-takeoff and after-landing
checklists, should be accomplished by one crewmember reading the
checklist items and a second crewmember confirming and responding to
each item. All critical items on the before-takeoff and before-landing
checklists should require confirmation and response by at least two
crewmembers.

\(11\) All checklists must be designed so that the flightcrew can
maintain an adequate scan and monitor air traffic control communications
while simultaneously controlling the aircraft. The recommended method is
for the operator to group the systems management checklist items after
the configuration, thrust, and flight guidance items for each phase of
flight. When systems management items must be accomplished in a high
workload environment, it is recommended they be accomplished by a single
crewmember. Usually the after-take off and after-landing non-critical
items can be accomplished silently.

\(12\) Operators should direct crewmembers to refrain from In

addition, crewmembers should be directed that when they

observe that another crewmember is not taking or has not taken a
required action, they must be required to inform the crewmember, the
pilot-in-command, or the whole crew, as appropriate.

\(13\) Checklists should not be depended on to initiate changes in
aircraft configuration. Operators should key aircraft configuration
changes to specific operational events. For example, direct the landing
gear to be extended at glide slope intercept. For any adjustment of
configuration, a command from the PF and an acknowledgment from the
crewmember taking the action is required.

\(14\) Crewmembers frequently cannot complete a checklist because of
interruption or an item on the checklist has not yet been accomplished.
Operators are required to develop policies for the management of these
situations. For short delays, the recommended policy is for the
flightcrew to hold the checklist until the interruption is over and the
item can be completed. When the checklist item is completed, the
challenge should be repeated, the proper response given, and the
checklist resumed.

When mechanical or electronic devices allow checklist items to be
accomplished in a random sequence an operator should develop a policy
appropriate to the system used.

Operator policies that allow flightcrews to skip checklist items that
have not been completed and then depend on memory to accomplish them
later are not considered for acceptance or approval.

\(15\) Operating procedures must be established to ensure that the
correct checklist sequence is re-established when unusual events
interrupt the normal sequence of a flight. For example, crewmember
actions during normal sequences of flight are interrupted on taxi-out.
In such a case, operators should require that the flightcrew return to
an earlier point on the checklist and reaccomplish the checklist.

c\. Development and Sequencing of Checklist Items

Operators should ensure that checklists are developed from a careful
task analysis and are consistent with the procedures section of the
operators flight manual. Phase checklist items must be in an appropriate
and logical sequence. When a checklist represents an abbreviated
procedure, that checklist must follow the procedural sequence.

The following guidelines are used by FAA personnel to evaluate
individual topics of checklist design.

\(1\) Operators should standardize the sequence of checklist items as
much as possible across aircraft types.

\(2\) When the operator has a choice as to where an item should be
placed on a checklist, it should be placed at a point where the crew
workload is lowest.

\(3\) Operators should keep checklists as short as possible to minimize
interruptions.

\(a\) Operators should sequence checklist items to minimize
interruptions of checklist accomplishment. For example, sequencing the
\"INS NAV MODE\" as the first item on the engine-start checklist may
allow the flightcrew to call for and complete the before-engine start
checklist at a convenient time even though INS alignment is not
complete.

\(b\) Two short checklists may be preferable to a single long one.
Operators may place a line or otherwise mark a checklist where the
checklist can be held until a specific event occurs. This practice in
essence creates two separate checklists.

\(4\) Operators must include required preflight tests on checklists but
should design checklists to preclude the unnecessary testing of systems.

\(a\) Warning systems with built-in test and automatic monitor circuits
do not need to be checked or included on checklists unless required by
the AFM or RFM.

\(b\) Many test switches in the cockpit are designed for use by
maintenance personnel. Operators should not require flightcrew members
to perform these tests as a normal procedure.

\(c\) The grouping of required functional checks on a specific checklist
which is performed before the first flight of the day, or at some other
logical interval, and not repeated on subsequent flights may be approved
on a case by case basis.

\(5\) Operators must clearly identify decision points and indicate the
correct alternative action to be taken or alternative sequence of
actions to be taken after each decision point. If the effect of adverse
weather requires an alternate action, the operator should design the
checklist to account for that alternate action. For example, if the auto
throttles are normally engaged for takeoff except when engine anti-ice
is being used, the checklist should contain a requirement that the
throttles cannot be engaged with the engine anti-ice on.

d\. Immediate Action Items

Immediate action items are those items accomplished from memory by
crewmembers in emergency situations before the checklist is called for
and read.

\(1\) A flightcrew\'s failure to correctly accomplish all immediate
action items can result in a threat to continued safe flight. For
example, should a crew fail to close the fuel tank valve during an
engine fire procedure, leaking fuel into the engine pylon may be
ignited. In such cases, the first items on the corresponding checklist
must be a verification that each immediate action item has been
accomplished.

\(2\) In some cases, an immediate action procedure may not be
incorporated in a checklist. For example, there is no point in verifying
that each item of an aborted takeoff procedure has been accomplished
after the aircraft has been brought to a stop. In most cases, however,
there should be a \"follow-on\" or \"clean-up\" checklist to be
accomplished after the situation has been brought under control.

\(3\) Immediate actions may be stated as policies rather than as
checklist items when appropriate. An example of an immediate action item
that can be stated as a policy rather than as a checklist item is the
following statement: \"All flightcrew members shall immediately don
oxygen masks and report to the captain on interphone in the event of
loss of cabin pressure.\" In this example the loss-of-cabin-pressure
checklist would contain subsequent items based on the assumption that
the flightcrew is on oxygen and has established interphone
communications.

e\. Checklist Terminology

Operators should ensure that their aircraft checklists contain
terminology that is tightly controlled to ensure clarity and common
understanding. The following recommendations should be considered when
developing checklists.

\(1\) The challenges and responses on the checklist should be consistent
with the labeling on the switches and controls in the cockpit.

\(2\) Terms such as \"Tested\", \"Checked\", and \"Set\" are acceptable
terms only when they are clearly defined and consistently used.

\(3\) Operators should have a consistent policy concerning responses to
items with variable settings. \"As required\" may be printed on the
checklist but should not be an authorized response. A response that
gives the actual

setting is appropriate. Items which require variable responses should be
carefully evaluated. Such items may not actually be required on the
checklist or may be more appropriately included in the system management
portion of a checklist.

\(4\) Responses to checklist items concerning liquid or fuel quantities
should be made in terms of the actual quantities on board compared to
the specific quantity required, for example: \"10,000 pounds required,
10,400 on board.\" When specific quantities are required, a response of
\"checked\" is not acceptable. A response of \"checked\" is acceptable
when a range of quantity is permitted and the range is marked on an
indicator, such as a green arc on an oil quantity gage.

\(5\) Excess verbiage on checklists is discouraged. For example, a
checklist item of \"Reduce airspeed to 130 KIAS for best glide\" can be
abbreviated as:

\"BEST GLIDE - 130 KIAS.\"

\(6\) Ambiguous verbiage on checklists should be avoided. For example,
\"takeoff power\" can mean either advance the power or to retard the
power.

f\. Aircraft Differences

Operators are required, by the FAA, to account in the aircraft checklist
for differences in various series of aircraft or in installed equipment.
When there are only a few minor differences, this may be accomplished by
using symbols to designate those checklist items that apply to only one
series of airplanes or that apply only when the equipment is installed.
When there are a significant number of differences, operators should
prepare separate checklists for each series of aircraft. Policies and
procedures should be established to account for differences in checklist
responses when operations are conducted with equipment removed or
inoperative, in accordance with minimum equipment lists (MELs).

g\. Sequencing Normal Checklists and Other Checklists

Normal checklist items may be incorporated in non normal or emergency
checklists to simplify cockpit management. An acceptable alternative
method is to require both the normal and non normal or emergency
checklists to be accomplished in a specified sequence. This method has
the advantage of allowing the normal checklist to be requested and
accomplished at the time that it would normally be accomplished.

Checklists should be designed so that two checklists are not in progress
simultaneously. The method may depend on the degree of sophistication of
the airplane involved.

In technologically-advanced aircraft with short, simple checklists, it
is usually preferable to keep the normal and the non normal checklists
separate. In older airplanes, however, it may be necessary to add the
normal checklist items to the non normal or emergency checklist simply
to keep the checklist manageable.

12\. [CHECKLIST FORMAT]{.underline}

If crew checklist performance is expected to be error free it is
necessary to build the checklist around known human capability and
limitations. Although research on the building of a checklist is
limited, there are several known factors that must be taken into
account. Format considerations include, the size of type and font used,
the lighting conditions under which the checklist will have to be read
by the crew, and the several age groups with different viewing abilities
that may use the checklist.

The checklist must be printed in a style that will accommodate different
age groups with different eye sight abilities, and it must be of
sufficient contrast that will allow easy reading in low ambient light
levels.

At about age 50 there is a fifty percent reduction in retinal
illumination as compared to age 20.[^17] This reduction in the level of
retinal illumination also plays a role in slowing the rate as well as
the level of dark adaptation. The thickness of the eye\'s lenses is the
major cause of farsightedness among the middle aged and elderly. As the
lens thickens, it becomes yellow and reduces the transmission of blue
light through it causing older people to have more difficulty in
differentiating between colors. Generally, these effects are amplified
while reading text under adverse environmental conditions.

Other factors include the order in which items are placed on the list.
The operator should ensure that the checklist is configured in an order
that is both compatible with the aircraft\'s systems operational
sequence and at the same time compliment the crews ability to perform
tasks in a logical and consistent order.

A poorly designed checklist that is hard to read or requires constant
fine tuning and modification while being actively used by flight crews
may carry with it the potential to cause or contribute to an accident.

a\. Checklist Flow Patterns

Considering the various configurations and arrangements in the many
different aircraft cockpits it is not possible to develop a universal
flow pattern (Motor and Eye coordination) that will accommodate all
aircraft. However, by designing a checklist flow pattern that begins at
the top of a panel or aft for the overhead panels, and progresses
downward (forward for overhead panels) the operator will have
established a flow sequence that accommodates the majority.

Researchers agree that the establishment of this type of a flow pattern
can serve as an initial step toward standardizing the checklist flow for
pilots. Other benefits from the establishment of a standardized flow
include, making the checklist sequence run parallel to the initial setup
flow-patterns (which are done before running the checklist), and thereby
simplifying the learning process and the daily use of the checklist
process. The establishment of a top-to-bottom flow pattern can aid in
making the checklist actions logical and consistent in motor movement of
the head, arms, and hands. From a comfort stand point, it is less tiring
to move the arms and the head from above to below instead of the
opposite direction.[^18]

It is recommended that operators validate any new design or flow
pattern, in a simulator environment or through other means before the
checklist is presented for use by line pilots.

13\. [PRESENTING THE CHECKLIST]{.underline}

This section focuses on several typographical and environmental factors
that affect the ability of the pilot to use, and read flight deck
documentation. Much of the information contained in this section has
been condensed from a December, 1992 National Aeronautics and Space
Administration (NASA), report entitled \"On the Typography of
Flight-Deck Documentation. The report represents a summary of the
available literature regarding the design and typographical aspects of
printed text as reported by several noted researchers.

It should be noted that most of the typography data presented in this
document was collected through laboratory studies conducted in a
non-aviation environment. Therefore, the information should not be
interpreted as being the only acceptable print size and contrast that
can be used. It should serve as a baseline in designing flight deck
publications. The information presented is however, based on the best
research information available to date and

should be considered during checklist development. Because of the many
variables encountered in the cockpit: the effect of age on eye-sight,
ambient and artificial light levels, fatigue, and other conditions, it
is recommended that any new flight deck documentation design be
validated in a flight simulator using copies of the new documents and a
representative sample of the pilot population.

Checklist legibility, readability, and contrast are a prime concern
during the formatting process. If the checklist is deficient in any of
the areas that affect ease of reading, the stage my be set for the crew
to miss or overlook items or, worst case, rely on memory and avoid the
list entirely.

*Legibility of Print* enables the observer to quickly and positively
identify an alphanumeric from all other letters and characters.
Legibility depends on stroke width, form of characters, illumination on
the page, and the contrast between the characters and the background.

*Readability* generally refers to the quality of the text and depends on
the spacing between characters, words, and lines of text.

*Contrast* is the difference in color or tone between the typed letters
and the adjacent background.

In many cases, the formatting of checklists requires a compromise and
presents the operator with a challenge. The type size, depending on
checklist size that the operator desires, may be an important decision.
In most cases a large type size is preferred for legibility, however; a
smaller type size may be needed to keep the checklist size manageable in
the cockpit. Operators must ensure that the checklist is formatted with
reasonable care and concern for the crews ability to perform the
checklist with maximum accuracy. This can only be done if it is
presented in a practical and usable format.

Of particular importance is the manner in which the operator presents
the abnormal, alternate and emergency checklists to the crew.
Deficiencies in the design of these checklists are critical because of
the time limitation, workload and level of stress involved in dealing
with the situation. These checklists must be in a format that allows
quick retrieval and rapid identification of the correct procedure. A
mistake during an emergency procedure has the potential to create a
irreversible action.

a\. Typeface (Fonts)

Fonts refer to the style of alphanumeric used in printing. Two major
groups of fonts are recommended for use on the flight deck: roman and
sans-serif. Roman is well known

since it is used daily in newspapers and books. Sans-serif is a
contemporary font that does not include the small strokes (serifs) that
project horizontally from the top or bottom of a main stroke.

Several researchers have reported that when other typographical factors
are controlled, sans-serif fonts are more legible than roman. The
premise behind this statement is that absence of serifs presents a more
simple and clean typeface, and therefore improves the legibility of the
print.[^19]

Serifs disrupt character discrimination and may add uneven appearance to
the shape of strokes and characters. However, it is also evident that
they somewhat aid the horizontal movement of the eye along the printed
line\...the serifs at the top and bottom of a character create a
\"railroad track\" for the eye to follow along the line of print.[^20]

Therefore, when using a typeface without serifs, adequate spacing
between the lines of print should be used in order to prevent the eye
from slipping to the adjoining line. The designer should safeguard
against this factor as it may lead to misreading a sequence while
reading a long list.

Although there is agreement between most researchers and most human
factors design handbooks, that sans-serif is probably the best font,
there is a difference of opinion concerning style. This is something
that the operator will have to decide based on what is considered the
most legible and comfortable to the eye. The only guidance available
concerning this is to avoid a font that has characters that are too
similar to one another, as this will reduce the legibility of the print.

One experiment conducted in 1993, evaluated two fonts: a sans-serif font
(Helvetica) and a serif font (Times Roman).[^21] Subjects for the
experiment consisted of 120 flight crew members with experience levels
ranging from 90 hours flight time to 17,000. Their age varied between 20
and 54 years. In addition, eighteen of the subjects wore glasses or
contact lenses with vision corrected to normal range.

The participants were required to read aircraft manuals printed in the
subject fonts while secondary tasks were interjected. In each case the
tasks were performed in both simulated daylight and nighttime cockpit
reading conditions.

During analysis of the collected data it was determined that the
differences between fonts were small compared to the

difference between daylight and nighttime conditions. On the task of
reading word lists there was no significant difference in performance,
but Helvetica showed slightly better results.At the end of each session
the participants were surveyed as to their preference of fonts. An
overwhelming number (90 percent) preferred Helvetica.

Another font related problem is the use of dot matrix printers. The font
produced by these printers is very modular, especially in uppercase
letters. In addition, it is difficult to discriminate between characters
because of the dot construction that make up a character and the uneven
spacing between dots. There are also instances that the print is almost
unreadable because of an old ribbon in the printer. The use of dot
matrix printers should be avoided for critical flight-deck
documentation.

b\. Lower-case vs. UPPER-CASE Characters

There is almost a consensus among researchers that, when other factors
are controlled, lower case characters are more legible than upper-case
One researcher performed an experiment to determine readers attention
between upper and lowercase in newspaper headings. He reported that
lower case headings were located faster than upper case headings.
Another test was conducted with lower case and upper case fonts for
legibility and pleasingness. It was reported that lower case was read
faster and ranked higher in pleasingness.

There are several factors that contribute to the reduced legibility of
upper-case words compared with lower-case.

\(1\) Most printed material that we read is lower-case.

\(2\) Readability of lower case words is superior. Lower case words are
perceived at a greater distance, suggesting that the \"total word form\"
and legibility of the elements is important while perceiving words set
in lower case.

Note: When researchers compared legibility of individual characters,
upper-case characters were perceived at a greater distance.

DURING READING OF UPPER-CASE WORDS, PERCEPTION OCCURS IN A
CHARACTER-BY-CHARACTER ORDER, THEREBY REDUCING THE SPEED OF READING AND
READABILITY OF THE ENTIRE WORD.

The pattern or shape of a familiar word is stored in the human memory.
While reading text, a matching sequence occurs between the observed word
and the memory patterns stored in the brain. The more unique the
patterns of the word, the easier it is to perform the matching sequence.

RESEARCH SUGGESTS THAT SUCCESSIVE LINES OF PRINTED TEXT, COMPOSING A
PATTERN OF \"STRIPES\" MAY INDUCE DISCOMFORT AND ANOMALOUS VISUAL
EFFECTS TO THE READER. THE LACK OF ASCENDERS AND DESCENDERS MAY FURTHER
INTENSIFY THIS EFFECT

Lower-case words consist of characters that have ascenders (the vertical
stroke of \"d\") and descenders (\"p,\"\"q\") that contribute to the
unique shape and pattern of a word. This makes the lower-case word-form
appear more \"characteristic\". Conversely, an upper-case word appears
like a rectangular box with no distinguishable contour.

Another explanation of the greater legibility of lower-case text is the
combination of a capital letter and lower-case characters at the
beginning of a sentence and/or proper names. Research has shown that
visual emphasis given to the first letter of a word will significantly
improve the speed of a search. This finding is true for lower-case words
as well and for uppercase words combined with a larger first character.
This can be useful when a documentation designer opts to make a
distinction by using typographical features such as lower and upper-case
words and still maintain discriminibility and search speed.

When designing the font size for lower case text, the \"x\" of the font
must be considered. Usually, font height (typesize) is measured from the
top of the ascender (b) to the bottom of the descender (p).
Nevertheless, the critical value for design is the height of the
character without ascenders or descenders (e.g., \"a,\"\"c,\"\"e,\").
This height is defined as the \"x\" size of the font. Unfortunately,
most of the graphs and data tables available for determining the font
height ignore this issue.

c\. Typesize

Several research documents have been written concerning character size
and the factors that affect comfortable reading levels and label
identification. Many things affect the comfort level of reading. The
overall size of the characters, viewing distance, and illumination
levels are major factors.

Normal reading distance is about 16-24 inches from the eye. However,
distance may vary depending on personal preference, amount of room in
the cockpit, the size print, and illumination available to the reader.

Overall, based on research and tables prepared by researchers, it
appears that a font size between 0.14 and 0.20 inches is suitable for
checklists and other critical documentation used on the flight deck.
However, for practical reasons a single page checklist is generally
desired and the range (0.14 to 0.20) may not be efficient.

Another researcher conducted experiments to evaluate typesizes for
optimum reading. The sizes evaluated ranged from 0.08 to 0.14 inch, all
set in lower case using a roman font. The researcher reported that a
0.11 inch type size was read significantly faster than 0.10 inch. The
majority of the readers judged the 0.11 inch type size as the most
legible.

It is recommended that a font size below 0.10 inch not be used for
checklists or any other important flight-deck documents.

Sample checklists are provided on pages 38 and 39 for side-by-side
comparison of previously discussed styles. Both are presented in the
minimum font size (0.10 inch) and preferred sans-serif style. In
addition, figure 1, is presented in upper-case only and figure 2, is
presented in mixed upper and lower-case. Both are printed in bold type
for maximum contrast.

THE REMAINDER OF THIS PAGE

INTENTIONALLY LEFT BLANK

Figure 1

Sample checklist set in UPPER-case

Helvetica (sans-serif) font.

Figure 2

Sample checklist set in mixed UPPER and lower-case,

Helvetica (sans-serif) font .

d\. Stroke Width and Height-to-Width

Stroke widths affect the ability of the eye to differentiate between the
stroke of the character (\"I\") and the space inside the character
(\"E,F\"). The width of a stroke is a function of height of the
character. Most human factors data books recommend the use of a
height-to-width ratio of 5:3.

The recommended ratio is applicable only when the document is in front
of the observer (a 90 degree angle between the line-of-sight and the
document). Fixed displays, such as a mechanical checklist, may not be
located in front of the pilot. Therefore, in designing a display that is
viewed from an unfavorable angle the operator must take into account
that the angle may reduce the apparent width of the character. When this
situation exists a different height-to-width ratio, e.g., 5:4 should be
considered.

e\. Horizontal and Vertical Spacing

The vertical and horizontal spacing between characters affects the
legibility and readability of the text. Increasing vertical spacing
between lines reduces the probability of adverse visual effect.
Researchers suggest that judgments of the clarity of text\... are
critically dependent on the spacing of lines, more so than the overall
density of lettering on the page. The clarity of text can be increased
by increasing the separation between the lines slightly and decreasing
slightly the mean horizontal spacing between the centers of letters,
within the limits of conventional typography. In addition, the opening
of vertical space between lines reduces the chance of optical bridging
between adjacent lines\... a critical factor for the design of any list
type document.

The recommended vertical space between lines is 25-33 percent of the
overall size of the font. The horizontal space between characters should
not be less than one stroke width. As for word spacing, the gap between
characters should be large enough to allow grouping of words. This is
achieved when the word space is 25 percent of the overall height and not
less than one stroke width.

f\. Connector Line Length

A connecting line between the challenge and the required response is an
important item in checklist design

(Challenge\...\....Response). Researchers indicate that a common problem
in checklist design is the large gap between the challenge and the
response.

The wider the gap, the greater the chance that the reader will make a
mistake through perceptual misalignment. The connector line guides the
readers eyes across the page.

Although the connector line assists the reader it should be noted that
as the distance between columns of information is widened and connector
line length is increased, the line becomes less effective. Checklists
that are designed to cover the entire page of an 8.5 by 11 inch piece of
paper are more prone to misalignment.

g\. Use of *Italics*, Bold, and [Underline]{.underline}

Several experiments have been conducted to determine the effects of
different type faces on legibility. One experiment indicated that the
reading of materials in italic face was 2.7 percent slower than roman
lower-case of the same height. In addition, 96 percent of a 224 subject
group judged italic to be less legible than a regular roman font.

Another study was conducted in which it was determined that bold face
text was read at the same speed as lower-case text. However, 70 percent
of the subjects commented about the unpleasingness of the text as
compared to the roman font.

Additional experiments indicate that bold and medium face do not differ
in readability, even under low illumination; suggesting that there is no
apparent advantage in printing long chunks of text in bold face.
Nevertheless, bold face can be safely and advantageously used for
contrast and emphasis. Although faces can highlight a specific item on a
document, over usage of this typographical technique can be inefficient.
Employing too many faces for contrast, emphasis, and attention may be
confusing and can reduce the legibility, and readability of the printed
material.

It is recommended that the designer avoid using long strings of text set
in italics and remain throughout the checklist with one or two
typefaces.

h\. Quality of Paper and Print

The quality of the paper that the checklist is printed on is important
for two reasons.

First, if the checklist paper is printed on both sides, it is difficult
to read if the print on the back side shows through the paper. Paper
checklists should be printed on a thick opaque paper that, when back
lighted, will prevent the print on the other side from showing through
and blurring the print on the front.

The second, is the quality of the actual print. Black print on a white
background is preferred. The print should be clear and the boundaries
between strokes and spaces should be sharp and distinguishable.

i\. Copying the Checklist

Operators should exercise care if they attempt to reproduce checklists
on commercial copy machines. To avoid any degradation of the
reproduction quality only an unlaminated original should be copied.

Reproduction from non-original copies may reduce the vertical spacing
between characters and reduce stroke discrimination, thereby, affecting
the reading quality.

When copies are made from non-original copies a certain amount of
definition is lost. If copies are made from documents that have been
laminated the quality of light transmission necessary for the
reproduction process may affect both contrast and legibility of the
copy.

14\. [THE CUMULATIVE EFFECT OF IMPROPER TYPOGRAPHY]{.underline}

For a given printed document, the combination of two or more non-optimal
or marginal conditions will have a greater affect on legibility and/or
readability. The designer should avoid combining non-desirable
conditions as they can reduce the overall efficiency of using the
checklist.

Appendix I

CHECKLIST DEVICES

Over the years several types of checklists and aids for completing
checklists have been developed. Each has inherent advantages and
disadvantages. This document does not recommend any one device over the
other as they all satisfy the requirements contained in Federal
regulation.

The following pages provide a listing of the more common methods of
providing flightcrew checklists and their related advantages and
disadvantages.

[PAPER CHECKLIST]{.underline}

  ----------------------------------- -----------------------------------
  [ADVANTAGES]{.underline}            [DISADVANTAGES]{.underline}

                                      

  1\. Easy to stow                    1\. Easily damaged or worn may lose
                                      its legibility

  2\. Inexpensive to produce          2\. Easy to misplace

  3\. Inexpensive to update           3\. Easy to remove from aircraft

                                      4\. May be difficult to read if
                                      type size or fonts are not adequate

                                      5\. May be difficult to read under
                                      low ambient light if paper and
                                      print are not of sufficient
                                      contrast

                                      6\. No memory or recall feature

                                      7\. No automatic means of noting
                                      progress if interrupted or
                                      distracted

                                      8\. Hand held

                                      9\. Promotes head down posture
  ----------------------------------- -----------------------------------

Appendix I

CHECKLIST DEVICES

[LAMINATED PAPER/CARD CHECKLIST]{.underline}

(MOST COMMON)

  ---------------------------------- ------------------------------------
  [ADVANTAGES]{.underline}           [DISADVANTAGES]{.underline}

  1\. Withstands hard use            1\. Easy to misplace

  2\. Retains legibility longer      2\. May be difficult to read if type
                                     size and fonts are not adequate

  3\. Easy to stow                   3\. Readability may be hindered by
                                     surface glare

                                     4\. Easy to misplace or remove from
                                     the aircraft

                                     5\. No memory or recall feature

                                     6\. No automatic means of noting
                                     progress if interrupted or
                                     distracted

                                     7\. Hand held

                                     8\. Promotes head down to read
  ---------------------------------- ------------------------------------

Appendix I

CHECKLIST DEVICES

[SCROLL TYPE CHECKLIST]{.underline}

(Common in military aircraft)

+-----------------------------------+-----------------------------------+
| [ADVANTAGES]{.underline}          | [DISADVANTAGES]{.underline}       |
+-----------------------------------+-----------------------------------+
| 1\. Permanent mount               | 1\. No memory or recall feature   |
|                                   | if items are deferred             |
+-----------------------------------+-----------------------------------+
| 2\. Promotes head up posture      | 2\. Back lighting required        |
+-----------------------------------+-----------------------------------+
| 3\. Has a reference line to mark  | > 3\. Normally mounted on the     |
| progress                          | > pilot side of the aircraft or   |
|                                   | > at the flight engineer\'s       |
|                                   | > panel. Size of print, type      |
|                                   | > font, and distance from other   |
|                                   | > crewmembers may affect          |
|                                   | > readability                     |
+-----------------------------------+-----------------------------------+
| 4\. Easy to update                | 4\. Cost and expense of           |
|                                   | installation                      |
+-----------------------------------+-----------------------------------+
| 5\. Remains in full view of the   |                                   |
| crew as a reminder to perform the |                                   |
| checklist                         |                                   |
+-----------------------------------+-----------------------------------+
|                                   |                                   |
+-----------------------------------+-----------------------------------+

Appendix I

CHECKLIST DEVICES

[ELECTROMECHANICAL]{.underline}

  ----------------------------------- -----------------------------------
  [ADVANTAGES]{.underline}            [DISADVANTAGES]{.underline}

  1\. Provides a systematic means of  1\. Limited to the number of task
  recall if items are deferred        items.

  2\. Provides a clear view to        2\. Cost and expense of
  crewmembers of checklist status     installation.

  3\. Promotes head up posture        3\. Back lighting required

  4\. Permanently mounted, can not be 4\. Limited to only critical items.
  removed from aircraft or misplaced  Requires the use of a supplemental
                                      list

  5\. Provides back lighting for      
  easier reading in low ambient light 

  6\. Does not require the user to    
  hold it                             

  7\. No surface glare                

  8\. Will not loose its legibility   
  through normal wear                 

  9\. Remains in full view of the     
  crew as a reminder to perform the   
  checklist                           
  ----------------------------------- -----------------------------------

Appendix I

CHECKLIST DEVICES

[ELECTRONIC/AUTOMATED (CRT)]{.underline}

  ----------------------------------- -----------------------------------
  [ADVANTAGES]{.underline}            [DISADVANTAGES]{.underline}

  1\. Is stationary in the aircraft.  1\. May displace or share time with
  Can not be lost                     other needed displays, e.g., Radar

  2\. Can not be removed from the     2\. Requires head down posture to
  aircraft                            read and operate

  3\. Depending on the system some    3\. May be hard to locate a list or
  are equipped with sensors that      return to a certain point
  verify checklist items completed    

  4\. Retains legibility              4\. Cost and expense of
                                      installation

  5\. Provides a systematic recall if 
  items are deferred                  

  6\. Does not require the user to    
  hold it                             

  7\. Provides immediate status of    
  items                               

                                      

                                      
  ----------------------------------- -----------------------------------

Appendix II

[REFERENCE MATERIALS]{.underline}

a\. The Use and Design of Flightcrew Checklists and Manuals. Report No.
DOT/FAA/AM-91/7, April 1991. Available through the National Technical
Information Service, Springfield, Virginia 22161

b\. Human Factors of Flight-Deck Checklists: The Normal Checklist.
Report No. NASA CR-177549, May 1990. Available through NASA Ames
Research Center, AFO-MS 262, Moffett Field, California 94035-1000

c\. On the Typography of Flight-Deck Documentation. Report No. NASA
CR-177605, December 1992. Available through NASA Ames Research Center,
AFO-MS 262, Moffett Field, California 94035-1000

d\. National Transportation Safety Board, Safety Study; A Review of
Flightcrew-involved, Major Accidents of U.S. Air Carriers, 1978 through
1990. Report No. NTSB/SS-94/01. Available through the National Technical
Information Service, Springfield, Virginia 22161

e\. Airman\'s Information Manual. Available through the Superintendent
of Documents, P.O. Box 371954, Pittsburgh, PA 15250-7954.

f\. Philosophy, Policies, and Procedures: The Three P\'s of Flight-Deck
Operations. Authors: Asaf Degani, San Jose State University, San Jose,
CA. and Earl L. Wiener, University of Miami, Coral Gables, FL.

g\. Wiener, E.L. & Nagel, D.C. (Eds.), Human Factors in Aviation, San
Diego: Academic Press, Chapters authored by, Christopher D. Wickens and
John M. Flach; Information Processing, and Aircrew Fatigue and Circadian
Rythmicity R. Curtis Graeber.

h\. Human Factors in Flight, Frank H. Hawkins, 2nd edition, Ashgate
Publishing Co. Brookfield, VT

i\. Federal Aviation Administration, Order 8400.10, Air Transportation
Operations Inspector\'s Handbook

j\. Font To Use In Cockpit Documentation, Hans de Ree, and Gerco Blok,
Proceedings of the Seventh International Symposium On Aviation
Psychology, Columbus, OH.

Appendix II

k\. Checklist Reports, Search request 3469, May 24, 1994, Aviation
Safety Reporting System.

l\. Flight Crew Distraction Reports, Search request 3482,

May 24, 1994, Aviation Safety Reporting System.

m\. Checklist Reports, Search Request 3409, April 5, 1994, Aviation
Safety Reporting System.

n\. Checklist Reports Involving Pilot Fatigue, Search Request 3535,
Aviation Safety Reporting System.

o\. On the Design of Flight-Deck Procedures. Report No. NASA CR 177642,
June 1994, Available through NASA Ames Research Center, AFO-MS 262,
Moffett Field, California 94035-1000.

Appendix III

[RELATED READING MATERIAL]{.underline}

In addition to the reference documents listed in Appendix II, the
following documents were reviewed in support of this writing.

1\. The Use and Design of Flight crew Checklists and Manuals. A report
developed by the Civil Aeromedical Institute CAMI), April 1991

2\. Human Factors of Flight Deck Checklists: The Normal Checklist,
Developed by NASA, May 1990

3\. On the Typography of Flight-Deck Documentation Developed by NASA,
December 1992

4\. Cockpit Interactions Incidents Analysis, Developed by ASRS, March
1994

5\. Crew Resource Management: An Introductory Handbook, Developed by
FAA, August 1992

6\. Operational Use and Modification of Electronic Checklists, Draft
Advisory Circular developed by AFS-400.

7\. Human Factors in Flight, A text book by Frank H. Hawkins, Second
edition 1993

8\. Cockpit Management Concept Instructor Guide, Prepared by Flight
Safety International

9\. Suggested Format for Field Investigation Guide Draft),Prepared by
the Transportation Safety Institute, Aviation Safety Division

10\. NTSB Safety Study - A REVIEW OF FLIGHT CREW-INVOLVED ACCIDENTS OF
U. S. AIR CARRIERS, 1978 THROUGH 1990

11\. ASRS Checklist Report, prepared April 1994.

12\. NTSB causal and contributing narrative reports relating to Part 121
and 135 accidents.

13\. Federal Aviation Regulations Parts 121, 135, 25,and 21.

Appendix III

14\. ASRS Cockpit Interactions Incidents Analysis, prepared March 17,
1994.

15\. Text book - Human Factors In Aviation, Academic Press.

16\. Text book - Aviation Psychology, Gower Publishing.

17\. Text book - Human Error - Cause, Prediction, and Reduction,
Lawrence Erlbaum Associates Publishing.

18\. FAA Accident/Incident Data System (AIDS) Database. Pilot induced
accidents and incidents.

19\. Research Paper - The Emergency Checklist, Testing Various Layouts.
Ohio State University, Aviation Psychology, Volume 1, 1991, Page 160.

20\. Research Paper - Electronic Checklists: Evaluation of Two Levels of
Automation. Ohio State University, Aviation Psychology, Volume 1, 1991,
Page 178.

21\. Research Paper - Philosophy, Policies, and Procedures: The Three
P\'s of Flight-Deck Operations. Ohio State University, Aviation
Psychology, Volume 1, 1991, Page 184.

22\. Research Paper - Time Pressure: As a Causal Factor in Aviation
Safety Incidents: The \"Hurry-up\" Syndrome. Ohio State University,
Aviation Psychology, Volume 1, 1993, Page 269

23\. Research Paper - Font To Use In Cockpit Documentation. Ohio State
University, Aviation Psychology, Volume 1, 1993, Page 115.

24\. Air Line Pilot Magazine, SET and CHECKED, authored by Mark L.
Majikas, Piedmont Airline, May 1989,

25\. Periodical, ASRS Callback, No. 178, March 1994 \"How Could We Let
This Happen to Us?!\"

26\. Human-Centered Aircraft Automation: A Concept and Guidelines, NASA
Technical Memorandum 103885, August 1991.

27\. FAA Aviation Safety Hotline Query, April 1994.

28\. Checklist Procedures and the Cost of Automaticity, University of
Colorado under contract to the U. S. Army Research Institute, 1993.

Appendix III

29\. NTSB Accident Report, Northwest Airlines, August 16, 1987, Detroit
Airport, Romulus, MI. (MD-82, No flaps or slats on take-off)

30\. NTSB Accident Report, USAIR, September 20, 1989, Laguardia Airport,
Flushing, NY. (B-737, incorrect rudder trim setting)

31\. NTSB Accident Report, Delta Airlines, August 31, 1988, DFW Airport.
(B-727, No flaps or slats on takeoff)

32\. NTSB Accident Report, Frontier Airlines, March 23, 1983, Casper,
WY. (B-737, Landed gear up following a proper checklist sequence)

33\. Cleared for the Visual Approach: Human Factor Problems in Air
Carrier Operations. NASA Report 166573, Authored by William P. Monan,
1983

34\. Age, Altitude, and Workload Effects on Complex Performance,
Authored by Mertens, Higgins, and McKenzie, FAA (CAMI) Report
FAA-AM-83-15, 1983.

35\. Information Transfer Problems in the Aviation System, NASA
Technical Paper 1875, 1981

36\. NASA Technical Memorandum, X-3546, Third Quarterly Report, Jan.
1977

37\. Boeing 777 Checklist Procedure and Checklist Design Philosophy,
Boeing Commercial Airplane Group, Flight Training Division, May 1994

38\. Air Carrier Operations Bulletin (ACOB) No. 8-88-4, Takeoff Warning
System Checks

39\. Air Carrier Operations Bulletin (ACOB) No. 8-90-6, Visual
Confirmation and Accomplishment of all Operating Checklist Items.

40\. Minutes of SAE, G-10 Committee, Aerospace Behavioral Engineering
Technology, a presentation by Frank Murphy, Chief Engineer, Flight Deck
and Systems Requirements, Boeing Commercial Airplane Group.

41\. NASA Aviation Safety Reporting System: Ninth Quarterly Report (NASA
Technical Memorandum 78608).

42\. ASRS Search Request No. 3469, Checklist Reports, May 24, 1994

Appendix III

43\. Safe Skies for Tomorrow, Aviation Safety in a Competitive
Environment, a report to the Congress of the United States prepared by
The Office of Technology Assessment.

44\. Checklists for the Boeing 757/767 provided by American, Delta,
United, and Northwest Airlines for comparison.

45\. Rejected Takeoffs: Causes, Problems, and Consequences,prepared by
ASRS, May 1, 1991.

46\. Who Or What Saved The Day? A Comparison Of Traditional and Glass
Cockpits, prepared by NASA-Ames based on ASRS data.

47\. Fatigue And Associated Performance Decrements In Air Transport
Operations, Prepared by ASRS for NASA-Ames, March 1981.

48\. Flight Crew Performance When Pilot Flying And Pilot Not Flying
Duties Are Exchanged, prepared by ASRS for NASA-Ames, June 1982

49\. Flight Crew Distraction, ASRS Report 3482, May 1994

50\. Intervention Strategies for the Management of Human Error, NASA
contractor report 4547, Pre-publication copy, August 1993.

51\. Flying Safety, USAF publication, Article on Fatigue, September
1994.

52\. On the Design of Flight-Deck Procedures, NASA contractor report
177642, June 1994.

53\. Use and Function of Checklists, by Flight Safety International,
April 1988.

54\. NASA Technical Memorandum 78608, Ninth Quarterly Report, Cockpit
Distraction: Causes and Types.

55\. ICAO Circular 240-AN/144, Human Factors Digest No. 7, Investigation
of Human Factors in Accidents and Incidents.

56\. Airline Pilot Magazine, \"Hurry Up\" Syndrome, July/August 1994.

57\. Air Transport World, The Search for Probable Cause, July 1994.

Appendix III

58\. Boeing Procedure and Checklist Philosophy, Chapter V, Design and
modification of checklists, provided by Boeing Flight Training.

59\. Human-Centered Aircraft Automation: A Concept and Guidelines, NASA
Technical Memorandum 103885, August 1991.

60\. Flight Safety Foundation, Human Factors & Aviation Medicine: Flight
Deck Confusion Cited In Many Aviation Incidents Reports, July/August
1994.

61\. Minutes of Meeting No. 25. SAE Committee G-10, Aerospace Behavioral
Engineering Technology. January 18-21, 1994, Daytona Beach, Florida.

62\. Air Transportation Operations Inspector\'s Handbook, FAA Order
8400.10.

63\. Military Specification - Mil-C-27278B (USAF), Flight Crew
Checklists.

64\. Military Specification - MIL-M-7700D (USAF), Flight Manuals.

[^1]: NTSB Safety Study, A Review of Flightcrew-involved, Major
    Accidents of U.S. Air Carriers, 1978 through 1990

[^2]: NTSB Safety Study, NTSB/SS-94/01

[^3]: ASRS Search Request Nos. 3469, 3409, and 3482

[^4]: ASRS Search Request No. 3469

[^5]: Information contained in these studies was gathered through
    observed pilots use of checklists, analyzed incident/accident
    reports, and observed simulator sessions.

[^6]: Degani, Wiener (1990)

[^7]: Miller (1956), Wickens (1988), Flach (1988).

[^8]: Wickens, et al, (1988)

[^9]: Degani, Wiener, (1990)

[^10]: Degani, Wiener, (1994)

[^11]: Degani, Wiener, (1990)

[^12]: Degani, Wiener, (1990)

[^13]: Degani, Wiener, (1990)

[^14]: Degani, Wiener, (1990)

[^15]: Degani, Wiener, (1990)

[^16]: The information contained in Section 11, is obtained from FAA
    Order 8400.10, Air Transportation Operations Inspector\'s Handbook,
    Volume 3, Chapter 15, Section 5. This section presents the
    guidelines used by POI\'s in their evaluation of operator developed
    checklists.

[^17]: Degani, (1992)

[^18]: Degani, Wiener, (1990)

[^19]: This paragraph is Helvetica, a sans-serif font.

[^20]: This paragraph is Roman, a serif font

[^21]: Hans de Ree, et al, (1993)