***Workshop***

***on***

***Flight Crew Accident and Incident***

***Human Factors***

***June 21-23, 1995***

![](media/image1.emf){width="3.0541666666666667in"
height="2.9444444444444446in"}

***Office Of System Safety***

***PROCEEDINGS***

**TABLE OF CONTENTS**

Page

*EXECUTIVE SUMMARY v*

*1. WORKSHOP SUMMARY*

*1.1 Introduction*

*1.2 Scope*

*1.3 Background*

*1.4 Workshop Organization*

*1.5 Objective 1 Discussion Items*

*1.6 Objective 2 Discussion Items*

*1.7 Objective 3 Discussion Items*

*1.8 Cautions and Pitfalls Heard Around the Workshop*

*1.9 Summary*

*2. HUMAN FACTORS DATA PROJECT SUMMARY*

*2.1 Background*

*2.2 Activities to Date*

*2.3 Activities Planned*

*3. LIST OF ATTENDEES*

*4. APPENDIX A - POSITION PAPERS A-*

*4.1 Summary of Position Papers Lloyd Hitchcock A-*

> *4.2 Databases for Effective Safety Enhancement Air Line Pilots
> Association A-*

*4.3 Incident Investigation: Paradigm Shifts toward*

*Exploiting Successful Human Factors Intervention*

*Strategies to Prevent Aircraft Accidents Ludwig Benner, Jr., P.E.*

*and Ira J. Rimson, P.E. A-*

> *4.4 Human Error Identification: Designing for the*

*Users Marilyn Sue Bogner, Ph.D. A-*

*4.5 Flight Crew Accident and Incident Human Factors -*

*Objective 1 A-*

*4.6 Human Factors Research into Cockpit/Cabin*

*Crew Performance Rebecca D. Chute A-*

*4.7 ICAO Views on the Application of Human Error*

*Models to Accident and Incident Data Stephan J. Corrie A-*

*4.8 Assessment of Human Error from Transportation*

*Accident Statistics Vernon S. Ellingstad*

*and David L. Mayer A-*

*4.9 Practical Issues Impacting Human Factors*

*Investigations at the Airline Level Captain L. R. Ganse A-*

*4.10 Flight Crew Accident and Incident Human Factors R. Curtis Graeber
A-*

*4.11 Human Factors Issues Associated with Flight*

*Crew Accidents and Incidents Steven D. Harper*

*and Mark J. Detroit A-*

*4.12 Human Factors Considerations to Be Used*

*by Certification Test Pilots in the FAA Approval of*

*Certain Electronic Systems in Various Aircraft George Kaseote*

*and George Lyddane A-*

*4.13 Human Factors in Aviation Safety Data and*

*Accident Investigation Thomas Lilli A-*

*4.14 Flight Crew Accident and Incident Human Factors*

*and Analysis Ronald C. McGarry A-*

*4.15 Issues Related to Use of ASRS Incident Data*

*with Human Performance and Error Models Dr. Rowena Morrison A-*

*4.16 Needed: A Systematic Approach for a Cockpit*

*Automation Philosophy Hop Potter A-*

*4.17 Baseline Assessment of the National Airspace*

*System: An Approach toward Modeling Operational*

*Errors Mark D. Rodgers*

*and Carol A. Manning A-*

*4.18 The Quest For Aviation Safety's Holy Grail:*

> *Finding Underlying Causes of Accidents and Incidents*
>
> *or if You Really Want to Improve Aviation Safety,*
>
> *You Must First Identify Systemic Problems Captain Robert L. Sumwalt,
> III A-*
>
> *4.19 Assessing the Quantity and Quality of Aviation*

*Safety Icident Reports: Two Studies Michal Tamuz A-*

*4.20 In Search of Answers without Questions:*

*Or, How Many Monkeys at Typewriters*

*Will It Take... John A. Wise*

*and Mark A. Wise A-*

*4.21 Learning from Accidents: The Need to Look*

*behind Human Error David D. Woods*

*and Nadine B.Sarter A-*

***EXECUTIVE SUMMARY***

On June 21 - 23, 1995, the Federal Aviation Administration's (FAA's)
Office of System Safety, as part of its Human Factors Data Project,
convened the Workshop on Flight Crew Accident and Incident Human Factors
at The MITRE Corporation in McLean, Virginia. The workshop provided a
forum for discussing and documenting activities that aviation accident
investigators, safety analysts, data specialists, and human factors
professionals are involved in relative to flight crew accident and
incident human factors. The following were the objectives of the
workshop:

-   *Identify the relevant human factors data elements, or types of
    data, required for appropriate analysis of flight crew error
    contributing to aviation accidents and incidents.*

-   *Define how various models of human performance and human error can
    be incorporated into a process for analyzing human factors data in
    aviation accidents and incidents.*

-   *Define the conceptual interface between users, data systems, and
    analytical models/taxonomies.*

Each objective was discussed at round tables of approximately seven
attendees with diverse affiliations, expertise, and interests. A total
of 60 individuals attended the meeting. Government agencies, industry,
and academia were represented, as were both foreign and domestic
interests.

Based on the discussions reported by the round tables, there were four
primary items identified for the first objective.

1.  *Provide a "history of actions by crew" that enabled the crew to
    resolve operational problems.*

2.  *Commuting/deadheading is an area of concern regarding crew
    operations in commercial aviation.*

3.  *Decision making processes of the flight crew are not traceable in a
    reliable manner.*

4.  *Normative data should be collected from line flying.*

The second objective provided eight primary discussion items.

1.  *When available, utilize the flight crews' knowledge to explore the
    event.*

2.  *Develop aviation community partnerships.*

3.  *Develop airline organizational models.*

4.  *Develop an automated analytical support capability.*

5.  *Develop analytical process based on user's needs and
    understanding.*

6.  *Develop [a]{.underline} process, not [the]{.underline} process.*

7.  

8.  *Define measures of human factors data.*

9.  *Define measures of human factors models.*

There were five primary discussion items for the third objective.

1.  *Define the users as having fundamental interest in aviation
    operations and safety.*

2.  *Develop a system for classifying the type of data observation.*

3.  *The user interface must be based on the types of users identified
    by the round tables.*

4.  *On-line access of accident and incident data.*

5.  *Narrative descriptions of events are required.*

A large amount of detailed information was provided at the workshop.
This information is being evaluated by the members of the Human Factors
Data Project and will be incorporated into the planning of future
project activities.

The quantity and diversity of the attendance and the enthusiasm
displayed throughout the workshop was overwhelming, clearly
demonstrating the need for the Office of System Safety to continue its
focus on improving the processing of flight crew accident and incident
human factors data. The workshop was also a successful beginning in that
it brought together for the first time, leaders in the field from
government, industry, and academia to discuss their ideas and concerns
about accessing, integrating, and analyzing flight crew accident and
incident human factors data. Another workshop is being planned for the
Spring of 1996 to discuss the progress made on the Project and to
revalidate the user requirements established at this workshop.

# Workshop Summary

## Introduction

A workshop to discuss and document industry activities related to flight
crew human factors in accidents and incidents was convened by the
Federal Aviation Administration\'s (FAA's) Assistant Administrator for
System Safety, Mr. Christopher Hart. Opening remarks were also provided
by Dr. Mark Hofmann, the FAA's Chief Scientist and Technical Advisor for
Human Factors. Held June 21-23, 1995 at The MITRE Corporation in McLean,
Virginia, the workshop focused on three objectives:

*¥ Identify the relevant human factors data elements, or types of data,
required for appropriate analysis of flight crew error contributing to
aviation accidents and incidents.*

*¥ Define how various models of human performance and human error can be
incorporated into a process for analyzing human factors data in aviation
accidents and incidents.*

*¥ Define the conceptual interface between users, data systems and
analytical models/taxonomies.*

## Scope

The background and organization of the workshop are briefly summarized.
Following these sections are the summaries of the primary discussion
items for each workshop objective. Due to the large amount of diverse
comments and in-depth discussion at each of the eight round tables of
the workshop, a detailed analysis of each round table is not within the
scope of this summary, therefore, only the topics and points on which
there were prevailing opinions are summarized. The total collection
(125+) of objective templates completed by each round table is available
electronically in an un-edited format. Workshop attendees may request
these templates by sending either an electronic mail message to
*jblancha@MITRE.org* or a self-addressed stamped mailer (10" x 14") and
a formatted diskette (High density only) to: Jim Blanchard, The MITRE
Corporation, 7525 Colshire Drive, McLean, Virginia, 22102.

## Background

The workshop was designed to provide guidance and expertise to the FAA's
Office of System Safety to define a process to support industry and
government users of aviation safety data systems. To implement this
process, the project team is designing and developing an integration
tool to access and integrate various data archives relative to aviation
accident and incident human factors. In the future, additional resources
available within the FAA and other aviation organizations may also be
included in the process and subsequently implemented through the
integration tool.

To provide a strong foundation for the functional requirements of the
process and integration tool, the project team conducted a one year
study and analysis of the users of aviation safety data. The results of
this study, presented in the background research report dated October
1994, identified a number of functional requirements for accessing,
integrating and analyzing flight crew human factors in accidents and
incidents. Three of these requirements are embodied in the objectives of
the workshop. To further understand the needs of the users of aviation
safety data and human factors related to flight crews, the project team
elected to conduct the workshop as a planning step in the design and
development of the process and integration tool. Involving the users
from the outset was thought to be the best method of defining and
designing an automation capability for the limited number of users known
to require human factors data on flight crews. Air carriers known by the
project team to be currently publishing and practicing aviation safety
data analysis were invited. Academic leaders in the field, as well as
interested government and industry professionals were also invited. The
workshop had representatives from all qualified user populations within
the United States, as well as a small international representation. The
attendance distribution by company or government organization (many
organizations/divisions representing the same agency) indicates the
breadth of the workshop.

> ¥ Government representatives totaled 20 organizations/divisions
>
> ¥ Industry representatives totaled 18 different companies, including
> associations
>
> ¥ Academia representatives totaled 4 different institutions

## Workshop Organization

The attendees were welcomed, heard opening remarks and were provided
with background information on the objectives and structure of the
workshop during the first half of day one. The second half of day one
was spent in randomly selected teams of five to seven attendees working
as \"round tables\" on the first objective. Templates were provided to
guide the reporting and participants discussed freely the objectives and
their implications on the aviation community. The second and third
objectives were the focus of day two. Round table participants were
organized by the staff to ensure a balanced team of five to seven
attendees was present at each table, relative to their personal
experience and interest. Again, templates were used to guide the
reporting of the discussions. A summary of the workshop and facilitated
discussion was held on day three. Following closing remarks from the
Assistant Administrator of System Safety, Mr. Christopher Hart, the
workshop adjourned at noon on day three.

## Objective 1 Discussion Items

***Objective 1.** Identify the relevant human factors data elements, or
types of data, required for appropriate analysis of flight crew error
contributing to aviation accidents and incidents.*

[Provide a \"history of actions by crew\" that enabled crew to resolve
operational problems.]{.underline} Flight crews often develop solutions
to problems as they occur in-flight. Too many times these solutions are
\"not reported\" or \"not described adequately\" in the post flight
analysis and therefore they are not part of the record or solution
published for others to consider. Data relevant to event reconstruction
is needed for \"successful outcome events\" as well as those where the
outcomes result in an accident and incident.

[Commuting/deadheading is an area of concern regarding crew operations
in commercial aviation.]{.underline} A large percentage (unknown but
estimated by the air carriers at greater than 40 percent) of the
commercial pilot population commutes to work in an environment where the
commute itself is mentally and physically taxing. Some of these commutes
include cross-country flying and jump seat rides that take the better
part of a day. While not recognized as a factor in the majority of
accidents and incidents, most of the round tables commented that more
data about the pilot\'s activities prior to flight might be helpful in
advancing the understanding of flight crew human factors.

[Decision making processes of the flight crew are not traceable in a
reliable manner.]{.underline} The process by which a crew member arrives
at a decision is a major \"missing link\" in the existing investigative
techniques of the accident and incident process. Understanding *how* a
crew member arrives at a decision is a critical component of
understanding and predicting flight crew performance.

[Normative data collected from line flying.]{.underline} There exists a
lack of understanding and documentation about what pilots do when they
are *flying safely*. Rarely is a normal or safe flight reviewed for what
went right. Collecting sample data about *normal operations* would be
useful in understanding the effectiveness of training operations and
safety concerns of the operators. Such data should be collected both in
a measured capacity, such as Flight Operations Quality Assurance (FOQA),
and observed process, such as Computer-Aided Debriefing Stations (CADS),
and post-flight analysis systems designed to highlight practices and
procedures where training and performance are observed as predicted.

Other significant discussions centered around:

¥ The use of a *multi-level criticality ranking scale* for incidents
which \"flags\" single occurrences that should (1) not be allowed to
escalate or (2) be treated as a single, isolated event.

¥ The need for a *flexible dictionary* of terms which supports the
integration of safety data systems and facilitates the discussion of
human performance among operators and users having dissimilar
backgrounds or goals.

¥ A *save button* which allows the crew to mark the flight data recorder
and cockpit voice recorder for post-flight analysis upon arrival at a
destination.

¥ A *fast review capability* for analyzing post-flight data in
faster-than-real-time modes.

¥ A system capable of detecting *patterns and trends* in data that is
being monitored in programs such as FOQA or Save Button.

¥ A capability to *control the reporting bias* (inter-rater reliability
is one type of bias which was discussed). Some round tables discussed
the need for data collection training at the source of the data, such as
an investigator course designed for human factors data collectors.
Others reported the need for users to be trained.

¥ Develop and support a more effective and widespread use of the
*call-back procedures* developed by the team at the Aviation Safety
Reporting System (ASRS). Include commercialization development of this
process for non-governmental parties interested in the procedure.

¥ Devise an *incentive program* to entice and ensure contribution of the
operator in the safety reporting systems of the future.

¥ Develop a general aviation digital flight data recorder to support
incident and accident reconstruction and understanding of pilot
performance.

¥ Review the investigative and analytical procedures used by insurance
companies for understanding human performance and the sharing of
information between operators and insurers.

¥ The use of continuous video recording and voice recording to support
the identification and analysis of pilot perceptions in flight.

## Objective 2 Discussion Items

***Objective 2.** Define how various models of human performance and
human error can be incorporated into a process for analyzing human
factors data in aviation accidents and incidents.*

[When available, utilize the flight crews' knowledge to explore the
event.]{.underline} The Federal Aviation Administration and the industry
need to focus on the development of \"Talk to the Pilot\" models. These
models of analyzing flight crew human factors should elicit information
about the event under study and be non-punitive towards participants.

[Develop aviation community partnerships.]{.underline} The Federal
Aviation Administration and the industry, along with academic and
government scientists, need to work together to develop teams of
expertise in next-generation human performance analysis. Two significant
comments surrounding this issue centered on (1) the need for teaming as
a method for tackling an analytical problem that is beyond any one
organization\'s ability to resolve, and (2) the need for partnerships as
a way to gain a significant amount of information about flight crew
activities in normal operations over a shorter period of time than would
be possible if only one carrier collected data on/for themselves and did
not share it with other interested parties.

[Develop airline organizational models.]{.underline} The influence of
the organizational leadership, operations management, and corporate
culture has been cited recently (and increasingly) as a factor in a
flight crew\'s ability to operate the aircraft. Without detailing the
value of such an influence (the workshop did not suggest a viewpoint as
a prevailing opinion), an understanding of the influence of corporate
culture on the crew\'s performance is critical to the development of
effective human performance and error models relative to flight crews.
This would include identifying and collecting data to support the
models. Organizational factors are *external influencers* known to be
observable in flight crew performance analytical models. Further
development of analytical models describing the influence of the
organization on the crew is needed to advance in this area of flight
crew human factors.

[Develop an automated analytical support capability.]{.underline} There
are certain activities that computers do well, and others that are
clearly the role of the human operator. Many operators have desktop
computers linked to information systems in their organization. Most of
those who have these technologies collect data and analyze it, sometimes
producing poor quality and useless analytical results. Developing an
automated process would help alleviate *errors of analysis* and improve
usability by suggesting areas requiring further investigation or
attention. Most operators' analytical capabilities are based on a human
analyst\'s ability to reason and apply common sense to a mountain of
data. These human reasoning skills must not be destroyed in the process
of developing analytical technology and systems designed to support
aviation safety analysts. Correctly using the available data systems
*and* analytical tools is a major requirement of any process to access,
integrate or analyze flight crew human factors in accidents and
incidents.

[Develop analytical process based on user needs and
understanding.]{.underline} Similar to the point immediately above, the
need to have the human expert developing and overseeing the analytical
process was stressed by the round tables as critical to the process of
studying human factors in flight crews. Specifically focused on the
reasoning and declarative abilities of the human mind, this issue
remains a strong underpinning of the entire workshop. Next-generation
analytical systems and models (such as those envisioned in the Automated
Performance Monitoring System - APMS and the Computer Aided Debriefing
Station - CADS) must be centered on the human user's needs and operate
as tools to support the analytical process. This workshop issue implies
the use of the human experts *and* automation tools in a systemic
process, not the automation of all analysis process steps.

[Develop *a* process, not *the* process.]{.underline} The task of
developing a process to support various users who require access,
integration or analytical capabilities relative to flight crew human
factors in accidents and incidents *should* be declared by the Federal
Aviation Administration as *only one of many* industry \"stepping
stones\" on the way to new and improved processes. The FAA\'s continued
support for other parties (working outside the participants of the
workshop) should be given recognition for their attempt to advance the
understanding of the aviation community relative to flight crew human
factors.

[Define measures of human factors data.]{.underline} Similar to the need
for a data dictionary in understanding and accessing various data
systems is the need for a clear description of the *measures* being used
to assess human performance in flight crews. Related to flight crew
activities, the need to clarify the measures precedes any legitimate
attempt to define collection and analytical processes whereby different
sources of data (collection activities and archiving) can be used to
create *super-sets* or *homogeneous data bases* for large-scale
analysis.

[Define measures of human factors models.]{.underline} A second trend
emerged in the workshop regarding the application of next-generation
data. Most round tables agreed that the industry has a lack of
understanding about the development of a hypothesis and methods of
experimental inquiry which attempt to measure the outcomes of flight
crew performance. Models and processes rely on the clarity of the
hypothesis to define what data should be collected to use the models and
processes. Most of the analytical models in use today are based on a
non-scientific prevailing opinions of post-event analytical methods. The
future of predicting and preventing flight crew performance will be
based on some method which has yet to be developed defining \"What
measures are reasonable and valid in *predicting* flight crew
performance?\"

Other significant discussions centered around:

¥ The need for the Federal Aviation Administration to support a limited
amount of *outside development of human factors research* relative to
predicting flight crew performance.

¥ Recognition of the *lack of verifiable and valid models* for use in
the industry has caused many operators to develop internal analytical
techniques which may or may not be useful.

¥ Developing a *next-generation analytical model* designed for measuring
flight crew performance should be explored in the FOQA program
(including supporting the tools of FOQA such as APMS, CADS and other
next generation flight crew analysis projects).

¥ The Federal Aviation Administration should attempt to *provide
specific guidance* in how to use the models which it might provide in
future data systems.

## Objective 3 Discussion Items

***Objective 3.** Define the conceptual interface between users, data
systems and analytical models/taxonomies.*

[Define the users as having fundamental interest in aviation operations
and safety.]{.underline} Further descriptions of the users of aviation
safety data emerged from the round tables, with the following list as
the primary types of aviation operations and safety-oriented users (in
frequency of occurrence order): Pilots, Technical/Professional
Certificated Airmen, Engineers, Lawyers, Managers, Journalists, and
Researchers (from academia, government and industry ).

[Develop a system for classifying the type of data
observation.]{.underline} Many of the data systems do not specify the
type of collection technique used to gather or archive the data
contained in the data system. This lack of a published classification
scheme interferes with usability, quality and \"trust\" of the value of
the data system contents. To move towards integrating many sets of
information gleaned from smaller data systems, the user\'s trust must be
maintained by ensuring that the data was collected for use in such
queries as flight crew human factors. Any interface into a data system
should identify the source(s) of the data and provide a detailed
description of the data for traceability. To support longitudinal study,
data systems should provide users with a notation (or some on-line
description) of the analytical methods to be used in the application of
source data. Round tables suggested this description as a technique to
reduce mis-use of questionable results.

[The user interface must be based on the types of users identified by
the round tables.]{.underline} A critical short-coming of the past data
access projects is the lack of utility and friendliness of the interface
to the types of users actually attempting to operate the systems.
Examples ranged from having a search process that requires a skilled
data base analyst, to the need for human factors experts \"on the spot\"
to interpret the results. Any interface must eventually lead to output
(a result) which is usable.

[On-line access of accident and incident data.]{.underline} Almost
unanimously, the workshop attendees agreed that all the accident and
incident data that is suitable/appropriate for use by qualified users
should be made available on-line. A strong point of concern was voiced
that this does not mean simply posting it to the internet. Developing a
secure interface for use by authorized personnel is needed in the near
future.

[Narrative descriptions of events are required.]{.underline} Access to
the record of an event should include a narrative description of the
event as documented by the investigative team or flight crew (in the
case of self-reporting), and both forms of reporting if available. The
user\'s interface should refer to all events in the system that may be
related according to the search/query parameters the user provided.
Additionally, a narrative search capability using key words *and*
context-level searching should be provided.

Other significant discussions centered around:

¥ The development of a process to ensure the *level of
de-identification* is appropriate to the user\'s authorization.

¥ *Protection of both the individual and the operator* must be
maintained if the intended purpose of safety data analysis and
dissemination is to be possible.

¥ *Data sharing schemes* and a system of \"depositors only\" access
should be explored. \[Editorial comment: Some legitimate users of this
data are not depositors. Future workshops will explore this further to
resolve "appropriate users" and applications of the data.\]

¥ Legislation and rules for the *protection of the users* and depositors
should be developed.

¥ The user\'s interface should strive to *prevent misinterpretation and
unintentional misuse* of the data and results by compelling correct
analysis and results.

¥ A *system level standardization* for depositors and users should be
implemented through a well-designed interface.

## Cautions and Pitfalls Heard Around the Workshop

¥ The use of subjective data for the analysis of flight crew human
factors is very difficult and should be carefully studied before being
implemented.

¥ The scope of the Federal Aviation Administration\'s role in the
furtherance of flight crew human factors data analysis should emphasize
a proactive role through innovative applied research projects and
support of the industry practitioners who seek to advance the sciences
of human factors in aviation.

¥ Inappropriate uses of the data must be prevented in the early stages
of any safety data \"network\" or participants may not be willing to
continue to be involved.

¥ A clear demonstration of commitment by the FAA to the operational and
fiscal stability of a safety data exchange (or other process operated by
the FAA) must be made clear before operators and researchers will make
contributions of data, fiscal resources or time.

## Summary

The workshop objectives provided a structure for the attendees on which
to focus their attention. In doing so, the templates elicited a large
set of points that need attention and further definition. The results of
the workshop clearly indicate a set of \"next steps\" and directions for
advancing the human factors analysis potential of the aviation
community. These next steps should be taken as a partnership between the
industry and the FAA.

Follow-on workshops will address the use of the integration tool, and
the clarification of data required to conduct meaningful analysis and
prediction of flight crew performance. Specific focus will be on the
next generation data systems and programs where users share data between
themselves in the interest of aviation safety.

#  human factors data PROJECT summary

The Flight Crew Accident and Incident Human Factors Project is a major
effort by the FAA's Office of System Safety to accomplish the primary
goal of the aviation community to substantially reduce aviation
accidents and incidents. A summary of the project activities conducted
to date and those planned is provided below.

## Background 

To address the high incidence of human factors related accidents and
incidents, the Office of Aviation Safety, predecessor to the Office of
System Safety, initiated the Human Factors Data Project in August 1993.
This project is included in FAA's Strategic Plan under Goal 3: Eliminate
accidents and incidents caused by human error. The goal of the Human
Factors Data Project is to identify causes of human error that
contribute to accidents and incidents in order to develop prevention
strategies. The general approach to satisfy this goal is twofold: (1)
Increase the Office of System Safety's capabilities to access,
integrate, and analyze human factors data relating to accidents and
incidents, and (2) Facilitate efforts of the aviation community to
improve aviation safety through the application of human factors data
and analytical models.

The initial focus of the project is to develop and implement a process
to access, integrate, and analyze flight crew human factors data
relevant to safety. In the future, the process will be extended to
include aircraft maintenance and air traffic control human factors data.
The Flight Crew Accident and Incident Human Factors Project has four
phases; background research, process development, process
verification/validation, and process implementation.

## Activities to Date

During FY94, teams from the Office of Aviation Safety performed a search
of available and emerging data systems that contain flight crew accident
and incident human factors information. An investigation was also
conducted on the models or techniques that are used or are being
developed to analyze these data. In addition, a significant sample of
human factors and safety experts were interviewed to determine what
human factors data and analyses are being used to understand the causes
and prevention of aviation accidents and incidents. The major finding
from the background research was that there are individual accident and
incident data systems that contain adequate information on "what
happened" but very little information on "why it happened." In order to
develop effective prevention strategies, as much contextual and causal
information as possible needs to be gleaned from all available sources
of accident and incident data. The major recommendation from the
background research was to develop a standardized and relatively easy
process for safety analysts, accident investigators, and human factors
professionals to access and integrate information contained in various
accident and incident data systems and perform analysis of this
information to determine the important human factors involved. A report
was prepared by the Office of Aviation Safety documenting the activities
and results of the background research.

Based on the background research, it was decided that an initial process
would be developed and demonstrated during FY95 that would permit an FAA
safety analyst to apply either of two human error models to an accident
database or an incident database to identify significant human factors
patterns and trends across accidents and incidents. Although there are
other relevant human error models that could be applied, the models
proposed by Norman-Reason and Laing-Graesser-Hemphill were selected for
the initial process. The databases selected were the National
Transportation Safety Board (NTSB) Accident/Incident Data System and
National Airspace Information Monitoring System (NAIMS)/Pilot Deviation
System (PDS). These databases are resident in the Office of System
Safety's National Aviation Data Analysis Center (NASDAC).

In order to associate the flight crew human factors information
contained in the selected accident or incident database with the
selected human error model in a context that will help the safety
analyst to interpret the results, a human factors framework or
categorization scheme was needed. The scheme proposed by Senders and
Moray, was selected for the initial process. Senders and Moray recommend
that all investigations of error should explicitly be included in their
four levels of taxonomy as a descriptive system before deeper analysis.
Generally, this taxonomy divides human error into two broad categories;
the situation and the consequences. The situation is comprised of two
main factors: Who and Why. The consequences are comprised of;
Where/When, and What. For the initial process, this general
categorization scheme will be decomposed into specific subcategories
that support the two selected human error models and relate directly to
data fields in the NTSB and NAIMS/PDS databases.

In order to efficiently and reliably connect different accident and
incident databases, human error models, and the users of safety and
human factors data, a computer-based integration tool is being
developed. Neuron Data's Smart Elements, introduced in 1994, was
selected as the software development environment. Smart Elements
supports rapid prototyping and meets our development requirement of
combining a graphical user interface (GUI), an object oriented and
rule-based expert system, and database interface. For our application,
objects represent data elements within the human error models and
accident/incident databases and the decision (If-Then-Else) rules
connect the objects and display the results in accordance with the
categorization scheme of Senders and Moray. The GUI permits real-time
creation of user-friendly input, intermediate, and output display
screens and the data access element makes access to a wide variety of
outside databases transparent to the user.

For the initial process, the user will select either or both the NTSB
accident or the NAIMS/PDS database and one of the two human error
models. For each query of the database(s), the output derived will show
the number of accidents or incidents in the database(s) that contained a
human error as defined by either Norman-Reason or
Laing-Graesser-Hemphill and interpreted by human factors professionals
assigned to the project. The accident or incident report numbers
associated with each human error can be accessed if desired. By clicking
on any of the report numbers, the analyst can call up the narrative
extract from the report to verify the presence of the human error and to
understand more about the context and causality of the accident or
incident. The entire process, from entering a query to receiving an
output, takes only a matter of a few minutes. Currently it takes days to
obtain the same information and that's assuming the analyst has a
working knowledge of the selected database and human error model.

In June 1995 the Office of System Safety sponsored a 2 1/2 day workshop
on Flight Crew Accident and Incident Human Factors in McLean, VA. The
purpose of the workshop was to hear from potential users of the process,
i.e., human factors professionals, safety analysts, and accident
investigators, concerning their human factors data collection and
analysis activities and needs. Sixty representatives from government,
industry, and academia attended the workshop. Inputs from the workshop
will be used to refine the initial process and prioritize FY96 project
activities within the Office of System Safety. Attendees agreed that the
workshop was an important milestone towards standardizing human factors
data and analysis requirements to support accident and incident
prevention. The workshop also provided a unique opportunity for those
attending to become familiar with other related human factors and safety
data initiatives being conducted within the aviation community.

## Activities Planned 

During the 4th Quarter of FY95, Version 1 of the integration tool
software, which will be used to demonstrate the operation of the initial
process or "proof of concept," will be completed.

Version 1 will contain decision rules to connect a subset of the objects
or data elements contained within the two selected human error models
and accident and incident databases. The queries permissible will
include time period, airspace user type, referenced to flight crew
member, aircraft make/model, phase of flight, and weather conditions.
Version 1 will only access the fixed fields within the NTSB and
NAIMS/PDS databases. Version 1 software will run on a stand-alone PC
with the NTSB accident and NAIMS/PDS incident data loaded into the hard
disk. Also during this period, a demonstration of Version 1 will be
provided to FAA safety and human factors management and technical
personnel, the proceedings from the June Workshop will be distributed,
and a bulletin board will be established on the Internet to permit those
attending to continue the dialog begun at the workshop. A report
providing a detailed definition of the initial process and the prototype
integration tool will be prepared.

During the 1st quarter of FY96, operational testing of Version 1 of the
integration tool software will be conducted by the project team and
selected individuals from government, industry, and academia. During the
first 2 quarters of FY96, Version 2 will be developed which will contain
additional decision rules to associate the remaining objects or data
elements contained within the two selected human error models and
accident and incident databases. Version 2 will also permit remote users
to access the NTSB accident and NAIMS/PDS databases in the NASDAC via
Internet with textual retrieval capability to process the narrative data
contained in the accident and incident reports. Operational testing of
the additional capabilities will be accomplished by the project team
with selected participants from the aviation community. Version 2
software and documentation will be completed during the 3rd quarter of
FY96.

The Office of System Safety is planning to have a contractor outside the
project team conduct an independent verification and validation (V&V) of
Version 2 of the integration tool software. A V&V plan will be prepared
during the 2nd quarter of FY96, V&V will be completed during the 3rd
quarter, and a V&V report will be prepared during the 4th quarter.
Another user workshop will be held during the Spring of 1996 to review
the design and operation of the initial process based on Version 2 of
the integration tool software and to discuss the preliminary results of
the independent V&V. User requirements obtained from the workshop and
the changes identified during the V&V will be incorporated into Version
2 of the integration tool software. Version 2.1 software and
documentation will be released by the end of FY96, at which time, the
Office of System Safety will begin using Version 2.1 to access,
integrate, and analyze flight crew accident and incident human factors
data.

It is expected that as additional databases (e.g., ASRS) are
incorporated into the NASDAC, the Office of System Safety will integrate
them into the process. Human Factors data from the Flight Operational
Quality Assurance (FOQA) Program, Automated Performance Monitoring
System (APMS), the Advanced Qualification Program (AQP), and emerging
industry safety programs will also be integrated as these data are made
available. As resources permit additional human error models and
statistical analysis techniques will be evaluated and integrated into
the process. It is also expected that the integration tool will be
enhanced to provide the user with references to sources of human factors
research data to better understand the causes and prevention of the
human errors identified. Throughout this period, the Office of System
Safety will maintain contact with the users of the process to discuss
improvements in collecting and analyzing flight crew accident and
incident human factors data and the means of implementing them.

# LIST OF ATTENDEES

Ludwig Benner

*Ludwig Benner & Associates*

12101 Toreador Lane

Oakton, VA 22124

703-758-4800

Ben Berman

*National Transportation Safety Board*

490 L\'Enfant Plaza East, SW

Washington, DC 20594

202-382-6574

bermanb@ntsb.gov

\* Jim Blanchard

*The MITRE Corporation*

CAASD/MW-W195

7525 Colshire Drive

McLean, VA 22102

703-883-6758

jblancha@mitre.org

Sue Bogner

*Food & Drug Administration*

1350 Piccard Road, HFZ 230

Rockville, MD 20850

301-594-3877

msb@fdadr.cdrh.fda.gov

Joseph Bracken

*AirLine Pilots Association*

P.O. Box 1169

535 Herndon Parkway

Herndon, VA 22070

703-689-4208

\* Jim Cieplak

*The MITRE Corporation*

CAASD/MS-W273

7525 Colshire Drive

McLean, VA 22102

703-883-5292

jcieplak@mitre.org

Sherry Chappell

*NASA/ASRS*

NASA Ames Research Center

MS 262-4

Moffett Field, CA 94035

415-604-6909

sherry@cos.arc.nasa.gov

Tom Chidester

*American Airlines, Incorporated*

P.O. Box 619617

MD 891 GSWFA

Dallas/Ft. Worth Airport, TX 75261

817-967-5182

Rebecca Chute

*NASA Ames Research Center/San Jose State University*

MS 262-4

Moffett Field, CA 94035-1000

415-604-0771

rchute@gmgate.arc.nasa.gov

Linda Connell

*NASA/ASRS*

625 Ellis Street

Suite 305

Mountain View, CA 94043

415-604-3969

linda_connell@abbott.arc.nasa.gov

Rich Dieckhans

*Federal Aviation Administration*

ASA-100, Rm 2322

400 7th Street, SW

Washington, DC 20590

202-366-2125

rdieckhans@mail.hq.faa.gov

**\* Flight Crew Human Factor Project Team**

\* Nancy Dolan

*Galaxy Scientific Corporation*

2500 English Creek Avenue

Building 11

Egg Harbor Township, NJ 08234

609-645-0900

galsci@acy1.digex.net

Jack Fearnsides (Opening Remarks)

*The MITRE Corporation*

7525 Colshire Drive

McLean, VA 22102

Mark Fischer

*AeroKnowledge, Incorporated/ASRS*

2425 Pennington Road

Trenton, NJ 08638

609-737-9288

pa24aero@ao1.com

Chuck Fluet

*Federal Aviation Administration*

400 7th Street, SW

Washington, DC 20590

202-366-4466

charles_fluet_at_asa@mail.hq.faa.gov

Stephanie Frank

*Battelle/ASRS*

625 Ellis Street

Suite 305

Mountain View, CA 94043

415-969-3969

stephanie_frank@gmgate.nasa.arc.gov

Larry Ganse

*Northwest Airlines*

5101 Northwest Drive

MS-N7180

St. Paul, MN 55111-3034

612-726-0916

\* Wrensey Gill

*Federal Aviation Administration*

400 7th Street, SW

Washington, DC 20590

202-366-6166

Curt Graeber

*Boeing Commercial Airplane Group*

P.O. Box 3707

MS 9W-AA

Seattle, WA 98124-2207

206-237-1610

grarcb00@ccmail.iasb.ca.boeing.com

Scott Griffith

*American Airlines Safety Action Partnership*

960 Breakwater Drive

Annapollis, MD 21403

410-263-8235

John Hansman

*Department of Aeronautics & Astronautics*

MIT Aeronautical Systems Laboratory

Rm 33-115

77 Massachusetts Avenue

Cambridge, MA 02139

617-253-2271

rjhans@mit.edu

Steve Harper

*CSERIAC Program Office*

AL/CFH/CSERIAC

Building 248

2255 H Street

Wright-Patterson AFB, OH 45433

513-255-4842

sharper@desire.wright.edu

\* Christopher Hart (Opening Remarks)

*Federal Aviation Administration*

800 Independence Avenue, SW

Washington, DC 20591

202-267-3611

**\* Flight Crew Human Factor Project Team**

Chuck Hedges

*Federal Aviation Administration*

400 7th Street, SW

Washington, DC 20590

202-366-4466

\* Lloyd Hitchcock

*Hitchcock & Associates*

2520 Pleasant Hill Road

Hatboro, PA 19040

215-675-2419

71250.1556@compuserve.com

Mark Hofmann (Opening Remarks)

*Federal Aviation Administration*

800 Independence Avenue, SW

Washington, DC 20591

202-267-7125

Beverly Messick Huey

*National Research Council, Marine Board*

Suite HA250

2101 Constitution Avenue, NW

Washington, DC 20418

202-334-3119

bhuey@nas.edu

Pierre Huggins

*AirLine Pilots Association*

535 Herndon Parkway

P.O. Box 1169

Herndon, VA 22070

203-689-4211

Sharon Jones

*NASA-UT*

P.O. Box 981064

Houston, TX 77098-1064

sharon.jones@mail.utexas.edu

George Kaseote

*Federal Aviation Administration*

800 Independence Avenue, SW

Washington, DC 20591

202-267-8541

george_kaseote_at_air100@mail.hq.faa.gov

Thomas Lilli

*Air Line Pilots Association*

P.O. Box 1169

535 Herndon Parkway

Herndon, VA 22070

703-689-4208

George Lyddane

*Federal Aviation Administration*

3960 Paramount Boulevard

Lakewood, CA 90712

310-627-5206

Callum MacGregor

*British Airways, Safety Services*

The Compass Centre (S742)

P.O. Box 10

Heathrow Airport

Hounslow TW-6 2JA

0815622925

Vincent Mancuso

*Delta Air Lines*

Department 024

Atlanta Hartsfield International Airport

Atlanta, GA 30320

404-209-1654

70232.1005@compuserve.com

Vincent Mellone

*Battelle/ASRS*

625 Ellis Street

Suite 305

Mountain View, CA 94043

415-969-3969

**\* Flight Crew Human Factor Project Team**

Rowena Morrison

*Battelle/ASRS*

625 Ellis Street

Suite 305

Mountain View, CA 94043

415-969-3969

rowena_morrison@abbott.arc.nasa.gov

\* Eric Neiderman

*Galaxy Scientific Corporation*

2500 English Creek Avenue

Building 11

Egg Harbor Township, NJ 08234

609-645-0900

Lou Nemeth

*USAir*

P.O. Box 7543

Charlotte, NC 28241-7543

704-588-0863

ususa32b@ibmmail.com

Hop Potter

*Federal Aviation Administration*

800 Independence Avenue, SW

Washington, DC 20591

202-267-3723

hop_potter_at_afs200@mail.hq.faa.gov

\* Phil Prasse

*Records Management Systems, Inc.*

11166 Main Street

Suite 200

Fairfax, VA 22030

703-218-1400

Larry Randall

*Federal Aviation Administration*

400 7th Street, SW

Washington, DC 20590

202-366-6003

Ira Rimson

*Validata Corporation*

P.O. Box 1015

Springfield, VA 22151

703-978-2944

73700.3376@compuserve.com

Mark Rodgers

*Civil Aeromedical Institute (CAMI)*

P.O. Box 25082

Oklahoma City, OK 73125

405-954-6848

mark_rogers@mmacmail.jccbi.gov

\* Barbara Sada

*Federal Aviation Administration/NASDAC*

400 7th Street, SW

Washington, DC 20590

202-366-6003

Nadine Sarter

*The Ohio State University*

210 Baker Systems Building

1971 Neil Avenue

Columbus, OH 43210-1210

614-292-6287

nsarter@magnus.acs.ohio_state.edu

\* Stew Schreckengast

*Advanced Systems Technology Incorporated*

2121 Crystal Drive

Suite 514

Arlington, VA 22202

703-218-1400

Ronald Simmons

*Federal Aviation Administration*

400 7th Street, SW

Washington, DC 20590

202-267-7058

rsimmons@mail.hq.faa.gov

**\* Flight Crew Human Factor Project Team**

Barry Strauch

*National Transportation Safety Board*

490 L\'Enfant Plaza East, SW

Washington, DC 20594

202-382-6642

strauch@ntsb.gov

Robert Sumwalt

*Air Line Pilots Association*

P.O. Box 1169

535 Herndon Parkway

Herndon, VA 22070

800-442-9007 (7216)

Michal Tamuz

*University of Texas*

School of Public Health

2100 Herman Presler

Houston, TX 77030

713-792-4364

tamuz@utsph.sph.uth.tmc.edu

Stan Trollip

*SR Trollip & Associates*

2200 East 22nd Street

Minneapolis, MN 55404

612-332-5725

70732.2431@compuserve.com

John Wise

*Embry Riddle Aeronautical University*

Center for Aviation/Aerospace Research

Embry Riddle Aeronautical University

Daytona Beach, FL 32114

904-226-6384

wise@db.erau.edu

\* John (Jack) Wojceich

*Federal Aviation Administration*

400 7th Street, SW

Washington, DC 20590

202-366-6431

jwojciech@mail.hq.faa.gov

**\* Flight Crew Human Factor Project Team**

# APPENDIX A - POSITION PAPERS

**\*Editor's Note: Some papers submitted were not included due to their
late submission.**

## SUMMARY OF POSITION PAPERS - Lloyd Hitchcock

**Preface**

The attempt to summarize the position papers submitted by workshop
attendees for consideration posed a significant challenge. Each of the
position paper authors presented points of consideration that were both
important and insightful. However, their very diversity makes any
attempt at general summarization extremely difficult. While an earnest
effort was made to capture the key concepts from each paper in the
summary which follows, there may be some ideas which were inadvertently
left out; for this the authors are given sincere apologies. While the
source of most of the ideas included in the summary are cited, it did
not prove to be practical to identify all the material included through
the routine use of paragraph separation or the use of quotation marks.
The original unedited and unabridged position papers remain the best
source of comprehensive coverage and proper attribution of the author's
ideas. In the course of searching for a framework which could serve as a
basis for organizing the varied contributions of the position paper
authors in a way that would facilitate the deliberations of the
workshop, the following categories were chosen:

-   *Where are we now?*

-   *How did we get here?*

-   *Where do we want to be (Objectives)?*

-   *How do we get there?*

-   *What obstacles must we overcome enroute?*

-   *How will we know when we have arrived?*

While the labels which have been given to these categories are
admittedly somewhat informal, it is hoped that both their basic meanings
and their potential significance to the workshop goals are apparent.

**Where Are We Now?**

Current attempts to study the contributions of the human component to
overall aviation system safety are fraught with difficulty. Much has
been made of the fact that from 2/3rds to 3/4ths of our aviation mishaps
involve some element of human failing.

One aspect which fosters confusion is the extreme breadth of the many
system elements which are frequently included under the rubric of "Human
Factors" as pointed out by Benner and Rimson (1995), the Federal
Aviation Administration's current definition of Human Factors is:

> *"\....a multidiciplinary effort to **generate** and **compile**
> information about human capabilities and limitations and **apply**
> that information to equipment, systems, facilities, procedures, jobs,
> environments, training, staffing, and personnel management for safe,
> comfortable, effective human performance."*
>
> (AAR-100 Pamphlet, March 1995)

Many of the databases used to identify human error were originally
developed to simply document the event without consideration to accident
prevention. Thus, many of the current safety data collection practices
lack methodological standardization and, as a consequence are not
susceptible to controls on either their validity or their quality.
Typical investigation methodologies tend to accept, uncritically, all
kinds of data inputs and formats with little attempt at quality control,
verification, corroboration, or scientific validation. There exists no
viable methodology for normalizing, or otherwise recasting, anecdotal or
episodic "rare event" data into a scientific format which could
allow/support their use in statistical inferences (Benner and Rimson,
1995).

Furthermore, as pointed out by Corrie (1995), there are vast differences
in the investigation cultures and capabilities between the 183
Contracting States of ICAO which result in marked differences in both
the nature and utility of the human factors information which they
generate.

**How Did We Get Here?**

The current mishap investigation approaches, objectives, and
methodologies were, for the most part, originated in the 1930's by the
Civil Aeronautics Board. They evolved to fulfill statutory and
regulatory requirements derived from legalistic issues of culpability
and liability (Benner and Rimson, 1995). The current negative
investigational focus on "what went wrong", with its implicit attention
on "who's to blame", serves to discourage candid witness participation
in a positive process to develop accident prevention strategies,
particularly in today's climate of litigation (Benner & Rimson, 1995).
Furthermore, while accident reports usually depict "what" happened and
"when", in too many instances they stop short of fully explaining the
"why" and "how" which led to the accident (Sumwalt, 1995). Callum
MacGregor, Senior Air Safety Investigator with British Airways, also
emphasizes the need for "*an open, penalty-free, culture* \[to\]
*inspire staff to report high quality safety information*".

Much potentially valuable data on "incidents" has been routinely lost
because of a common, though mistaken, attitude that such events as
inappropriate touchdown point or improper rotation rate are too minor to
be perceived as significant (Ganse, L. R, 1995). Graeber (1995) concurs,
noting that "*many events which would be of interest to a Human Factors
investigator are not reported because pilots either don't see them as
potential infractions or don't think they are important enough to bother
filling out the paperwork*". In addition, incident data provide
first-hand information from participants directly involved at a level of
detail often not available following an accident. Incident data are
"ecologically valid" in that they were generated within the operational
environment and are not merely laboratory phenomena. However, such data
are often difficult to validate and face the potential hazards posed by
reporter biases (Chappell, 1994). Errors that are near misses
(incidents) should be considered in error analyses since what proves to
be a near miss for one may well be a fatal error for others (Bogner,
1995).

The study of aviation errors has also been hampered by the tendency to
view problems as residing either in the human or in the engineering
world. A more enlightened approach would recognize the synergism between
the man and machine which sees errors as breakdowns not only in
communications between humans but between man and machine as well (Moll
van Charante et al., 1992, Sarter and Woods, 1994, Wiener, 1989).

We have tended to focus on the safety implications of flight crew errors
because they are listed more frequently as primary accident causes. This
approach ignores the well recognized multiple-cause etiology of most
accidents and incidents (Greaber, 1995). Ganse (1995) cites the
importance of a consideration of the interactions between flight crew,
flight attendants, mechanics, station personnel, air traffic
controllers, and airline policy and procedure mandates to a full
understanding of the origin of an accident. Wise and Wise (1995)
compellingly illustrate the criticality of this interdependence by
citing the case of the loss of a DC-10 because a cargo door, which could
not be locked properly by the unskilled ground crew (because of the
complexity of the required operation), blew open in flight. This event,
coupled with a design deficiency which prevented adequate pressure
compensation when the door was lost, the placement of control cables
under the section of floor that buckled from the over pressure, and the
resultant jamming of the control system created a sequence of events
whose end product was a major accident which cost many lives.
Unfortunately, most human factors specialists lack the operational
background needed to assess many of the more subtle nuances of such
interfaces. With such an interactive pattern in mind, Corrie (1995)
reminds us that the "*causes of human error can be complex and any
examination must include the entire scenario in which the accident
occurred*". To really understand what caused a particular mishap, an
investigation must dig beneath the surface - look beyond the obvious -
to determine what systemic factors may have served as the root causes of
the ultimate error. Investigations should include management philosophy
and practice, regulatory influence, and the corporate culture (Sumwalt,
1995). Additionally, seemingly unrelated issues, such as corporate
policies, family relationships and language/communication exchange are
difficult to qualify or quantify.

**Where Do We Want To Be?**

We need to establish standardized techniques of data collection which
yield comprehensive and valid answers to the questions "*What was
supposed to happen*?" and "*What actually did happen*?" These techniques
should be fully applicable to both accidents and to the potentially more
data rich investigation of incidents. The techniques adopted must be
compatible with, and support, a rapid, user friendly methodology for the
exchange of data and information between all parties and agencies with
an interest in the human factors aspects of aviation safety.

The aviation safety community must come to an agreement on the logical
context, or model(s), which will be used to proceed from data
acquisition and summarization to the development of causal inferences
and, subsequently, to recommendations for techniques for enhanced
safety. The model(s) selected must be equally sensitive to issues of
"what went wrong?" and "what went right?" if they are to generate
meaningful insights into how to prevent similar incidents in the future.

In short, those interested in establishing ways in which the aviation
community can successfully reduce the human error component of aviation
incidents and accidents must work together to reach a well documented
prevailing opinion on the answers to at least the following primary
issues and the questions they generate:

Data Issues:

-   What is an "error"?

-   What data should be collected?

-   How should the data be collected?

-   How do we control data quality?

-   How do we validate the data?

-   How do we code the data?

-   How do we standardize these processes?

-   How do we handle questions of confidentiality and liability?

Analysis Issues:

-   How do we generate hypotheses?

-   What framework (models) do we use to organize our analyses?

-   How do we determine the statistical significance of "rare events"?

-   Who adjudicates disagreements in analytic interpretation?

Information Exchange Issues:

-   How do we integrate existing data bases?

-   How do we meet the needs of almost 200 separate agencies?

-   How do we facilitate and support rapid data transfer?

-   How do we manage the combined data base and who does it?

Perhaps the greatest challenge still remains. The aviation human factors
community must then build the "bridges" needed to transform analytic
exercises into compelling arguments for what may well be the
significant, and potentially costly, changes in flight equipment,
training, personnel, practices, procedures, and operational climate that
are necessary requisites to a meaningful impact upon future safety.

**What Do We Have To Do To Get There?**

Simply developing a new arsenal of analytic models is not sufficient to
meet the safety research challenge. These models must be made compatible
with, and integrated into, the analysis process itself. There are a
variety of human performance and human error models that have become
popular among aviation human factors professionals, These include
Norman's separation of errors into slips, lapses, and mistakes Jim
Reason's concept of latent failures and "resident pathogens",
Rasmussen's concepts of skill-based, rule-based, and knowledge-based
behavior, and Edwards' SHEL (**[S]{.underline}**oftware,
**[H]{.underline}**ardware, **[E]{.underline}**nvironment,
**[L]{.underline}**iveware) model (Bogner, 1995; Graeber, 1995; Sumwalt,
1995). Ongoing efforts to identify the strengths and weaknesses of
these, and all other similar approaches, should be evaluated as part of
the development of a prevailing opinion regarding the model to be used
in forthcoming investigations. In addition, the wealth of experience of
both NASA and Battelle Memorial Institute, gained through their
cooperative efforts to develop a human factors taxonomy and attempts to
apply human error models and human performance models to the data
contained in the Aviation Safety Reporting System (ASRS) data bases,
should be used as a source of practical counsel in the development of
any new model(s) (Morrison, 1995). Effort should be made to draw upon
the NTSB's experience in applying extensive safety issue taxonomies to
accident data (Ellingstad and Mayer, 1995). The taxonomy of data
elements should be focused, unambiguous, and designed for ease of coding
yet possess the flexibility needed to allow for elaboration and the
coding of unique circumstances (Bogner, 1995).

Benner and Rimson (1995) argue that successful reduction in human
factors related accidents mandates a change in investigation philosophy
from a focus on "post-accident" analyses which emphasize what went wrong
("What went wrong to 'cause' the accident?") to a study of
"pre-accident" incidents which can yield data on what was done right
and, as a consequence, kept an accident from ultimately unfolding ("What
went right to prevent it?"). Redirecting attention from accidents, which
identify only operational failures, to incidents, which identify both
operational failures and successful strategies, is seen as essential to
reducing the likelihood of future accidents. The current "sound of one
hand clapping" tendency to look only at what human failure(s) led to a
given accident tells us all too little about how to avoid such accidents
in the future. This position is supported by MacGregor (1995) who states
that "*the approach is not just to gather data but to use empirical data
in a predictive manor to identify future problems and take remedial
action before the incident occurs*".

More work must be done to separate issues of accident cause, effect, and
prevention. For example, automation is frequently proffered as a remedy
for workload induced human error(s). However, as pointed out by Potter
(1995), errors in the keystroking activities associated with inputs to
the Flight Management Systems (FMSs), touted as palliatives for human
failings, tend to increase as the pilot workload they are designed cure
increases. Such a tendency toward error could be critical since one FMS,
currently certified by the FAA, requires 164 keystrokes to program a
single IFR approach (Kasseote and Lyddane, 1995). Thus, the "cure" could
prove to be "cause" as well. What is needed is a philosophy of
automation that is based not on anecdote and assumption but on hard data
gathered on both the specific features of automation hardware and
software and the subtleties of automation interaction with other cockpit
processes. This philosophy must be incorporated into a clear statement
of the issues, and their supporting standards, which govern the testing
associated with the certification of automated systems.

British Airways has initiated the British Airways Safety Information
System (BASIS) which has now become the medium for the world-wide
exchange of safety information between airlines of all sizes, aviation
authorities and aircraft manufacturers. Based upon data gathered in a
non-punitive collection climate, the information contained in BASIS
provides the user with a targeted risk management matrix, and provides
the user with user-defined graphic presentations of analyses of trends
in accident/incident quantity, probability, delay impact and cost. This
information is both managed and distributed by a reliable,
user-friendly, paper-free computer system which is networked to all
sections of the Company for use by line managers. In addition, the BASIS
has been interfaced with other aviation agencies such as IATA, ATA, UK
Air Safety Committee, ALPA, and the European Regional Airlines
Association. The experience and expertise of such agencies as the Crew
System ERgonomics Information Analysis Center (CSERIAC),
Wright-Patterson Air Force Base, should also be called upon to aid in
the development of data exchange methodologies and the human-computer
interface (HCI) techniques, such as Graphical User Interfaces (GUIs)
which can serve to make the exchange methodologies more user friendly.
The exchange methodology developed should be fully integrated with the
approach developed by the international aviation safety community
(Graeber, 1995).

Those concerned with the role of human error in aviation safety must
develop a philosophy which recognizes such errors as a "symptom" rather
than a "cause" (Sarter and Woods, 1995). When viewed as a symptom of a
breakdown in crew coordination, communication, organizational pressure,
or the introduction of faulty technology, the emergence of an error
should call forth more in-depth investigations of how the total system
involved both functions and malfunctions (Rasmusson, et al., 1987;
Reason, 1990; Hollnagel, 1993). Such an approach should be both future
and prevention oriented and go beyond phenotypical description of the
processes involved in the evolution of a single, potentially unique,
accident/incident and would look for the genotypical patterns in safety
data.

What Obstacles Can We Expect To Have To Overcome On The Way?

Currently, there exist significant disagreements over even the basic
nature of the sources of human error and, as a consequence, differing
interpretations of how best to reduce their likelihood. As pointed out
by Woods and Sarter (1995), "*the label human error is prejudicial and
unspecific and\...retards rather than advances our understanding of how
complex, man-machine systems fail and how the same operator behavior can
contribute to both success and failure*".

We must develop a better and more complete understanding of the human
factors related causes of accidents and incidents than can be derived
from the current accident data bases which capture only a small portion
of the problems occurring in the National Airspace System. Such programs
as the *Airline Safety Action Partnerships* should be explored as a
possible way to broaden the scope of safety data collection (Chidester
and Griffith, 1995; McGarry, 1995).

Currently, task-oriented human factors information about accident
scenarios is often missing or unusable in transportation accident
databases. This category of information is often overlooked because the
investigators involved lack an adequate understanding of, and
sensitivity to, human factors issues. Furthermore, these data are often
not gathered because they do not leave the same kind of tangible
physical evidence as that provided by damaged structural components
(Ellingstad and Mayer, 1995).

Techniques must be developed to generate estimates of the cost of the
human factors analysis program to the airline industry, the government,
and the passenger. Without such a foundation of fiscal justification, it
will be difficult, if not impossible, to successfully implement any new
safety initiative.

Perhaps the most formidable hurdle which must be overcome is the fact
that the very nature of scientific inquiry argues against the ability to
extrapolate knowledge derived from the past to the vagaries of the
future. The scientific method rests upon the empirical testing of
hypotheses and, as Wise and Wise point out (1993, 1994), "How do we
formulate reasonable hypotheses about the future?". While it is clear
that any major change(s) in aviation technology and its operational
environment will serve to render the knowledge gained from the past less
useful, it is reasonable to assume that, for the immediate future at
least, pilots and support crews will continue to have to contend with
many, if not all, of the problems and challenges that plague them today.

**How Will We Know When We Get There?**

None of the position papers submitted for this workshop addressed
directly the issue of quantification of progress. However, in view of
today's heightened concern for fiscal prudence, it would seem
appropriate that some thought be given to setting criteria which would
define when the system has reached maturity, when it still has a way to
go, and when it has gone beyond the minimum necessary to insure
successful implementation. Statistical minimums for levels of
significance should be considered for such attributes as validity and
reliability. How will we know when we have moved far enough away from a
punitive investigation philosophy toward a positive climate of free
exchange of accident avoidance data collection? Is it possible to
establish benchmarks which can assess the adequacy of the taxonomy
established and/or the model(s) selected as the analytic framework? How
much predictive power is enough? Sooner or later it is highly likely
that such issues will require, perhaps even demand, answers.

**Bibliography**

Bogner, M. S., 1994, *Human Errors in Medicine: A Frontier for Change*.
In Bogner, M. S. (Ed), *Human Error in Medicine*, Lawrence Erlbaum
Associates, Inc., Hillsdale, New Jersey

Bogner, M. S., 1995, *Human Error Identification: Designing for the
Users*, Food and Drug Administration.

Bracken, J., 1995, *Human Factors Symposium,* Airline Pilots
Association.

Brenner, L. Jr., & Rimson, I. J., 1995, *Incident Investigation:
Paradigm Shifts toward Exploiting Successful Human Factors Intervention
Strategies to Prevent Aircraft Accidents.*

Chappell, S. L., 1994, *Using Voluntary Incident Reports for Human
Factors Evaluations*, In Jonston, N., McDonald, N., and Fuller, R.
(Eds), *Aviation Psychology in Practice*, Ashgate, Aldershot, England.

Chidester, T. R. (American Airlines), and Griffith, K. S. (Allied Pilots
Association), 1995, *Potential for Human Factors Accident/Incident
Research through Airline Safety Action Partnerships (ASAPs)*.

Corrie, S. J., 1995, *ICAO Views on the Application of Human Error
Models to Accident and Incident Data*, Chief, Accident Investigation and
Prevention Section, ICAO.

Ellingstad, V., and Mayer, D., 1993, *Assessment of Human Error from
Transportation Accident Statistics,* National Transportation Safety
Board, Washington, D.C.

Fischer, M., 1995, *The Benefits of CD-ROM for Aviation Safety Data
Distribution*, AeroKnowledge, Inc.

Ganse, Capt. L. R., 1995, *Practical Issues Impacting Human Factors
Investigations at the Airline Level*, Northwest Airlines.

Graeber, R. C., 1995, *FAA Workshop on Flight Crew Accident and Incident
Human Factors, Position Paper*, Boeing Commercial Airplane Group.

Harper, S., 1995, *The Human Factors Issues Associated with Flight Crew
Accidents and Incidents,* Crew System ERgonomics Information Analysis
Center, Wright-Pattern AFB, Ohio.

Hollnagel, E., 1993, *Human Reliability Analysis: Contex and Control*,
Academic Press, London.

Kaseote, G., and Lyddane, G., 1995 *Human Factors Considerations to Be
Used By Certification Test Pilots in the FAA Approval of Certain
Electronic Systems in Various Aircraft.* FAA Test Pilots.

MacGregor, C., 1995, *Towards a Safety Superhighway*, British Airways.

McGarry, R., 1995, *Partnership for Safety Programs,* FAA Southwest
Region Headquarters, Fortworth, Texas.

Mangold, S. and Eldredge, 1995, *Flight Management System Information
Requirements,* Battelle, Columbus, Ohio.

Moll van Charante, E., Cook, R. L., Woods, D. D., Yue, L., and Howie, M.
B., 1992, *Human Computer Interaction in Context: Physician Interaction
with Automated Intravenous Controllers in the Heart Room*. Fifth
Symposium on the Analysis, Design, and Evaluation of Man-Machine
Systems, The Hague, Netherlands.

Morrison, R., 1995, *Issues Related to Use of ASRS Incident Data With
Human Performance and Error Models,* Battelle, Mountain View,
California.

Potter, H, 1995, *Needed: A Systematic Approach for a Cockpit Automation
Philosophy*. Air Carrier Training Branch, Flight Standards Service, FAA.

Rasmussen, J., Duncan, K., & Lepat, J. (Eds.), 1987, *New Technology and
Human Error*, Wiley, Chichester, England.

Reason, J., *Human Error*, Cambridge University Press, Cambridge,
England.

Sarter, N. B., & Woods, D. D., 1995, *Strong, Silent, and
Out-of-the-loop: Properties of Advanced (Cockpit) Automation and Their
Impact on Human-Automation Interaction*,

Technical Report No. TR-95-01, Cognitive Systems Engineering Laboratory,
Ohio State University.

Sumwalt, R. L. III, 1995, *The Quest for Aviation Safety's Holy Grail:
Finding Underlying Causes of Accidents and Incidents* or *If You Really
Want to Improve Aviation Safety, You Must First Identify Systemic
Problems*, Working Group for Human Factors in Accident Investigation,
Air Line Pilots Association.

Woods, D. D. & Sarter, N. B., 1995, *Learning From Accidents: The Need
To Look Behind Human Error*, Ohio State University.

Wiener, E. L., 1989, *Human Factors of Advanced Technology (Glass
Cockpit) Transport Aircraft*, Technical Report No. 177528, NASA Ames
Research Center, Moffett Field, California.

Wise, J. A., and Wise, M. A., 1993, *Basic Considerations in
Verification and Validation*, In Wise, J. A., Hopkin, V. D., and Stager,
P. (Eds), *Verification and Validation of Complex Systems: Human Factors
Issues*, Springer-Verlag, Berlin.

Wise, M. A., and Wise, J. A., *On the Systems Approach to Certify
Advanced Aviation Technologies*, In Wise, J. A. Garland, D. J., and
Hopkin, V. D. (Eds), *Human Factors Certification of Advanced Aviation
Technologies*, Embry-Riddle Aeronautical University Press, Daytona
Beach, Florida.

Wise, J. A., and Wise, M. A., 1995, *In Search of Answers Without
Questions: Or, How Many Monkeys at a Typewriter Will It Take\...*,
Embry-Riddle Aeronautical University & the University of Central
Florida.

## DATABASES FOR EFFECTIVE SAFETY ENHANCEMENT - Air Line Pilots Association

We must work with our databases to identify incident trends in order to
prevent future accidents. Past studies have concluded that human factors
issues are prevalent in nearly every accident. We must recognize those
issues prior to experiencing a catastrophe in order to avoid accidents.
This can be accomplished through the use of air safety databases. These
databases may contain not only basic information about incidents but
also details of past accidents, information regarding mechanical
discrepancies, and uneventful hazards avoided by luck and/or skill.

Many organizations have established and maintain databases to track and
analyze occurrences of specific interest. Efforts must be made to
standardize the information recorded with the goal of sharing database
information among many organizations. This would result in the
development of a master database which could then be used to identify
trends by many organizations. Of course, prior to the implementation of
such a master database, sever hurdles must be overcome, including
confidentiality, security, and legal issues.

**Introduction**

The Air Line Pilots Association was founded in 1931 under great secrecy,
with fear that discovery of a pilotÕs membership in the Association
could cost the pilot his job. Now, membership is displayed proudly with
the ALPA pin, and we now have membership in 38 airlines of over 42,000
pilots. Since its early years, our AssociationÕs motto has been,
ÒSchedule with Safety.Ó

The Air Line Pilots Association has an extensive safety organization
that works to support each individual airline as well as the airline
pilot in general. Each airline has a Central Air Safety Chairman (CASC)
that has been designated by that airlineÕs elected officers. This CASC
forms a safety network within his or her airline that is responsive to
their membersÕ individual needs. Also, the CASC participates in meetings
with the Central Air Safety Chairs from the other airlines represented
by ALPA, to discuss and make recommendations for safety issues common to
all airlines.

At the top of this group of Central Air Safety Chairs is the designated
Executive Central Air Safety Chairman. As with each Central Air Safety
Chair, the Executive Central Air Safety Chair (ECASC) is also a
full-time line pilot. Within the scope of the ECASC is the formation of
Technical Committees that address issues of national scope. These
include the following committees:

Accident Investigation Board

Accident Survival Committee

Air Traffic Control Committee

Airport Standards Committee

Airworthiness & Performance Committee

All Weather Flying Committee

Aviation Weather Committee

Charting and Instrument Procedures (CHIPS) Committee

Dangerous Goods Committee

Human Performance Committee

New Aircraft Evaluation/Certification Committee

Noise Abatement Committee

Pilot Training Committee

Regional Airline Committee

These committees work on issues common to all the pilot membership.
Members are assigned to the committees from the airlines to ensure broad
representation. There is a technical staff in Herndon, Virginia at the
main ALPA offices that assist these technical committees to interface
with the government and industry.

In addition, there is a Regional Safety Chair structure that addresses
issue common to geographic regions. These regions generally match the
FAA regions, and within their scope is the oversight of the airports and
the Air Traffic Management in the region.

The Association has an award winning magazine, *[Air Line
Pilot]{.underline}*, that is distributed monthly to its membership. The
contentÕs focus emphasizes advances in air safety and flight technology;
industry developments; labor union information; and regulatory and
economic issues affecting the airline industry and the airline piloting
profession.

An Air Safety Hotline is provided for the membership to use in calling
in accident, incident, enforcement events, as well as any other pilot
reports of safety problems. This is the link between the line pilots and
the technical staff. It serves the caller having been involved in a
safety incident, and the technical committees and staff as a
communications channel.

The ALPA Air Safety Structure is complex interwoven network of expert
pilots and staff that coordinate to advance air safety. The efforts of
this structure go towards solving both day-to-day issues and advancing
the state-of-the-art within the aviation community. In support of these
efforts, ALPA has available for members use numerous aviation databases.
These databases are constantly evolving, and are used to assist in
developing documentation for our proposals for safety improvements.

ALPA believes that good use of databases can greatly aid in reducing the
number of aviation accidents that occur. ALPA feels that it is critical
to input as much information as possible into, and review these
databases on a regular basis. Such activity will assist the Department
of Transportation in reaching its goal of, ÒZero Accidents.Ó We suggest
that the way to prevent accidents is to investigate incidents. An
incident is a potential accident where the crew did something to avoid
the accident. The information from the incident investigations, if
stored in a database with sufficient detail and an extensive set of
keywords, should serve to identify issues for changes before they cause
an accident.

In order to be successful in achieving this goal, air safety databases
must be standardized and refined. The databases need as many accident
and incident reports as possible to be an effective tool in spotting
trends. The databases will need to be user friendly enough to have
reports entered easily. A common set of variables and their formats will
also aid greatly in increasing the number of reports. This will make the
data available for sharing among collection points. Also, there will
always be a need for the data recording to be objective and uniform in
its description of an event.

With these issues in mind, however, there also needs to be work done to
remove some of the barriers that hold back database utilization. These
barriers include the exclusion of data submitted to these databases in
the discovery process of litigation, and the de-identification of the
data sufficiently to prevent application of the data for purposes of
competitive advantage.

Proactive utilization of comprehensive incident databases could lead to
preventing numerous accidents, improving safety for all in aviation.

**Databases Utilized By Alpa**

The use of databases within ALPA is for the purpose of supporting the
air safety activity. In this realm, this includes discussions involving
the government and industry on proposed regulatory changes. Databases
are also used in the process of identifying past trends and documenting
accident investigation causes. Finally, air safety databases are used to
resolve pilot reports of safety problems. The databases function in
support of explaining reasons for recommended safety improvements.

The databases used within ALPA for air safety span numerous sources. The
Aircraft Information Services Limited (AISL) database covers accidents
where the airframe was destroyed or substantially damaged, and begins
with the introduction of jet aircraft. It is an international set of
data and is focused on airframe histories. This database is often useful
in tracking a particular aircraft, or identifying trends associated to
the hardware design of the aircraft.

The Service Difficulty Reports (SDR) are also used by our Association.
These are reports from the mid 70Õs to present, and focus on hardware
anomalies among national (flag and domestic) carriers. The purpose of
this data is also to identify trends in aircraft hardware failures.

The Civil Aviation Authority (CAA) of the United Kingdom provides data
on accidents from the mid 70Õs. This scope includes international
flights involving all commercial aviation. This data is beneficial for
its thorough documentation of the accident event.

Also used within ALPA is the recently acquired NASA Aviation Safety
Reporting System (ASRS) Database. This is a large dataset, consisting of
approximately 40,000 full event descriptions over the last five years,
and over 140,000 abbreviated event reports. The reports can span all of
aviation, but are mostly focused on issues of regulatory nature, as the
program includes a feature providing for immunity from penalty for most
regulatory violations. The purpose of this database is to provide candid
reporting of safety hazards without fear of backlash that could cost the
pilot their job.

ALPA also uses a database to manage the calls made to the ALPA Air
Safety Hotline. This program utilizes the British Airways Safety
Information System (BASIS). The reports in the database date from the
mid 80Õs to present, and involve events and reported problems that are
identified during line operations. The database is utilized to identify
trends that may lead to accidents and incidents, and to resolve
repeating issues without having to retrace all the steps discovered in
prior efforts. This data is useful for problem resolution at the local,
regional and national level.

#### A new database that may be used in the future involves the Flight Operations Quality Assurance (FOQA). This data is in-flight recorded data from onboard monitoring systems. It could be used to analyze trends, and improve air safety. It could influence accident investigation, pilot and crew inflight performance, training, performance, air transport operations efficiency, air traffic control enhancement, and aircraft and airport design. 

**Suggested Evolution of Features in Air Safety Databases**

ALPA recognizes that the industry is changing, but on the other hand,
some aspects seem to remain the same. Thus, we must be able to use our
databases in both aspects, examining both the long term and the short
term, as appropriate. In order to do so effectively, we must make
efforts to improve our databases, so they can perform effectively. We
suggest a review of air safety databases by their users should consider
changes to improve standardization, data sharing, and protecting data
from inappropriate use.

There is a significant need for the databases that can be shared with
other air safety organizations. In order to accomplish this, the data
must fit a common format and use a common set of keywords. Those
keywords must mean the same thing to the parties exchanging the data. We
suggest that to accomplish this, there should be a convergence of safety
database representatives under the guidance of experts in intelligent
systems, to establish a system for event description that combines the
conflicting goals of simple-to-use and thorough-and-objective. The
results of such a meeting could be made available in an Advisory
Circular or other guidance material. A depository could be established
that allowed participants to tap into the database for their evaluation
of safety issues, as necessary.

ALPA suggests that provision of a depository for air safety data is the
best way to get more data. Getting more data can be beneficial by making
trends more clear. But the trend only becomes clear if the description
is lucid. A vague or inappropriate description will not aid in
establishing a set of supporting documentation to an expected safety
problem. So there is a tradeoff again, between getting more data and
having the data be of sufficient detail and clarity to assist by being
applicable to the situation at hand.

There is a chance that there also will be a chasm between the need for
sharing data and the need for protecting proprietary data. In our
organizationÕs pilot report database, we de-identify the airline
identifier. There may be reason to know the carrier is the same if a
trend caused by their procedure or training caused the trend. Yet, this
could be addressed within the airline. Finally, another way to address
this issue could be to trade refinement in one area for increased
vagueness in another area. For example, the request to group the data by
common carriers could be made to be limited to training events, or to
non-specific aircraft. Overall, what this discursiveness comes down to
saying, is that we must restrain our use of this data to only address
safety issues.

**Flight Operations Quality Assurance (FOQA)**

The use of the data from line operations as proposed in FOQA is a
critical activity, with great potential, according to some advocates. We
can understand such excitement, but wish to have it clear that there are
also great concerns over the use of this data. Such data, if applied
maliciously, could easily be career threatening. The intent of FOQA
clearly is to improve safety and prevent accidents. Thus, it should be
limited to improving and not destroying careers.

ALPA sees a need for FOQA to ensure any event that is evaluated is
compared to a valid facsimile of the comparable events. It is essential
that the data collected from the sampling in FOQA is determined by law
to be unusable in the discovery process. The data itself explains what
happened, but also needed is an explanation of why the event happened.
As such, a mechanism for prompting the submission of ASRS reports to go
with flight data sampling would aid in providing this. Absence of such
linkage will probably cause the reasons behind performance anomalies to
remain unexplained.

**Conclusion**

There is a definite need for proactive use of incident data in
identifying ways to prevent further similar incidents before an accident
occurs. Data needs to be gathered in a format that will be uniform and
objective, detailed and clear, that can be used to determine if a
proposed fix will resolve the problem. Sharing of data will increase the
chance of discovering the trends, but sharing needs to be done in a way
that we do not mix unlike events. Security of the data needs to be a
priority.

## INCIDENT INVESTIGATION: PARADIGM SHIFTS TOWARD EXPLOITING SUCCESSFUL HUMAN FACTORS INTERVENTION STRATEGIES TO PREVENT AIRCRAFT ACCIDENTS - Ludwig Benner, Jr., P.E. and Ira J. Rimson, P.E. 

Ludwig Benner & Associates *The Validata Corporation*

12101 Toreador Lane P.O. Drawer 1315

Oakton, VA 22124 Springfield, VA 22151

(703)758-4800 (703)978-2944

*\"Admittedly, most crashes these days are blamed on pilot error; an
analysis two years ago showed the behavior of the flight crew as the
dominant cause of 60% of crashes. But this is somewhat misleading since
**it includes crashes which the crew failed to avert after something
else had gone wrong first**.\"* (Emphasis added) *Ñ The Economist*,
April 8th, 1995, p. 71.

**Position**

Specific incidents, analyzed scientifically, will immediately identify
successful interventions which have averted accidents, from which
effective prevention strategies can be directly derived.

**Problem**

The objective is to reduce the incidence and severity of aviation
accidents and incidents by formulating changes to human behavior which
contribute to those adverse outcomes. Historical approaches have failed
to achieve enduring modification of errant behavior. Traditional
paradigms must be altered to overcome the inherent inadequacies of
prevailing mishap origination concepts and investigation methodologies.

*\"If you do what you did,*

*you\'re gonna get what you got!\"*

\- Yogi Berra

**I. Background**

A. Current mishap investigation approaches, objectives and
methodologies:

¥ Originated in 1930\'s by U.S. Civil Aeronautics Board, based on
legalistic concepts of culpability and liability.

¥ Evolved to fulfill statutory and regulatory demands rather than
scientifically-based accident prevention requirements.

¥ Mandate finding of \"probable cause(s)\" as the primary goal for
mishap investigation, which diminishes prevention initiatives.

¥ Focus on problems rather than on actions which overcame problems,
failing both to prevent recurrence of old causes or emergence of new
problems.

¥ Define \"accident\" or \"incident\" in terms of relative outcomes,
which are irrelevant for identifying data significance or analyzing
processes leading to the outcomes.

¥ Trivialize mishaps in which surviving participants could describe the
process by which they prevented catastrophic outcomes.

¥ Lack investigative standardization, therefore lack the capability to
measure or control either internal quality or inter-investigation
consistency.

¥ Accept uncritically all data sources and formats without applying
standards for verification of individual data elements, corroboration
among data elements, or validation by scientific testing.

¥ Discourage candid, objective contributions by subjecting surviving
participants to negative judgmental evaluation.

¥ Lack methodology for recasting anecdotal or episodic data into
scientific format, therefore lack ability to discriminate among
variables with sufficient certainty to verify statistical inferences.

B. Current analytic approaches, objectives and methodologies:

¥ Do not require variables to be normalized.

¥ Accept data at least four levels abstracted from observation as
statistically valid:

1\. Investigator\'s descriptions of observed data

2\. Report format and statistical category designers\' assumptions

3\. Investigator\'s interpretations of descriptive data to fit undefined
format criteria (\"Fill out the Form\")

4\. Data analyst\'s reinterpretations of formatted data to fit undefined
statistical category criteria

¥ Render reliable statistical inference and productive safety
recommendations virtually impossible because of high level of
abstraction of safety data.

**II. Human Factors/Human Error Models**

A. Traditional

1\. Prospective

¥ Probabilistic Risk Assessment

¥ Fault Tree

¥ Error-tolerance Limit

2\. Retrospective

a\. Qualitative

¥ MORT

¥ \"Root Cause\"

b\. Quantitative

¥ Epidemiology

B. Innovative

1\. Reason: Latent vs. Active failures

¥ \"Windows of opportunity\"

2\. Perrow: \"Normal\" accidents

¥ System complexity vs. coupling density

3\. Ratner: Enhanced Safety Net

¥ Facilitating Operational Situations/Predisposing Underlying Factors

4\. Benner: Multilinear Events Sequencing

¥ Event Pairs and Logic Testing

C. Common requirement of all models: A scientifically accurate
description of

***WHAT HAPPENED?*** - *what people did* to produce the outcome Ð

is seldom formulated by current post-facto investigations.

Investigations must provide data from incidents or accidents to
determine:

¥ Who did what, when, where, how and why?

¥ What specific behavior increased the likelihood of an undesired
outcome?

¥ What specific behavior mitigated or prevented the undesired outcome?

¥ What behavior is preferred within the operational context?

¥ How can the preferred behavior be achieved?

**III. Real-World Problems of Data Relevance, Quality, Quantity and
Accessibility**

A. Relevance. Investigation outputs must, at minimum, include compatible
descriptions of:

¥ What was supposed to happen?

¥ What did happen? \-- yet frequently do not.

B. Quality and quantity.

¥ Post-accident investigation data rely principally on second-hand (or
more abstract) sources, inferences or opinions, and rarely exist in
sufficient quantity or quality to fulfill requirements without
interposing abundant unverifiable conjecture.

¥ Incident investigation affords opportunities to obtain first-hand data
directly from partici­pants, including how they perceived deviations from
plans, diagnosed effects of devia­tions, evaluated and selected among
alternatives, and evaluated consequent interven­tion actions and their
results; in short, the entire process by which the incipient accident
was mitigated to achieve an acceptable outcome. \[ref. \"DECIDE\"
Model - *see Hendrick, Benner & Lawton (1987) and Lawton, Benner,
Clarke, et al (1987)*\]

C. Accessibility. Two mutually exclusive problem areas exist:

1\. Accessibility of data possessed by mishap participants:

¥ Current \"Attitude\" paradigm focuses on error, blame and liability,
provid­ing little motivation for survivors\' willing cooperation.
\"Parties\" to post-accident investigations commonly restrict
participant employees from candid interviews, and interpose strict legal
oversight to protect their perceived vulnerability as potential
litigants.

2\. Accessibility of information/data possessed by investigation
agencies:

¥ Judgmental \"Attitude\" paradigm leads to restrictive covenants
between investigation agencies and participants/witnesses which inhibit
flow of data needed to fulfill current statistically-driven database
approach to problem definition.

As a result of both problem areas, numerous independent incident-derived
data bases have been established and are maintained by manufacturers and
opera­tors for \"proprietary\" internal risk management programs, which
are rarely shared for prevention purposes.

**IV. Alternative Approach, Objective and Methodology**

A. The aviation community\'s quest for a reality-based initiative
designed to reduce the incidence and severity of aviation accidents
involving human factors will be better served by:

1\. Adjusting the \"Opportunity\" paradigm from \"Post-accident\" to
\"Pre-accident\"; i.e., redefine \"accident\" and \"incident\" from mere
outcome attributes to functional process descriptors: e.g., *\"\...an
incident is an incipient accident which failed to attain its full
potential because of successful intervention by persons, things or
fortuity within the system.\"*

¥ Redirect data acquisition concentration from accidents (which identify
\"causes\" or operational failures) to incidents (which identify both
operational failures and successful recoveries therefrom).

2\. Adjusting the \"Attitude\" paradigm from Negative: \"What Went Wrong
to \'Cause\' the accident?\" to Positive: \"What Went Right to Prevent
it?\"

¥ Acknowledge both the ubiquity of human error, and human capability to
recover from errors. Redirect resources toward successful intervention
processes which thwart accident progression, thereby focusing on
adaptation to error rather than error perpetuation.

¥ Expand investigations\' focus to include positive factors. Encourage
witnesses to provide accurate data for constructing effective prevention
strategies, in contrast to defensive \"CYA\" strategies fostered by
current judgmental perspectives which emphasize failures and errors.

3\. Adjusting the \"Investigation Methodology\" paradigm from
\"Investigator\'s Option\" to stan­dardized investigation techniques
which generate scientific data formats, permit critical data analyses,
and allow data to be evaluated against measurable quality standards.

¥ Institute data input, process and product quality standards and
controls.

¥ Solicit, document and collect actions during actual successful
anecdotal and episodic experiences to obtain timely source data for
defining prevention initiatives which actually worked.

4\. Adjusting the \"Data\" paradigm from unscientifically-defined
statistically-driven data base fulfillment to direct observation,
recasting the raw data into scientific formats accessible to quality
control, verification and confidence-testing procedures.

¥ Use observed data both to devise and validate models of successful
error compensation. Current theoretical models employing abstract data
categories or forms as templates drive investigation data selection and
analysis, and are rarely (if ever) subjected to validation or
verification testing.

¥ Compile participant-sourced data from intervention-focused
investigations which identify successful accident mitigation tactics
immediately and accurately, and are self-verifying. Current
error-focused investigation data deficiencies beget unverifiable
assumption and conjecture about opportunities for error avoidance or
response.

¥ Require accurate process timing data to support time-sensitive
modeling methodologies; e.g., Perrow\'s \"coupling density\" and
Benner\'s STEP/Multi­linear Events Sequencing.

5\. Adjusting the \"Analysis\" paradigm from current styles which are
incapable of identifying defective data and rely on normal statistical
methods, to techniques which enable investigators to define and describe
interactions and interfaces accurately and reliably.

¥ Recognize necessity for \"Rare-Event\" statistical treatment,
including techniques for recasting anecdotal and episodic data to permit
valid statistical treatment.

¥ Measure \"What happened?\" against \"What was supposed to happen?\" to
establish disciplinary relevance boundaries, and identify when incipient
accident-processes begin, not merely where they end.

**V. Recommendations**

1\. Redirect FAA investigative resources to develop more useful data
about successful inter­vention actions during incidents, rather than
investigating relatively data-poor NTSB-delegated mishaps.

2\. Standardize investigation methodology to facilitate exercising
objective quality controls on data and analyses.

3\. Require that investigation methodology include accurate
chronological data to define events and relational logic within the
mishap process.

4\. Establish a reporting process which

a\. encourages operational level persons to submit their incident
experiences and anecdotes directly and immediately, and

b\. captures their data in a format that enhances the data\'s accuracy
and utility with minimal need for editing or manipulation.

*(This could easily be an enhancement of the current NASA/ASRS.)*

5\. Establish a centralized repository through which participants
worldwide can contribute and extract data and information.

7\. Conduct prompt triage on anecdotal and episodic data to identify
incidents which hold high potential for producing useful intervention
actions. Analyze those incidents and provide priority feedback to
operators who can benefit therefrom.

8\. Enhance the accuracy and quality of scientific investigations to
improve the efficacy of their outputs by establishing a single
independent government-sponsored National Investigative Agency to
replace the myriad organizations currently conducting unstandardized
investigations which fail to achieve established quality management
objectives. A dedicated national agency, staffed by profession­ally
educated and trained investigators intimately acquainted with quality
management, could overcome parochial bias and enhance investigations\'
effectiveness in preventing recurrence of mishaps of national influence.
Core investigative expertise would be reinforced on a case-by-case basis
by technical specialists from appropriate regulatory and operational
agencies.

**References**

Benner, Ludwig Jr.; *Four Accident Investigation Games: Simulations of
The Accident Investi­ga­tion Process*. Events Analysis, Inc., Oakton, VA
1979. 2nd Edition 1982

Benner, Ludwig Jr.; \"Rating Accident Models and Investigation
Methodologies.\" *J. Safety Research* 16:3, Fall 1985, Chicago, IL.

Benner, Ludwig Jr. & Ira J. Rimson; \"Quality Management for Accident
Investiga­tors\". ***forum*** 24:3, October 1991 (Part 1); 25:1, March
1992 (Part 2). International Society of Air Safety Investigators
(ISASI), Sterling, VA.

Feil, John A. & C. L. Lewis; \"Accident/Incident Prevention through
Airport Safety (An Airline Program for Airfield Inspection and
Audits).\" ***forum*** 23:4, February 1991. ISASI, Sterling, VA.

Ganse, Larry R.; \"Integrating Accident Prevention and Investigation.\"
***forum*** 27:4, December 1994. ISASI, Sterling, VA.

Hendrick, Kingsley & L. Benner, Jr.; *Investigating Accidents with
STEP*. New York, Marcel Dekker, 1987.

Hendrick, Kingsley, L. Benner & R. Lawton; \"A Methodological Approach
to the Search for Indirect Human Elements in Accident Investigations.\"
*Proceedings of the Fourth International Symposium on Aviation
Psychology.* (R. S. Jensen, Ed.) The Ohio State University, Columbus;
April 29, 1987.

Lawton, Russell L., L. Benner, R. Clarke, R. Jensen, A. Adrion & J.
Maresh; *The DECIDE Approach to Training Pilots & Pilot Decision
Making.* AOPA Air Safety Foundation, Frederick, MD; June 1987.

Perrow, Charles; *Normal Accidents*. New York, Basic Books, 1984.

*Rare Event/Accident Research Methodology*; NBS Special Publication 482.
Washington DC. U.S. Department of Commerce, National Bureau of
Standards, 1977.

Ratner, Robert S. & J. Guselli; \"Techniques for Identifying Indirect
Causal Factors in Aviation Incidents and Accidents.\" ***forum*** 28:2,
June 1995. ISASI, Sterling, VA

Reason, James T.; *Human Error*. Cambridge, Cambridge University Press,
1990.

Rimson, Ira J., & J. B. Galipault; \"Incident Analysis: A Key to
Accident Prevention\". ***forum*** 12:3, Winter 1979. ISASI, Sterling,
VA.

Rimson, Ira J.; \"Standards for the Conduct of Aircraft Accident
Investigations\". ***forum*** 23:4, February 1991. ISASI, Sterling, VA.

## HUMAN ERROR IDENTIFICATION: DESIGNING FOR THE USERS - Marilyn Sue Bogner, Ph.D.

Food and Drug Administration

Even though human factors efforts have been directed to reducing the
likelihood of human error in **a** number of domains, accidents
attributed to human error persist. If human factors efforts can affect
the incidence of error, then the lack of their effectiveness may He the
identification of the factors contributing to error. This paper
addresses that possibility by presenting a human factors approach to the
identification of factors that contribute to error and the development
of error prevention strategies whether they involve training, design of
new equipment, equipment modification, or policy change.

The following discussion describes an error data collection effort
designed to provide sufficient information to identify the problem and
target prevention strategies to specific factors that contribute to the
problem. This involves two techniques: grounding the data collection
effort in the perspectives of the user populations, and applying a
systems model to the design of the data collection, analysis, and
applications of the findings..

It should be noted that although this paper discusses the human factors
approach in terms of developing a data collection effort, the approach
also can be used to structure and analyze existing data collection
activities. The effectiveness of the approach in existing activities is
limited, however, because the perspectives of the user populations
almost certainly were not considered in the development of the effort.
The importance of those perspectives is addressed in the next section of
this paper.

A number of steps are involved in developing an error data collection
effort and identifying preventive strategies based on the findings from
the data. For the purpose of this paper, the steps include: defining the
problem, decomposing it into its constituent parts, determining the
appropriate data to collect, devising a means for collecting data that
will address the concerns of the users, and analyzing the data to
provide meaningful findings. Those steps organize the following
discussion of an empirically based, domain-free set of data elements
which can be augmented to address the idiosyncratic issues of any
specific domain.

**Problem Definition**

The definition of human error for the purpose of this paper is an
amalgam of dictionary

definitions of error: *Human error is an act, assertion, or decision
that deviates from a norm and results in an actual or potential adverse
incident. That incident may or may not eventuate in an adverse outcome.
The norm which defines an error is consensually accepted by the
constituents of the domain under consideration An error may reflect a
number of factors or may, be the final act in a series of contributing
errors, i.e., a cascade of errors.*

There have been a number of refinements of the term human error. The
rubric of error has been differentiated into slips and mistakes (Reason,
1990) as well as being linked to a level of performance as in AM-based,
rule-based, or knowledge-based activity (Rasmussen 1986). To have a
simple definition of error for ease of conceptualization and discussion
in this paper, those refinements although admittedly useful for more
detailed considerations, are not addressed.

Human error is not unique to any one domain. There is ample
documentation of human error in industries such as aviation, nuclear
power, and health care. The observation that error is ubiquitous and has
been over time is confirmed, albeit only for a certain class of error,
by the existence of a mechanism in the human brain that is dedicated to
monitoring a person\'s performance and compensating for errors (Dehaene,
Posner, and Tucker, 1994). The class of errors which activates this
mechanism is that of errors detected in time for a correction to be
attempted, i.e., near misses.

Errors that are near misses should be considered in an error data
collection activity because what is a near miss for one individual may
be a fatal error for others. Information on near misses can only be
captured from the person experiencing the error. Similarly, information
about what actually occurred in any error can best be provided by the
person experiencing the error. The perspective of those who experience
errors, i.e., those who use the data collection instruments by
responding to them is rarely considered. Typically, the users of data
collection activities are presumed to be the professionals who collect
and analyze the data.

The data collection and analysis activities described in this paper,
consider the perspective of two groups of users: those who report the
information and those who use the information. If the information from
the former group of users isn\'t viable, then the products from the
latter group may not be accurate and lead to ill conceived preventive
strategies. Because the perspective, of the information providers is
central to the success of a data collection effort, the data collection
activity should be designed and organized for relevance and
acceptability by the respondents as well as providing the necessary
information to the data collectors. The inclusion of the perspectives of
both groups of users throughout the ensuing discussion is implicit when
not explicit.

Errors are broadly categorized by their outcome: an adverse incident
which nearly occurs, i.e., a near miss; an adverse incident which
occurs, but its impact is immediately reversible; an adverse incident
with an adverse outcome lasting 3 months or less; an adverse incident
with a long term (more than 3 months) adverse outcome; and an adverse
incident resulting in death. This paper addresses only errors which are
quantifiable either directly or indirectly. Near misses are quantifiable
via the report of the person involved. Although efforts to reduce the
likelihood of an error and reluctance to report an error are related the
severity of its outcome, outcome is only tangentially considered in this
discussion of error identification. The focus of the paper is error, not
outcome.

Human error is considered as having two components: behavior that is
determined to be

erroneous, and the conditions in which the error occurred. The broad
categories of behaviors consist of acts, assertions, and decisions which
may lead to near misses, adverse incidents, or adverse outcomes. There
also are two components to the conditions in which the error occurs: the
contributing factors, both proximal and distal and the preceding or
anticipated circumstances. This consideration of error inducing
conditions is structured by decomposing them into categories of
contributing factors.

**Problem Decomposition \-- Contributing Factors**

Given the vagaries of human beings, it is unlikely that error ever will
be eradicated. The incidence of error, however, can be reduced by
targeting preventive activities to the factors that contribute to it.
Those factors can be identified by decomposing the problem through
analyzing the conditions in which the error occurred and developing data
elements to reflect those factors. Determination of categories of
contributing factors can aide this process by focusing the analysis.

An error involves more than the person associated with the error even in
the apparent solitary act of decision-making. It is necessary for an
individual to have something about which to make a decision.
Decision-making entails at a minimum, the psychological cognitive, and
perceptual facilities of the decision maker and an entity or set of
circumstances to stimulate those factors. Even the most
intra-personal/intra-psychic error involves extra-personal factors. This
diad can be conceptualized as a system of the individual and the
extra-personal factor(s). Thus, error reflects an event of a system. If
the basic unit of consideration meets the criteria for a system then it
is reasonable that a systems approach be applied to the problem of error
identification.

Moray (1994) has proposed a domain-free systems approach that consists
of a hierarchy of levels of description. The hierarchy starts with
equipment, followed by the physical ergonomics of the equipment, the
behavior of the individual using the equipment, team and group behavior,
organizational and management behavior, legal and regulatory rules, and
finally societal and cultural pressures under which all else operates.
These levels assist the analysis of the problem of error identification
by providing structure for the decomposition of human error into broad
categories of contributing factors.

The literature suggests that there is a core of factors that contribute
to error regardless of the domain in which the error occurs. Those
empirically based findings are included in the decomposition of problem
of error. The well documented error-contributing factor of fatigue
(Kreuger, 1994) provides an example. That factor is induced by the
conditions such as consistently working extended hours or varying
shifts. A preventive strategy directed to the individuals might be to
instruct them to spend more of their off-work hours steeping. This is
not necessarily a realistic admonition. Using the systems approach,
fatigue is considered in terms of its cause, the work schedule, which is
within organizational and management behavior. Thus, preventive
strategies to reduce the likelihood of fatigue induced error would be
directed to the organizational and management component.

In addition to the classes of contributing factors within the
hierarchical categories that are common across domains, there are
idiosyncratic characteristics of each of the domains. These
characteristics are domain-specific foci for the common contributing
factors. For example, the organizational and management behavior
category, there is the class of delegation of responsibilities.
Delegation of responsibilities for an airline cabin crew with respect to
the cockpit crew has different foci from that for personnel in a
hospital\'s intensive care unit with respect to the emergency room
staff. To best represent the actual nuances of the contributing factors,
the foci should be developed from the respondents\' perspective as well
as that of the data collectors,

**Data Element Determination** to **Capture Acts and Circumstances**

To be relevant, data must reflect the problem being addressed, i.e.,
human error. This involves capturing information about the erroneous act
or behavior and the circumstances, the context of the situation in which
the act occurred from the perspective of the respondent. For the purpose
of this discussion, the situation in which the error occurs can be
envisioned as a vertical representation of the contributing factors
within the categories of the systems approach. The circumstances are
factors in preceding or anticipated situations which contribute to the
error. Circumstances can be represented on a horizontal time line.

The circumstantial factors do not directly affect the person, yet they
contribute to the error. The influence of circumstances is
intra-psychic; however, the contributing circumstantial factors can be
documented by records of previous activities or schedules of future
ones. For example, a person omits a critical step in an operational
procedure for no apparent reason, i.e., the work conditions are as usual
with no observable stress or fatigue and the equipment being operated
equipment is familiar. Analysis of circumstances finds that the person
observed a fatal incident the previous week and is scheduled to use the
equipment that was involved in that incident during the next day. This
could be pre-occupying the person and contribute to the omission.
Further information should be gathered and preventive strategies
developed.

To provide as complete a picture as possible of what leads to human
error, the elements of an error data reporting activity should tap into
the respondents\' experience with the erroneous acts and the
contributing factors in the specific situation and circumstances in
which the error occurred. The elements should be designed so the
respondents provide sufficient detail about the error and the context in
which it occurred to identify all contributing factors. Those factors
then can be organized into the system categories for analysis and use in
targeting preventive strategies.

Because every aspect of human endeavor evidences behavior that is judged
erroneous, it can be assumed that some consistency in characteristics of
erroneous behavior exists across domains. This consistency can be found
in ongoing research and training activities in various domains as well
as in the empirical literature. Similarly, because the conditions in
which errors occur are finite and even similar across domains, it can be
assumed that classes of factors that contribute to that behavior might
be identified. In addition, there are intra-personal consistencies
across errors and situations. For example, some people tend to blame
themselves for an error when the

contributing factors actually are external to them These people are
considered to be intrapunitive rather than extra-punitive (Sellen,
Senders, and Russell in press). Data elements should be designed to
address this tendency as well as other consistencies.

Given that the purpose of the data collection activity is to develop
prevention strategies to reduce the likelihood of the error, it is
imperative that the data elements elicit information which after
appropriate analysis, indicates the conditions in which the error
occurred and the contributing factors. It is not uncommon that data
collected by a problem reporting system reflect the occurrence of a
problem with only the readers\' imagination to determine the
contributing factors. An example of such a report is: \"Person X
experienced **a** severe injury when person Y raised widget Q on which X
was standing. Widget Q was inspected and found to be functioning in
accordance with its specifications.\" Such a report would be interpreted
as indicating that human error was the cause because the equipment,
widget Q, was functioning according it its specifications. What
prevention strategies might be developed from this information?

Numerous comparable reports may be received from a reporting system
which collects data as in the previous example. From such data, it can
be stated that human error with Q caused severe injury in Z number of
instances. Although providing information for summary statistics, these
data provide no viable information to address the problem. That is, no
information is provided to identify: what aspect of Q might contribute
to the problem any deficiencies in the skills and abilities of Y,
circumstances of the work environment that affect Y\'s behavior, or
contributing factors in the process of using Q that contributed to the
error.

The example underscores the importance of data elements that provide
sufficient, appropriate information to accomplish the purpose of the
data collection activity, i.e., develop prevention strategies. In
addition, data elements should be designed in a way to elicit
information from the respondents, i.e., the data elements should be
acceptable to the users of the data collection instrument. The phrasing
of the elements should be unambiguous, the elements should be simple and
concise, but not preclude detail. The elements within the categories of
contributing factors should stimulate the respondent to address all
aspects of the situation. The elements should elicit extra-punitive
contributing factors from intra-punitive respondents.

In striving to develop data elements from the perspective of the
respondents, it is necessary to consistently consider the issues of
quality and quantity of data elements from the perspective of the other
group of users \-- those who will code and analyze the data and
ultimately use the findings to develop strategies to prevent error. The
elements should be designed for ease of coding yet be amenable to the
respondents. The elements should allow the response to be focused, yet
allow for elaboration that can be readily coded.

Both groups of users should be involved in the final determination of
the data elements. Previous data collection efforts in the domain under
consideration and other domains to the extent possible, should be
analyzed and lessons teamed, both positive and negative, applied to the
current activity. This provides an expanded empirical basis for the data
elements.

**Data Collection**

To optimize the effectiveness of the data collection for both groups of
users, it is necessary to design and implement that activity in accord
with its purpose of obtaining information about error. Collecting error
data has particular problems; there is a major accessibility issue of
the repercussions from reporting errors. Given the current. propensity
of our society to solve all problems through litigation, the possibility
of personal liability from error reporting is a non-trivial factor in
collecting error data. Common parlance is that the Aviation Safety
Reporting System (ASRS) has successfully circumvented that concern to a
considerable degree. Thus, the ASRS provides lessons teamed that are
valuable to all domains involved in collecting error data.

Another facet of effective data collection is that of accessibility of
the data reporting instrument to the respondent. If the means for
reporting data is not available or is inappropriate for the task, then
it won\'t be used and even the most sensitive data elements are for
naught. A paper form has high accessibility to the user whereas a
computer terminal may be relatively inaccessible. Information from the
paper form particularly a written narrative, may be very difficult to
read, resolve ambiguity, and code. Direct entry into a computer is
readable, however, coding of a narrative remains a problem.

For accuracy and completeness, data pertaining to an error should be
collected as close to the time and location of the error occurrence as
possible. These qualifications argue for collecting data via a palm-top
computer or possibly a small lapel-mounted tape recorder from which the
data would be transcribed and coded to conform to the data elements. The
miniature computer and the transcription and coding of audio recorded
error descriptions probably are not feasible due to fiscal constraints.
An affordable alternative might be for the respondents to reply to the
elements presented on a written form via audio tape which could be
computer coded.

How representative the number of errors reported is to what actually
occurs within a given domain is of tantamount concern. Under-reporting
leads to inappropriate representation of the magnitude of the problem.
Preventive efforts based on under-reported information although well
conceived and directed, may not be effective because the portion of the
problem addressed is insufficient. This could lead to an effective
preventive effort being considered ineffective.

Skewed reporting, i.e., data collected which disproportionately
represent one among many contributing factors could lead to preventive
activities-\'ties that appear to be ineffective for the problem while
actually being effective for the aspect of the problem identified, These
examples emphasize the importance of thoroughness in data collection and
suggest that the process of data analysis be carefully executed so that
its potential can be realized.

**Data Analysis**

Issues to be addressed by the analysis of data should be determined
prior to the development of the data elements. Data elements typically
address a narrowly defined problem; however, additional information for
a nominal increase in effort might be provided by further analysis of
responses to some data elements.

The analysis of data for the development of preventive strategies is
driven by the identification of what constitutes error, factors both
proximal and distal that contribute to the error, and sources of those
factors. Another purpose of the interpretation of the findings from the
data analysis could be an explicit as well as implicit evaluation of
current preventive activities. That would provide lessons learned
information which can be particularly useful for a domain\'s
geographically distributed activities. The information could be applied
to a planned activity that is comparable to one implemented elsewhere.

The analysis of data can produce findings which are applied to the
purpose of the activity. It also can produce findings that contribute to
the research literature. This dual use of data analysis, that of
application and research, is iterative (Bogner, 1994). Findings from the
data analysis can be used to direct the development of preventive
strategies activities to a specific error, and contribute to the,
refinement of the data collection activity and elaboration of the models
upon which it is based. Those refinements when reported in the research
literature can be integrated into the development of other data
collection activities within the specific domain or across domains. The
process is repeated with further refinement of data collection
activities and enrichment of the literature that may be have a number of
applications both within and apart from error data collection
activities. Such uses of data multiply the impact of resources.

**Summary**

Rather than error being considered a shameful act to be buried for fear
of being punished, error should be considered a flag, an occurrence
indicating a problem to be solved. From the systems perspective, an
error will persist regardless of the individuals involved until the
circumstances that induced the error are remedied. The problem is in the
system not the individual so shame and blame are not appropriate.
Efforts directed to reducing the likelihood of the recurrence of an
error should be directed not just to reducing the error *qua* error, but
to redressing the conditions that caused it

There are several audiences for the reports of the findings from the
data analysis which are represented by the categories of contributing
factors in the hierarchical systems model e.g., organization and
management personnel and those concerned with legal and regulatory
issues. From those findings, each audience can affect preventive
strategies which synergistically can impact the problem In some
instances, format organized activity may not be necessary; performance
can improve in a performing unit when that unit is given a report of its
activity together, with comparison data for comparable units (Barbour,
1994).

Two approaches to error identification were presented in this paper both
of which represent the application of a human factors approach to error
identification. One approach is that the perspective of each of the
groups of users should drive every aspect of the error data collection

activity. The second approach is that a systems model is an effective
guide for the development and implementation of the data collection
activities and preventive strategies. These approaches not only are
parsimonious as advocated by OccamÕs s razor, they are good business
because they enhance the impact of scarce resources by building on
previous work and affording multiple uses of the findings.

**References**

Barbour, G.L. (1994). Development of a quality improvement checklist for
the Department of Veterans Affairs. *Journal of Quality Improvement,
20,* 127-139.

Bogner, M.S.(1994). Human effort in medicine: A frontier for change. In
M.S. Bogner(Ed.) *Human Error in Medicine* (pp- 373-383). Hillsdale, NJ:
Lawrence Erlbaum Associates, Inc.

Dehaene, S., Posner, M., & Tucker, D. (1994). Localization of a neural
system for error detection and compensation. *Psychological Science, 5,*
September.

Kreuger, G.P. (I 994). Fatigue, performance, and medical error, In M.S.
Bogner (Ed.), *Human Error in Medicine* (pp. 311-326). Hillsdale, NJ:
Lawrence Erlbaum Associates, Inc.

Moray, N. (1994). Error reduction as a systems problem. In M.S. Bogner
(Ed.), *Human Error in Medicine* (pp. 67-91). Hillsdale, NJ: Lawrence
Erlbaum Associates, Inc.

Rasmussen, J. (1986). *Information Processing and Human-Machine
Interaction. Amsterdam:* North-Holland.

Reason, J. (1990). *Human Error.* Cambridge, England: Cambridge
University Press.

Sellen, A-J., Senders, J.W., & Russell, W.J. (in press). Design
deficiencies in anesthetic incidents. *Human Factors.*

## FLIGHT CREW ACCIDENT AND INCIDENT HUMAN FACTORS - Objective 1

Identify the relevant human factors data elements, or types of data,
required for appropriate analysis of flight crew error contributing to
aviation accidents and incidents.

The AOPA Air Safety Foundation (ASF) works frequently in concert with
various offices of the FAA and other aviation entities to further
everyoneÕs goal of reducing aviation accidents. For instance, we are
actively involved in producing safety reviews of accidents involving
general aviation aircraft. Accidents for specific aircraft models or
specific hazards to general aviation flight operations are studied to
help other pilots avoid the same problems, whether they are human,
mechanical, or environmental factors. In addition to the accident t
analyses, each review includes an extensive training outline for
instructors and pilots to use in incorporating risk management
techniques into their transition and recurrent training. Emphasis is
given to those specific areas that the studies have been shown to have a
high risk factor.

As extensive as these studies are, however, there are some risk factors
that we have not been able to identify because the supporting data is
not available. Here are the additional data items that the Air Safety
Foundation would like to see collected during investigations of general
aviation aircraft accidents. While we realize that it may not be
practical to expect NTSB/FAA investigators to be able to obtain all this
data, it represents our ideal for research and analysis into accident
causes and their prevention.

Type of weather avoidance equipment installed (radar, stormscope, etc.).

Was is functional, in use, helpful?

Type of deicing equipment installed (boots, fluid, electric, etc.).

Was it functional, in use, helpful?

Type of ground proximity warning device installed (radar altimeter,
etc.).

Was it functional, in use, helpful?

Aircraft ÒhomeÓ base.

PilotÕs Training:

Year of first solo flight.

Date and type of training for each certificate/rating.

(Part 141 school, university, FBO, free-lance instructor, etc.)

Date and type of any refresher/transition training.

Date and type of any simulator-based training.

Date of last BFR and make and model of aircraft flown.

Date and type of any safety seminar attended.

Level of participation in FAA Pilot Proficiency Award Program.

PilotÕs Flight Activity:

Flight times in the last year.

Instrument flight time in the last year and last six months.

Night flight time and time in make/model, last year, last six months,
and last 90 and 30 days.

Multi-engine flight time and time in make/model, last year, last six
months, and last 90 and 30 days.

PilotÕs Background:

Marital status.

Highest level of formal education received.

Aircraft accident history.

FAR violations.

Automobile license suspensions/revocations.

Alcohol/drug arrests.

Automobile accidents.

However, any data about specific accidents will become really useful
only when corresponding data about the pilots who do not have accidents
is also available. A recent joint conference with the FAA investigating
general aviation pilot proficiency requirements identified a compelling
need for more timely and comprehensive general aviation exposure data.
The present FAA annual survey that collects data and estimates
statistics has several serious shortcomings that limit its overall value
to the FAA and to industry safety organizations.

The FAA estimates the number of hours flown in general aviation aircraft
only on an annual basis. The estimate is typically released anywhere
form nine months to a year after the fact. Currently, there is no
official estimate for the number of hours flown in general aviator for
1994, much less for the early par t of 1995. There are only projections
and forecasts for that time period. Although there are some monthly
estimates of general aviation activity in the IFR arena and at FAA
control towers, in reality general aviation conducts much of its flight
activity outside of these environments. In effect, the national has no
reliable estimate of how much flying is actually being conducted by
general aviation until almost a year has passed. Even then it has only
annual data - monthly data is just not available.

Thus a void exists in reliable, timely exposure data with which to
relate the more frequently reported safety measures (i.e., accidents,
incidents, fatalities, near mid-air collisions, etc.). Analyses
conducted without a reliable basis for comparison can lead to erroneous
or misleading conclusions. The ASF has consistently found that due to a
lack of exposure data in several categories of general aviation
activities, many desired studies and analyses cannot be performed.
Studies which are done are often based on imperfect data and with too
high a reliance on intuition. Our FAA colleagues have acknowledged
similar frustrations.

Despite the lack of current exposure data for all of general aviation,
the FAA, and to some extent the NTSB and other national aviation safety
organizations, continues to release a variety of safety information
about general aviation on a monthly basis. These range form NMACs and
runway incursions, to accidents and fatalities. In many cases,
government entities must make informed decisions based on this data.
However, any conclusions drawn form such data must remain suspect until
they can be weighed against how much flying is actually being done. As
an example, if the number of fatal accidents declines compared to a hear
ago, does this mean that the system is getting safer, or does it mean
that less flying is being conducted? Until reliable, current exposure
data is available for all of general aviation, we just do not know.

The requirement for current activity data against which to compare
accident and other failure data is essential for the production of
effective programs to reduce the general aviation accident rate. It is
obvious that exposure data is needed on a monthly basis with perhaps
more comprehensive data in certain areas.

## HUMAN FACTORS RESEARCH INTO COCKPIT/CABIN CREW PERFORMANCE - Rebecca D. Chute

San Jose State University and Ames Research Center

The cabin crew on board the commercial aircraft of today are an
essential component in pilot decision-making and the prevention of
aviation accidents. However, government and industry have persisted in
the perception of the primary roles of the flight attendant as service
and survivability. Moreover, the cockpit and cabin crews of today have
evolved into two distinct cultures (Chute & Wiener, 1994). These factors
have resulted in communication and coordination problems which have
jeopardized flight safety.

In 1989, 24 people died when an Air Ontario jet crashed on takeoff from
Dryden, Ontario due to wing contamination. Despite concern on the part
of the flight attendants, they did not notify the pilots because they
had been trained to trust the judgment of the pilots and not to question
it. Additionally, past experience had shown that pilots treated
operational concerns expressed by the cabin crew with disdain. The Board
of Inquiry cited the organizational policy of Air Ontario that
reinforced the suppression of operational information by the flight
attendants (Moshansky, 1992).

In 1988, on approach into Nashville, an American Airlines flight
attendant and an off-duty first officer notified the cockpit of smoke in
the cabin. The captain was skeptical of their report of smoke as there
had been a problem with the auxiliary power unit (APU) on a prior flight
which resulted in fumes. This time the problem was the result of
improperly packaged hazardous materials. Even when informed that the
floor was becoming soft and passengers had been reseated, the cockpit
crew persisted in refusing to acknowledge that there was serious
jeopardy to the aircraft and their passengers. No in-flight emergency
was declared. Consequently, the aircraft was not evacuated immediately
on landing, exposing the crew and passengers to the threat of smoke and
fire longer than necessary. The NTSB determined the cabin crew used CRM
techniques well; however the cockpit crew did not. The NTSB found a
\"deficiency in communication between the cockpit and cabin crews and
expressed concern about the reluctance by the captain to accept either
crew member\'s report as valid or to seek additional information.\"
(NTSB, 1988).

Our past research has shown that crews do not see themselves as one
team, but rather as two crews with separate responsibilities This
division can result in territorial, and even hostile, attitudes. For
example, when on a flight conducting field research, I observed a
situation where the refusal of the flight engineer to check out an
inoperational lav resulted in the cabin crew withholding food from the
pilots. Food seemed to be the only currency the flight attendants had to
express their frustration. However, both crews brought the aircraft in
on a five-hour leg, and I picked it up on a subsequent five-hour leg.
During that flight, the pilots were only given nuts for sustenance out
of retaliation. Is it reasonable to assume that in an abnormal situation
the crews would have transferred information to each other without
hesitation? Moreover, in an emergency would those crew members have
instantly united and trusted one another?

We have identified five factors which either created or perpetuate
barriers between the two cultures: Historical background, physical
separation, psychosocial issues, regulatory factors, and organizational
factors (Chute & Wiener, in press). Some manifestations of these
influences are: discrepancies in manuals and procedures; attitudes such
as distrust, alienation, and skepticism; withholding important safety of
flight information due to fear of rebuke or disciplinary action; an
unawareness of the otherÕs duties during flight; and insufficient
technical knowledge on the part of the cabin crew to communicate with
optimal effectiveness and timeliness regarding aircraft anomalies.

It should be emphasized that, in addition to survivability duties
post-crash, flight attendants also have a significant *preventive* role
which has been discounted. This is where effective bi-directional
communication becomes vital. Flight attendants are sometimes privy to
abnormal sights and sounds in the aircraft of which the pilots are
unaware. Automated flight decks and the two-pilot crew have accentuated
the role of the flight attendants in the detection and reporting of
mechanical anomalies aft of the cockpit door which was formerly the
province of the flight engineer. On long flights, especially at night,
flight attendants can play a significant role in aiding pilotsÕ
vigilance with periodic checks and conversation. Additionally, the
serving of food to the pilots can, and should, be viewed as a safety
duty (as evidenced by my flight experience related earlier). It is the
safety, rather than the service, duty of the cabin crew to ensure that
the pilots are as alert and properly nourished as possible. The cabin
crew can also ameliorate potentially serious situations, such as fires,
troublesome passengers, and medical emergencies by timely and
coordinated actions. Any impediment in the teamwork of the cockpit and
cabin crews can lead down a slippery slope to an unmanageable crisis.
Currently, the types of data collected (primarily post-crash and
survivability data) do not reflect the preventive responsibilities of
the cabin crew nor the nature of interactions between the crews and the
proactive role of the flight attendants.

In addition to generating our own data, we review accident and incident
reports from the NTSB and ASRS. Yet there is a dearth of information
collected in any systematic manner on the interface between the cockpit
and cabin crews. The NASA Aviation Safety Reporting System has
developed, with the assistance of the Cabin Safety Issues Identification
Team (Nora Marshall, Pat Coleman, and myself), a cabin crew reporting
form that will greatly enhance our knowledge of the safety issues that
transpire aft of the cockpit (e.g. turbulence injuries, mechanical
malfunctions, passenger disruptions, etc.). Additionally, this
information would supplement cockpit/cabin communication reports that
are already being received from pilots by giving a more complete picture
instead of the limited view currently available. For example, Vicki
Hoang and I recently conducted an analysis of cabin turbulence injuries
by categorizing and analyzing 79 ASRS reports (Hoang & Chute, in press).
However, not one report was a first-person account by a witness in the
cabin. Data is also needed that illuminates any flight attendant
reluctance or pilot resistance in the information-transfer system.

The importance of the focus of human factors research for the next
decade moving into cockpit and cabin crew coordination was underscored
in the following statement by Chidester (1993): \" The unit of analysis
should become the flight deck and cabin crew. Almost every safety
problem encountered on one side of the cockpit door soon becomes a
problem for the other. Coordination between these parts of the crew has
been assumed in the operational community and unstudied by the research
community.\" Additionally, the National Plan for Civil Aviation Human
Factors (1995) cites Òcoordination between the flight deck and cabin
crews in abnormal situationsÓ as a specific research need. The research
begun by Chute and Wiener, necessarily scratches the surface of the
deeper investigation needed in these critical areas. Increased emphasis
on the importance of the interface between the two crews, the preventive
responsibilities of the cabin crew, a corresponding increase in
available, relevant data, and support for future studies will ensure
that cockpit/cabin deficiencies do not continue to jeopardize flight
safety.

**References**

Chidester, T. R. (1993). Critical issues for CRM. In E. Wiener, B.
Kanki, & R. Helmreich (eds.) [Cockpit Resource Management.]{.underline}
San Diego: Academic Press

Chute, R., & Wiener, E. (1994). Cockpit and cabin crews: Do conflicting
mandates put them on a collision course? Flight Safety Foundation [Cabin
Crew Safety,]{.underline} [29]{.underline} (2).

Chute, R., & Wiener, E. (in press). Cockpit-cabin communication: I. A
tale of two cultures. [The International Journal of Aviation
Psychology]{.underline}.

Chute, R., & Wiener, E. (in press). Cockpit-cabin communication: II.
Shall we tell the pilots?. [The International Journal of Aviation
Psychology]{.underline}.

Federal Aviation Administration (1995). [The National Plan for Civil
Aviation Human Factors: An Initiative for Research and
Application.]{.underline} Washington, D. C.

Hoang, V. R. & Chute, R. D. (in press). Cabin turbulence injuries: An
analysis of Aviation Safety Reporting System incidents. [Proceedings of
the Twelfth International Aircraft Cabin Safety Symposium.]{.underline}

Moshansky, V. P. (1992). [Commission of inquiry into the Air Ontario
crash at Dryden, Ontario.]{.underline} Toronto, Canada.

National Transportation Safety Board. (1988). [Hazardous Materials
Incident Report:]{.underline} [Inflight Fire, McDonnell Douglas DC-9-83,
N569AA, Nashville Metropolitan Airport, Nashville, Tennessee, February
3, 1988]{.underline} (NTSB/HZM-88/02). Washington, DC: Author.

## ICAO VIEWS ON THE APPLICATION OF HUMAN ERROR MODELS TO ACCIDENT AND INCIDENT DATA - Stephan J. Corrie

Chief, Accident Investigation and Prevention Section, ICAO

**Summary**

Human factors data pertaining to flight crew error is obtained by ICAO
primarily from reports of accident and incident investigations conducted
by Contracting States. In return, this data is provided to all
Contracting States for their use in preventing accidents. The
international provisions which deal with the investigation and reporting
of aviation occurrences is provided in *Annex 13* to the Chicago
Convention and in the *Manuals Of Aircraft Accident Investigation* and
*Accident/Incident Data Reporting (ADREP)*. Human factors data elements
were upgraded and incorporated into the ADREP computer data base system
in 1984. The work reflected the state-of-the-art of investigation
authorities in the early 1980\'s. This initiative resulted from
recommendations from the 1979 Accident Investigation and Prevention
Divisional Meeting and the ADREP Study Group formed after the meeting.

While many models of human error have been developed for various
purposes, there are limitations in the ability to conduct error analyses
from human factors data in accident/incident data bases. The causes of
human error can be complex and any examination must include the entire
scenario in which the error occurred. As a result, it can be very
difficult to select a common error model to cover all conditions and
circumstances under which error will occur and from which it can be
analyzed.

The quantity and quality of human factors information produced in an
investigation is dependent on the methodology used to, collect, organize
and analyze the data gathered. Of the 183 Contracting States in ICAO,
there are vast differences in their investigation capabilities and
cultures, and thus in the extent of human factors information contained
in their reports. Furthermore, inconsistent ADREP reporting has resulted
in ICAO placing more emphasis on human error associated with known
accident/incident precedents and scenarios, from which to learn how to
mitigate human error, rather than on attempting to look for trends and
conduct research using such data.

In 1993, ICAO established the ADREP 2000 Study Group to assist the
Secretariat in making several enhancements to the present system. One of
these areas pertains to human factors which will involve reviewing the
present classifications for adapting it to advances in the human factors
field. This work has stagnated and ICAO is eager to obtain the
additional support needed from those States which nominated the members
on the study group in order to successfully complete the work.

ICAO will continue to improve its ADREP system and is interested in any
further progress made by States and the industry to improve the
reporting, collection and classification of human performance data for
understanding the nature of human error and for accident prevention
purposes. In view of the limitations of data models, ICAO must emphasize
the importance of activities leading to the development of standardized
methods of exchanging safety data as a means to improve international
capabilities to identify hazards and safety deficiencies and for
formulating accident prevention strategies.

**ICAO Safety Activities**

**Accident Prevention**

Of the many issues facing the air transport field, the relationship
between economics and safety is enduring. While governments and the
aviation industry strive to ensure that commercial air transport
operations are conducted in a manner that does not adversely affect
safety, economics inescapably plays an important role in the major
decisions of organizations in their pursuit of this endeavor. The task
of balancing the production goals of an organization against safety
considerations is difficult. Yet, managers are in control of finances
and resources and, therefore, are directly responsible for managing
safety. Accordingly, ICAO has tried to emphasize the need for decision
makers to be continually aware of the risks involved in their decisions
and of the overall impact these decisions have on the safety of their
operations. In this context ICAO established the Flight Safety and
Accident Prevention and Flight Safety and Human Factors programs
designed to educate and influence the decision makers into becoming
better safety managers.

In 1984, ICAO published the *Accident Prevention Manual* to assist
States and their industry, to understand the concepts and activities
involved in accident prevention and to develop a system and programs
designed to identify, eliminate or control aviation hazards before they
result in accidents. Attempts are being made to update this manual based
on successful prevention initiatives in States. In addition, a draft
Assembly Resolution has been prepared calling on States to improve
accident prevention and it will be on the Assembly agenda for its
meeting this September.

**Human Factors**

As a result of Assembly Resolution A26-9 in 1986, ICAO implemented the
Flight Safety and Human Factors program. The objective of the program is
to improve aviation safety by making States more aware and responsive to
the importance of human factors in civil aviation through the
publication of practical human factors material and the successful
measures developed and taken on the basis of experience in States. It
focuses on the organizational aspects of the aviation system and on the
essential role played by management in fostering a system tolerant to
unavoidable human errors. In this regard, ICAO recently published Digest
No. 10 - *Human Factors, Management and Organization*. ICAO is also
fostering the development and use of practical tools which will help
management balance effectively production goals and safety
considerations. The program has developed several other digests which
provides additional guidance for enhancing the examination of human
factors stemming from accidents and incidents. The program complements
the guidance material in the *Accident Prevention Manual* and helps to
integrate current human factors knowledge in prevention methodology.

**Accident and Incident Investigation**

ICAO published many years ago the *Manual Of Aircraft Accident
Investigation* to assist States in conducting investigations in
accordance with the provisions in Annex 13. The manual outlines the
procedures for organizing and conducting an accident investigation and
contains detailed information on how to apply many of the specialized
areas of investigation. While the manual has been amended, its last
major edition was in 1970. Based on recommendations of the 1992 Accident
Investigation and Prevention (AIG) Divisional Meeting, an expert group
called the Accident Investigation Methodology Study Group (AIMSG) was
formed to assist ICAO in improving the format of the final written
report, an appendix to *Annex 13*, in order to encourage examination of
the deeper, systemic causes of accidents. The group is also tasked with
updating the investigation manual. The study group met in April of this
year and made substantial progress in these two areas. A complete new
edition of the investigation manual is expected to be drafted by the end
of 1996 and published in 1997.

**ADREP System**

The implementation of the ADREP system in 1976, was designed to provide
computer-generated information routinely on the circumstances and causes
of accidents and incidents, as determined by national authorities, for
use in the technical work program of ICAO and for dissemination to
Contracting States for prevention purposes. The success of this program
is dependent on the consistent reporting of aviation occurrences and on
the dissemination of the ADREP bi-monthly Summary reports from ICAO to
Contracting States and the industry.

An upgrade of the human factors data elements in the ADREP computer data
base system was made in 1984. It reflected the state-of-the-art by
investigation authorities in the early 1980\'s and resulted from
recommendations from the 1979 AIG Divisional Meeting and the ADREP Study
Group formed after the meeting. As a result of recommendations from the
1992 AIG Divisional Meeting, ICAO established the ADREP 2000 Study Group
in 1993 to assist the Secretariat in developing an expanded, integrated
incident information system, based on ADREP classifications.

ICAO has co-operated with selected States, through the informal
International Data Exchange for Aviation Safety (IDEAS) Group, in the
development of a standardized data exchange interface with its members.
ICAO has consulted with this group and members of the ADREP 2000 Study
Group (ADREPSG) to determine the scope and structure of the additional
incident data to be collected. It is envisaged that an integrated
accident/incident data base will be created by the end of 1996, but this
new data base will require additional support from the study group
members to achieve this objective.

It is essential that an integrated system for collecting and
disseminating accident and incident information be developed in order to
address more effectively safety hazards and deficiencies in the air
transportation system. The study group is reviewing existing accident
and incident data bases, on-line access and statistical methods as the
basis for developing this new integrated system. One of the other areas
under consideration is human factors which will involve reviewing the
present classifications for adapting it to advances in the human factors
field.

The ADREP system has recently been upgraded with new hardware and
software which will provide it with the capability to furnish better
access to timely integrated safety information. The AIG Section, which
operates ADREP, is now linked to the Internet mail system and is an
effort to improve the section\'s capability to communicate with States
more efficiently. This system has been used for about one year on a
trial basis. Results are encouraging. States wishing to communicate with
AIG are requested to send their communication to:
**\"rmenzel@icao.org\"**.

**Data Issues In Analyzing Human Error**

**Data Relevance**

The method of determining the relevance of human factors data as it
applies to causes of accidents and incidents is linked to the entire
investigation process and the quality of the evidence available. How the
available evidence is gathered and examined in an investigation is
central to that process and occasionally is an issue among aviation
investigator circles. Not only can the investigation results differ from
what is needed to satisfy the questions on the ADREP system data base
forms, but widespread differences between the investigation capabilities
of the 183 Contracting States of ICAO results in difficulties in
capturing relevant data on the causes of human error.

The prominent accident model used by Contracting States is the
chain-of-events, single to multiple cause. The investigation attempts to
identify all the causes that contributed to the chain of events that
culminated in the accident. There are limitations to this model and the
data provided often tends to describe what happened in the accident, but
not why nor why it was not prevented. It should be noted, however, that
improvements have been implemented to eliminate the single cause concept
and to stress the need for identifying systemic causes as a result of
the 1992 AIG Divisional Meeting and publication of Amendment No. 9 to
*Annex 13*.

In an attempt to provide better guidance to investigators for
investigating and reporting on relevant human factors data from
accidents and incidents, ICAO produced Digest No.7, *Investigation of
Human Factors in Accidents and Incidents*. The digest focuses on using
the model developed by Professor James Reason as a means to help guide
investigators in the identification of systemic factors. It also
mentions the SHELL model, developed by Captain Frank Hawkins, as a means
to help the investigator analyze detailed factors which may underlie
human error. An example of an analysis method used by a particular State
is included in Digest No.7 and the example involves a test for the
existence, influence and validity of the human factors evidence for
causal determinations.

**Use of Human Error Models**

It is difficult to identify all of the foreseeable human factors issues
which may require future research. Consequently, it is difficult to
design a data base system to answer all potential questions concerning
these future issues. The ADREP system contains a wide variety of data
elements to satisfy a variety of needs. However, experience has shown
that the amount of human factors details presently in ADREP is not
adequate for in-depth research into human error. At the same time, it is
capable of providing the entire context in which errors occur and can
indicate where human factors deficiencies exist in any occurrence.
However, the amount of detail found in the ADREP data and final written
reports can only reflect the thoroughness and depth of the
investigation.

If a human error model is used to develop a data structure for a
computer data base, then the data elements needed are collected in an
investigation of accidents/incidents to fit the structure required by
the model. Consequently, in order to ensure the validity of the data
structure, an investigation must be performed so that the expected level
of detail concerning human error in an accident/incident sequence can be
determined with some level of confidence. This model approach, however,
may make it difficult to use the data collected for other purposes and
for applying it to other models. More importantly, the model may not
provide a full understanding of the accident.

If a model is to be used by ICAO, it must be compatible with the ADREP
multi-event causal model and useable by the Contracting States of ICAO.
It is essential that data models be kept simple so that they can be
easily understood and applied by investigators. Otherwise, investigators
may find it difficult to obtain the data elements required by the model.
The lack of adequate resources, legal and financial constraints can also
preclude collecting all desirable human factors information.

The *ADREP* manual provides guidance to investigators on how to code the
evidence gathered in the investigation for input into the computer data
base. In addition to the numerous codes that have been developed for
gathering routine statistical information, the ADREP system incorporates
a multi-event causal model with descriptive and explanatory factors
which can describe how and why the accident occurred. As the model
applies to the coding of human factors, a variety of subject codes,
under the descriptive category, can be used to identify general causal
factors. These factors can then be explained further under the category
of organizations and persons by using various other subject factors and
modifiers. For example, flight crew performance could be identified as a
general causal factor by using the ADREP classification, *flight crew
procedures/crew co-ordination/inadequate* which could be further
explained by using the classification , *operator-training
staff-instructor/experience-competence/unacceptable*. The ADREP model is
flexible enough to allow for further detail depending, once again, on
the depth and methodology used in the investigation. It is only limited
to the number of factors which can be developed based on objectives and
experience.

The ICAO ADREP causal model is similar to the existing NTSB model.
However, NTSB data must be converted using software programs and the
results reviewed before the data is added to the ADREP data base. The
ability to use the ADREP system effectively requires training and
practice. The manual needs to be updated because of the numerous
improvements that have been made since it was published in 1987.

**Data Quality and Quantity**

In addition to the factors mentioned previously, another fundamental
issue affecting quality and quantity of human factors data is whether
safety specialists take a statistical approach to identifying safety
problems or consider the value of known precedents as more suitable.
Depending on which approach is taken will determine, in part, the
quality and quantity of data. We do not favor a statistical approach in
all cases for reasons mentioned earlier. The use of accident data for
statistical purposes is not very helpful because of the limited amount
of information available. Moreover, the industry cannot rely on the use
of accident data to identify adverse trends. Inconsistent ADREP
reporting has resulted in ICAO placing more emphasis on human error
associated with known accident/incident precedents and scenarios, from
which to learn how to mitigate human error, rather than attempt to look
for trends and conduct research using such data. However, the prevalence
of incident data is more valuable for prevention purposes and suitable
for trend analyses.

There may be limited resources for investigating incidents and minor
accidents, thus any data model must also allow for some classification
of very general findings so that this limited information is available
for analyses. Simplicity is favored. This also helps to ensure that the
data collected contains fewer errors.

**Identifying Performance**

Any taxonomy or model of human error must encompass the psychological,
situational and organizational factors which can affect individual
behavior. The model should permit assessing the effectiveness of the
different controls used by organizations to foster safe and compliant
individual behavior. Such controls may be administrative, technical or
social in the nature. Errors can be a consequence of inherent human
limitations and can result from inadequate communications, policies and
procedures. Violations, on the other hand, can either be deliberate or
unintentional and occur within a regulated environment. An examination
of performance must consider the adequacy of rules and regulations in
conjunction with compliant behavior. The objective of assessing human
performance should be to encourage and assist organizations to develop
controls which are suitable for guiding safe and productive individual
behavior within which the organizations operate. This objective should
be a part of any accident prevention program. The program will reduce
violations by including motivational and organizational remedies, and
will foster a safety-oriented corporate culture.

**Accessibility**

Privacy legislation in several States prohibits the exchange of
personnel related information between States. Thus provisions are needed
to de-identify the data for international exchange.

Another important problem is the question of whether access should be
given to the public at large. Efforts underway to encourage \"penalty
free\" reporting could be jeopardized if the information in human
factors related data bases is used for purposes other than accident
prevention. In addition, the potential for litigation during the course
of an accident investigation can stifle a free exchange of information
and prevent access to valuable data.

ICAO is proceeding carefully on the question of making its data publicly
available on the \"information highway\". Information will continue to
be provided to authorized officials in ICAO, Contracting States and
international organizations in a form suitable for their needs. However,
ICAO will have to review its policies on ADREP data dissemination in
order to provide more efficient access to this data by the industry.

**Conclusions**

The relevance, quantity and quality of human factors information from
accidents and incidents is dependent on the methodology used and depth
of the investigation, and the capabilities of States. Legal and
financial factors also play a role. The potential value of human factors
information from incidents versus accidents for prevention purposes is
unquestionable. ICAO moved forward to stress the importance of incident
investigations through Amendment No. 9 to *Annex 13*.

Insofar as this idea is accepted, it must be recognized that there are
limitations to conducting error analyses from human factors data found
in final written reports and accident/incident data bases from accident
and incident investigations. Difficulty in obtaining consistent
reporting from Contracting States is another factor in the quantity of
human factors information made available from official government
sources.

Additional efforts to develop a human error model that is flexible
enough to accommodate all error situations and simple enough for
investigators internationally, to use to satisfy the data requirements
of the model, is a significant challenge to States and the aviation
industry. ICAO is not optimistic about success in developing such a
model. In view of the fact that extensive knowledge about human error is
already available, there appears to be more justification for applying
it to mitigate error and limited value in pursuing further attempts to
model and study human error in the short term. Although the causes of
human error can be complex, any examination must take into account the
complete environment, conditions and circumstances under which error
occurred before it can be understood. Thus, ICAO will continue to
emphasize and encourage the improvement of investigation and prevention
processes in States and the industry.

ICAO will continue to improve its ADREP system through the support of
the ADREP 2000 Study Group, coordination with the IDEAS Group and the
support from States. Thus, ICAO will appreciate the additional support
needed to successfully complete the work on the study group agenda from
those States which nominated the members. It is very interested in any
further progress made by States and the industry to improve the
reporting, collection and classification of human performance data for
understanding the nature of human error. In the long term, such efforts
as this workshop, could prove beneficial.

Inasmuch as ICAO recognizes the present limitations of investigations
and data models, it must reemphasize the importance of activities
leading to the development of standardized methods of exchanging safety
data as a means to improve international capabilities to identify
hazards and safety deficiencies and for formulating accident prevention
strategies.

**Recommendations**

1\) Emphasize the importance of improving investigation and prevention
methodologies, processes and skills.

2\) Apply existing human error knowledge aggressively through prevention
processes to improve the effectiveness of hazard identification,
elimination and control.

3\) Continue developing knowledge about the causes of human error on a
long term basis, consistent with technological and operational changes
and integrate this knowledge into operations.

4\) Develop easy to use human error data models giving consideration to
the difficulties in obtaining human factors data through investigations
and to their international application and compatibility with the ICAO
ADREP system.

5\) Continue to actively support and participate in the development of
standardized methods of exchanging safety data through the IDEAS Group.

## ASSESSMENT OF HUMAN ERROR FROM TRANSPORTATION ACCIDENT STATISTICS - Vernon S. Ellingstad and David L. Mayer

National Transportation Safety Board

**Introduction**

Baker and Lamb (1992) have recently reported on a study of commuter and
air taxi accidents during the period from 1983 through 1988 they
obtained data from the National Transportation Safety Board\'s Aviation
Accident Data Base on a total of 719 fixed wing aircraft involved in 122
commuter and 597 air taxi accidents and subjected these data to an
extensive process of analysis. They identified twelve major crash
categories (as well as an \"other\" and an \"undetermined\"
classification) that provided useful groupings of the Part 135 accidents
for more focused analysis. They also evaluated each accident record to
determine whether factors associated with (a) the pilot, (b) ground
personnel, (c)air traffic control, (d) aircraft malfunction, (e) airport
conditions, and (f) weather had contributed to the accidents. Pilot
condition or pilot error was identified in about 74 percent of these
accidents. Human factors issues such as fatigue, improper procedures,
and decision errors were observed for individual cases and emerged as
safety issues when the cases were aggregated. The Baker and Lamb study
provides a useful description of an important class of aircraft
accidents.

At last year\'s Transportation Research Board meeting Hegwood (1992)
presented an analysis of general aviation accidents from 1988. She
attempted to evaluate the prevalence of human factors issues in these
accidents by applying a modification of Feggetter\'s (1982) checklist to
a sample of 50 general aviation accident records in the NTSB data base.
She coded cognitive, social and situational human factors in these
accidents after inspecting the NTSB factual reports, briefs of accident
and accident narratives. Her analysis identified human factors as
contributing to 90 percent of the 50 accident sample, as compared to 82
percent that had been originally identified by the NTSB as caused or
contributed to by human factors flaws in information processing (80
percent of the accident sample) and errors in judgment or decision
making (66 percent of the sample) were particularly notable findings.
Again, this study provides useful descriptive information to the
aviation safety and human factors communities by examining aviation
accidents in the aggregate.

On October 14, 1992 the Safety Board adopted a study of alcohol and
other drug involvement in fatal general aviation accidents during the
period from 1983 through 1988. This study revealed a small decline in
the number and percent of alcohol related general aviation accidents
over the study years, to a rate of about 6 percent in the late 1980s. A
slightly higher proportion of alcohol related fatal (to the pilot)
crashes occur at night than is the case for fatal to the pilot crashes
that do not involve alcohol. Disappointingly, no strong evidence of
differential causation between alcohol involved and non-alcohol involved
accidents emerged from the study \-- that is to say we did not discover
human failures that were clearly associated with alcohol impairment in
these accident records. This study depended, of course, on factual and
analytic data derived from the NTSB Aviation Accident Data Base.

As a final example of what I am sure that you have guessed by now to be
illustrations of the application of accident data bases (and their
associated accident statistics) I would like to mention a study that is
currently in progress in the Safety Studies Division at the Safety
Board. This study is an assessment of flight crew performance in Part
121 air carrier accidents determined by the Board to have involved
flight crew error. Ben Berman and his colleagues are now in the process
of refining taxonomies of flight crew errors that were identified
through a detailed analysis of accident data, including factual and
analytic records, as well as cockpit voice recorder transcripts and
other investigative information. They are also deriving, from the same
data sources, empirical characterizations of operational factors such as
workload, situational awareness and communication flow whose
relationships to flight crew error can then be assessed. We hope that
this analysis, in the aggregate, of a fairly large collection of major
air carrier accidents will reveal some of the human performance issues
that may not be readily apparent in a single accident.

The balance of this paper will explore a couple of issues that affect
the usefulness of accident databases for safety research generally, and
human factors research in particular. Mayer and Ellingstad (1992) note a
number of problems in the use of accident data bases designed for
purposes other than research and analysis, including: treatment of
missing data; database structure and design; and representativeness of
the records in the database. These are important technical
considerations that will influence the quality and usefulness of
accident research, but they are outside the scope of our discussion
today. Instead, I would like to concentrate on two issues: (a)the
importance of examining accidents in the aggregate, and (b)the need for
improving our measurements of \"cause.\"

**Why Bother With Accident Statistics?**

The National Transportation Safety Board (NTSB) is a premier accident
investigation agency and it produces definitive analyses of individual
transportation accidents. These analyses are based on extensive field
and laboratory investigations, a party system that ensures the
consideration of widely differing points of view, and very extensive
deliberation. They produce, in most cases, a formal statement of the
Òprobable cause,Ó of the accident, and, where appropriate,
recommendations for action to correct safety defects. You will shortly
have evidence from my colleagues Jim Danaher and Jerry Walhout of the
impressive scope and quality of these investigations. Why then do we
bother to collect and analyze collections of accident data stored in our
computers?

The first answer to this question has to do with seeing the forest, in
addition to all of the individual trees that are represented by the
separate accidents. Assessment of accident trends requires the
aggregation of data from all of the individual accidents that are
investigated. Standardization of data elements and methods of data
collection have obvious importance in accounting for the patterns of
accidents over time, as do considerations of reliability and validity of
the data that these trends are based on.

A second, and perhaps even more important rational for aggregate
analyses (accident studies) is that accident causes are not always
evident, even to the most extensive, well organized, and professionally
conducted single investigation. Sometimes this is due to the presence of
what we might call \"weak causes,\" influences which, in a statistical
sense, account for only a modest (but reliable) proportion of the
variance. other accidents, or classes of accidents, may be produced by
multiple causes that interact in complex ways. It should not be
surprising that the kinds of causes that we are focusing on today \--
the human factors \-- are often (if not usually) both weak and multiple.

Finally, transportation accidents always occur in a context that must be
understood and accounted for. The influence on accidents of factors such
as operator workload, hours of service, task complexity, and the like
can probably only be understood statistically \-- that is, on the basis
of aggregate studies of accidents for which the requisite human
performance data has been collected.

**Measuring Accident Causes**

The Safety Board makes an important formal distinction between \"fact\"
and \"analysis\" in its investigation of accidents. The investigative
process of yields a body of \"fact\" that describes and documents the
accident circumstances and that supports \"analysis\" intended to yield
an assessment of probable cause. Similarly, in addition to a collection
of factual information, the accident database may include analysis and
some representation of the cause(s) of accidents.

One of the implicit assumptions of accident analysis has always been
that if the *cause* of an accident is known, similar accidents can be
prevented in the future. This notion has its

roots in fault tree analysis. If specific accident-producing modes of
failure can be identified, then accidents can be prevented by
strengthening these weak links. Some modes (e.g., metal fatigue or tire
failure) are relatively well understood and, more importantly, they
leave identifiable

physical traces that survive the accident, however, generally leave
little direct evidence for later analysis. Consequently, accident
databases usually capture more information representing hardware
failures and other directly observable phenomena, than human errors.

Grouping similar accidents by type or category is perhaps the simplest
and most common representation of causation in accident data bases.
While it is often possible to classify accidents as belonging to a
specific type (e.g., mid-air collision, VFR into IMC, loss of control,
etc.), this rarely explains why an accident occur. Accidents \-- even
relatively simple ones \-- often result from multiple causes.

Some accident data bases address this issue by recording a narrative
statement of accident causation, generally produced by a trained
analyst, using a somewhat structured vocabulary. The Aviation Safety
Reporting System (ASRS) maintained by NASA (Rosenthal and Mellone, 1989)
utilizes this kind of text-based key-word system. The NTSB Aviation
Accident System also contains a 200 work narrative statement of probable
cause, although this is not the primary method of recording accident
causes in the NTSB database. While this approach provides the
opportunity for rich expression of causal relations, methods of analysis
for text data are, at present, limited.

The current NTSB aviation database uses a somewhat more complex coding
system that identifies from one to five *\"occurrences\"* (see Figure 1)
that make up the accident Sequence *of Events* Associated with each
occurrence is a \"Phase of Flight\" code (see Figure 2).

For each occurrence/phase of flight recorded the accident investigator
also records a set of coded explanations or \"findings\" that account
for that occurrence. A primary set of findings consisting of a
\"subject\" (23107 - Altimeter), a \"modifier\" (3121 - Misread), and a
\"person\" (4000 - pilot in command) can be entered to account for the
occurrence. An underlying explanatory factor (e.g., 33130 - physical
impairment, alcohol; pilot in command) can also be associated with this
occurrence. The sequence of events system is intended to comprehensively
represent the events in a single accident in a formal coding structure
that permits the examination of common patterns across accidents of
particular types.

This approach is complicated somewhat by the fact that more than one
\"sequence of events\" may be necessary to account for a particular
accident. In many accidents a simple chronological listing of
occurrences in the order in which they occur is sufficient to account
for accident causation. In other circumstances the causal sequence of
events may be different from the temporal sequence of events. This is
particularly true when factors that significantly pre-date the accident
sequence of occurrences (e.g., maintenance failures, pilot sleep loss,
etc.) must be causally associated with accident events.

An additional complication in attempting to capture the details of
accident causation in a sequence of events coding structure concerns the
assessment of relationships between multiple accident factors or
findings. It would be useful, for example, to assess the extent to which
the pilot\'s sleep loss contributed to his vigilance decrement, and how
much that in turn contributed to failure to detect a critical signal.
Current database redesign efforts at the Safety Board are directed to
the incorporation of such information in the sequence of events data
system.

A related issue in quantifying accident causation is the assessment of
the strength of the relationship of each separate occurrence or factor
in the sequence of events to the accident itself. Military aviation
investigation systems have, for example, indicated which event in the
sequence made the accident inevitable. The Safety Board does not
presently code that information.

While possessing great potential explanatory and analytic power, coded
representations of causal chains such as that just described can be very
complicated to use. Current efforts to improve the Safety Board\'s
database are directed to improvements in this area as well.

**Additional Information Needed To Account For Human Causes**

In addition to documentation of the factual aspects of an accident and
an assessment of causation; a human factors analysis is an important
component of a full investigation. In this context \"human factors
information\" must be understood to refer to a complete accounting of
human-equipment interaction in the accident situation, and not the
\"mental state\" or disposition of the people involved in the accident.
There must, for example, be a thorough accounting of task demands placed
on the operator as well as the operational requirements of the task(s).
Preferably, this analysis should be standardized across all accidents in
the database. In effect, what is needed is a retrospective task analysis
which helps to identify and code system failures. Drury (1983) detailed
several such alternatives for coding consumer product accidents, but no
such method has emerged for transportation accidents. The need for
standardization and the realization that not all accident investigations
will be conducted by professionals trained in human factors, suggests
that checklists or other \"cookbook\" methods may be needed.

**Conclusions**

Transportation accident databases will continue to provide the primary
basis for most empirical diagnoses of safety problems and evaluations of
safety countermeasures. Improvements in database technology as well as
database design can be expected to make these sources of information
increasingly useful but significant attention must be directed to
improving both the collection and analysis of relevant data regarding
the circumstances, contexts and causes of accidents \-- and particularly
the human factors.

Task-oriented human factors information about accident scenarios is
often missing or unusable in transportation accident databases. This
kind of information is sometimes overlooked

because of an inadequate understanding of human factors by accident
investigators. More often, however, these data are not collected because
human failings do not leave the same kind of permanent physical traces
that broken vehicular components do.

Sometimes human factors information, and other analytic findings, are
collected but not coded well or completely. Improved methods of
quantifying causality, and representing relationships between multiple
causes are needed to render databases more useable in this regard.

Human factors researchers should and need to use accident databases in
their work, but great care must be taken to use these tools effectively.
Greater participation by researchers in the design of databases and the
collection of data will increase their suitability for our work.

**References**

Baker, S.P. and Lamb, M.W.(1992) Human factors in crashes of commuter
airplanes and air taxis. Final Report, FAA Contract #DTFA01-90-C-00046.

Drury, C. and Brill, M. (1983). Human Factors in consumer product
accident investigation. [Human Factors]{.underline}, 25(3).

Feggetter, A.J. (1982). A method for investigating human factor aspects
of aircraft accidents and incidents. [Ergonomics]{.underline}, 25**,**
p. 1065-1075.

Hegwood, J.A. (1992). Application of modified Feggetter model to
identification of selected human factors in 1988 general aviation
accidents Paper presented at the 71st Annual Meeting of the
Transportation Research Board, Washington, DC.

International Civil Aviation Organization. (1987). Accident/Incident
Reporting Manual. Author: Montreal, Quebec, Canada.

Mayer, D.L. and Ellingstad, V.S. (1992) Using Transportation Accident
Databases in Human Factors Research, [Human Factors Society Annual
Meeting]{.underline}, Atlanta, GA.

National Transportation Safety Board (1992) [Alcohol and Other Drug
Involvement]{.underline} in [Fatal General Aviation Accidents. 1983
Through 1988]{.underline}, Washington, D.C.

Rosenthal, L.J. and Mellone, V.J. (1989). Human factors in ATC
operations: anticipatory clearance in ASRS Research Workshop: [Human
Factors Work Group]{.underline}.

**Figure 1**

NTSB OCCURRENCE CODES

100 ABRUPT MANEUVER

110 ALTITUDE DEVIATION, UNCONTROLLED

120 CARGO SHIFT

130 AIRFRAME/COMPONENT/SYSTEM FAILURE/MALFUNCTION

140 DECOMPRESSION

150 DITCHING

160 DRAGGED WING, ROTOR, POD, OR FLOAT

170 FIRE/EXPLOSION

180 FORCED LANDING

190 GEAR COLLAPSED

200 HARD LANDING

210 HAZARDOUS MATERIALS LEAK/SPILL

220 IN FLIGHT COLLISION WITH OBJECT

230 IN FLIGHT COLLISION WITH TERRAIN/WATER

240 IN FLIGHT ENCOUNTER WITH WEATHER

250 *LOSS* OF CONTROL - IN FLIGHT

260 LOSS OF CONTROL - ON GROUND

270 MIDAIR COLLISION

280 NEAR COLLISION BETWEEN AIRCRAFT

290 NOSE DOWN

300 NOSE OVER

310 ON GROUND COLLISION WITH OBJECT

320 ON GROUND COLLISION WITH TERRAIN/WATER

330 ON GROUND ENCOUNTER WITH WEATHER

340 OVERRUN

350 LOSS OF ENGINE POWER

360 PROPELLER BLAST OR JET EXHAUST/SUCTION

370 PROPELLER/ROTOR CONTACT TO PERSON

380 ROLL OVER (Helicopter)

390 UNDERSHOOT

400 UNDETERMINED

410 VORTEX TURBULENCE ENCOUNTERED

420 MISSING AIRCRAFT

430 MISCELLANEOUS/OTHER

**Figure 2**

NTSB PHASE OF OPERATION CODES

500 STANDING

510 TAXI

520 TAKEOFF

530 CLIMB

540 CRUISE

550 DESCENT

560 APPROACH

570 LANDING

580 MANEUVERING

590 HOVER

600 OTHER

610 UNKNOWN

## PRACTICAL ISSUES IMPACTING HUMAN FACTORS INVESTIGATIONS AT THE AIRLINE LEVEL - Captain L. R. Ganse

Director Flight Safety

Northwest Airlines

In the world of the air safety investigator data on human performance
can be difficult to obtain. Most of us have been trained very well on
the technical aspects of incident or accident investigation. However,
our education on human performance issues is mostly gained from
experience and from some outside reading or seminars such as this. To
the best of my knowledge, in the airline world, no one has the luxury of
keeping a trained human performance expert on the very small safety
staffs currently employed. One overseas carrier has attempted to do this
with additional off line pilots, but the results have been mixed, at
best.

Let\'s look at the principle limitations that impact today\'s airline
safety investigator.

**I. Limitations on Data Acquisition**

Today\'s investigator must depend primarily on voluntary inputs from the
individuals who are involved in an incident or accident. This poses a
series of problems:

1/ Some \"incidents\" may be so minor that they are not even perceived
as such by the crew. Examples may be inappropriate touchdown point on
the runway or improper rotation rates on takeoff.

2/ When an incident is of a \"minor\" nature and only the participants
are aware of it, there is no incentive for them to come forward and
report it. Examples might be minor altitude or navigation deviations or
mis-programming an FMS. In addition to the natural desire to avoid
exposure to a possible disciplinary action, there is also a reluctance
to inform on a fellow crew member.

3/ When a flight crew is involved in a major incident or even a minor
accident the fear of punitive action will inevitably deter a candid
discussion of the events leading up to it.

**II. Limits on the Investigator.**

1/ In addition to the lack of training noted in the opening paragraph
and the lack of data outlined in section I above, the ordinary airline
safety investigator suffers from a lack of time. Most airline safety
departments are very small, and effective human factors investigation
tends to be very time consuming. Consequently, the press of more obvious
problems tends to preclude any effective human factors work on a routine
basis.

2/ Most human factors specialists/researchers lack the operational
background required to assess many of the more subtle nuances of the
interface between cockpit crews and:

\- Flight Attendants

\- Mechanics

\- Station Personnel

\- Air Traffic Control

\- Airline Policy/Procedures development

The upshot of (1) and (2) is that it normally takes a major accident to
bring together the combined expertise of the air safety investigator and
the human factors specialists and the lessons from precursor incidents
escape the system.

**The Real World of Today\'s Airline Safety Investigator**

Given the constraints outlined above, I shall illustrate how one
airline\'s safety division is functioning today.

At Northwest we have two air safety investigators that deal with 5500
pilots operating approximately 1600 flights a day to more than 150
airports over a route network spanning 2/3 of the globe. Our primary
source of incident information is the pilot submitted Air Safety Report
(ASR). Although NWA offers immunity from company discipline for any
information submitted through the ASR, our inability to protect the
confidentiality of those documents or to provide similar immunity from
FAA certificate action seriously compromises our ability to obtain
meaningful incident information during routine operations.

However, when an incident rises above a certain threshold of
noticeability, we do have a process that allows us to conduct more
meaningful investigations into all aspects of the incident. We call it
our flight crew debrief.

As with the ASR, NWA will provide immunity to the affected crew for all
information divulged during the debrief. The big difference between the
ASR and the debrief is that the latter is a face to face meeting with
the crew (often including the flight attendants) and confidentiality can
be afforded the participants since no document obtainable by the FAA is
produced.

When a debrief is conducted we attempt to uncover every causal factor
leading up to the incident as well as the consequences of the manner in
which it was handled. We use as a basis for the debrief James Reason\'s
model since many incidents also include issues of organizational
structure and its affect on flight crew behavior. The following incident
outlines how the process was applied to an incident where there were
serious injuries and aircraft damage after an unanticipated encounter
with turbulence over the Pacific. The incident was somewhat unique in
that all but one of the flight attendants were Asian nationals based at
our various far eastern crew domiciles.

The flight crew was initially debriefed upon their return to the U. S.
three days following the incident. During that debrief it became
apparent that numerous difficulties had occurred in dealing with the
flight attendants. Consequently, the flight attendants and the captain
were reconvened at a subsequent debrief back in Tokyo the following
week. Together, the two debriefings allowed us to ascertain the
following:

\- There were differences in the training provided to U.S. based flight
attendants and their Asian counterparts which led to misunderstandings
between the lead flight attendant (a U. S. national) and his working
crew.

\- There were cultural issues involving attention to assigned duties vs.
passenger care.

\- Company emphasis on \"Service\" lead to a disregard for observation
and enforcement of the \"Fasten Seat Belt\" sign.

\- Northwest had become lax in teaching and enforcing good discipline
with respect to operation of the \"Fasten Seat Belt\" sign.

\- There was no standard language for communicating turbulence
information between the flight deck and the flight attendants.
Consequently, the flight attendants were left to their own
interpretation of terms such as \"light\", \"moderate\" or \"severe\".

\- There were no appropriate guidelines in place for flight attention
actions when advised of varying levels of turbulence.

\- Existent Northwest procedures for interphone communication (no
requirement for the speaker to identify her/himself) led to ambiguities
and assumptions that exacerbated the difficulty of handling the
situation.

A more complete discussion of these issues is contained in Appendix A,
the analysis section of the incident report prepared by the flight
safety division.

The total time to complete this investigation, including the trip to
Tokyo, was in excess of 75 man hours. Additional time was spent writing
the report and working with other divisions within the company to devise
and implement remedies to the problems discovered.

**A Look To The Future**

**Finding The Means To Improve Human Factors Investigations**

If we are to truly mine the field of human factors incidents we must
necessarily find a way to address the concerns enumerated at the outset
of this paper.

Perhaps nothing has been more difficult and frustrating as the inability
to have ready access to the minor incidents that are often the
precursors to the more serious incidents or accidents.

At the heart of the problem in the U. S. has been the legal and
regulatory climate. It may be an oversimplification, but a
[very]{.underline} human factor is to avoid exposing oneself to any
circumstance that can or will adversely affect either your livelihood or
professional status. Although there are the glimmerings of hope on this
front in the form of the Secretary of Transportation/FAA encouragement
of such programs as digital flight data recorder monitoring (FOQA) and
the experimental ASAP program at American Airlines, we are a long way
from the actual regulatory and legislative foundation required for full
implementation.

Another major hurdle that will need to be overcome here in the U.S.,
given the economic state of most of our air carriers, will be the cost
justification of these programs, especially FOQA. The recent FAA
proposed rule making on additional flight recorder parameters will
remove the cost of updating the recorders from the cost/benefit studies
since it will be mandated. However, FOQA programs also require
additional resources in terms of handling, processing and analyzing the
data. This cost will vary depending on the size of the airline and the
level of automation introduced but, at our airline it is estimated that
the initial cost of hardware and software will be \$250,000 and at least
three analysts (one air safety, one engineering and one information
services) will be required at a recurring annual cost of
\$150,000[+]{.underline} The challenge will be to justify the need for
those resources.

In summary, if we are to make any meaningful strides at improving the
human factors analysis here in the U.S., it will require major changes
in the regulatory climate, more and better trained investigators and a
method for cost justifying the effort. The latter may be the biggest
challenge since, historically, accidents have only been priced
[after]{.underline} the fact. No credit is given for avoiding one.

**APPENDIX A**

\[Excerpt from Flight Safety Division Accident Report\]

**2. ANALYSIS**

**2.1 General**

The flight crew was current and qualified in accordance with all FAA and
NWA regulations. There was no evidence of non standard operation either
in the testimony of the flight crew or in the flight data recorder
readout.

The airplane was certificated, equipped and operated according to
applicable regulations. The airplane was properly loaded and the cg was
within the normal operating range.

**2.2 Meteorological Factors**

The area of the turbulence encounter lies along the northern boundary of
the Intertropical Convergence Zone. This is the area where cooler air
from the Northeast Tradewinds is mixed with the warm moist equatorial
air mass. Weather in this area is characterized by nearly continuous but
mostly scattered to broken areas of thunderstorm activity with
individual cells often attaining heights in excess of 50,000 feet. This
thunderstorm activity is in a continuous life cycle process of building,
maturing and dissipating. However, due to the spacing between and among
the cells these thunderstorms do not present the hazard to enroute
navigation often found associated with closely spaced frontal
thunderstorms.

On the day of this incident the crew was totally reliant on the aircraft
weather radar due to the absence of ground based radar at the Guam
center and real time satellite imagery at the NWA meteorology office in
Tokyo. Based on the flight crew\'s observation of the heavy thunderstorm
near the north end of Saipan during their departure and the issued NWA
TP for scattered thunderstorm activity it is probable that there were
developing as well as dissipating thunderstorms in the area of the
planned flight back to NRT. NWA meteorology believes that the most
likely cause of the turbulence was undetected convective activity at a
lower altitude, probably beneath the aircraft\'s radar coverage owing to
its tilt adjustment. However, absent the record that could be provided
by satellite imagery this cause cannot be definitively proven.

**2.3 Air Traffic Control**

Air traffic control is not considered to be a factor in this incident.

**2.4 Operational Factors**

**2.4.1 Thunderstorm Avoidance Procedure**

The flight was operating in an area of scattered thunderstorm activity.
The activity had been continuous throughout the daytime hours and the
flight crew had operated through the area upon their arrival
approximately 2 1/2 to three hours earlier. During the arrival the
captain had observed that the storms appeared to be \"old and dying\"
with no lightning or convective activity. The aircraft weather radar
showed only wear or no returns in both the \'normal\' and \'contour.
modes. Nevertheless, he advised the flight attendants to stow the
service carts, turned on the \"Fasten Seat Belt\" sign and advised the
inbound passengers to remain seated with their seat belts on.
Subsequently the flight encountered only \"minor bumps\" during its
descent and arrival to SPN.

Based on his experience during the arrival to SPN the captain decided
that it was not necessary to request the flight attendants to remain
seated during the climb out of SPN. Indeed, no turbulence was
encountered during the climb out and initial level off. Nevertheless,
the captain elected to keep the \"fasten Seat Belt\" sign illuminated
until they had exited the thunderstorm area.

The encounter took place about two minutes after the airplane leveled
off in a thin layer of cirrus clouds. According to the flight crew\'s
statements they were following published NWA thunderstorm avoidance
procedures while navigating their way through the area of thunderstorms.
As the airplane was being established in the cruise configuration with
an indicated airspeed of 289 knots, the DFDR indicates that the airspeed
began to fluctuate (+9/-4) knots. Within seconds the airplane began to
experience minor vertical acceleration oscillations (+.2G/-.15G).
Twenty-eight seconds later, the airplane began experiencing significant
vertical acceleration oscillations ranging from +1.83G to -0.56G. During
the significant vertical acceleration oscillations the airplane
experienced pitch attitude oscillations ranging from +4.0 degrees to
-0.8 degrees and roll attitude oscillations ranging from 12.8 right wing
down to 12.7 left wing down. The major oscillations lasted for thirty
seconds with the maximum positive to negative G reversal confined to a 2
second interval. Minor oscillations continued for anther 20 seconds.

At the time of the severe turbulence encounter the captain estimated
that they were ten miles to the upwind side of the nearest cell off the
left side. The first officer, who was flying, estimated the distance as
closer to twenty miles. These distances are in accordance with the
guidance in the FOM (7.31.2 & 3).

The impact of the encounter caused the cockpit to fill with flying
debris, the autopilot and autothrottles to disengage and engine \"Low
Oil Pressure\" lights to flash on.

**2.4.2 Cockpit to Cabin Coordination**

Within one minute after ascertaining that the aircraft was still in
controlled flight the second officer attempted to contact the cabin to
ascertain the situation. He was answered by what he believed to be a
\"male Chinese\" flight attendant but was unable to understand what he
was saying. At the direction of the captain he then proceeded downstairs
to the cabin. The second officer conducted a rapid walk through of the
cabin during which he spoke briefly with several of the flight
attendants, but not all of them. He returned to the flight deck and
reported to the captain that there numerous injuries to passengers and
crew and that one passenger appeared to be unconscious and that their
was debris everywhere in the cabin.

Following some discussion with both the cockpit crew and the lead flight
attendant the captain decided to continue toward NRT. His decision was
based on the following:

\- Uncertainty about the weather in either Saipan or Guam.

\- Uncertainty about the availability of hospital on Saipan.

\- A return would require penetrating the area of thunderstorms which
caused the problem.

\- A minimum time of one hour would be required to clean up the cabin
suitably for a landing.

\- The unconscious passenger had regained consciousness.

\- There did not appear to be any life threatening injuries to either
passengers or crew.

Subsequent to the arrival at NRT it was determined that a number of
misunderstandings had developed relating to the manner in which the
cabin and flight crew had handled their respective roles following the
turbulence encounter. Therefore, a subsequent meeting was arranged in
Tokyo on December 9 between the cabin crew and the Captain to more fully
discuss these issues. The results of that meeting indicate that their
are some fundamental problems that have arisen from the policies and
procedures for staffing flights south of Narita, the so called
\"interport\" flights. These impact both safety and customer service.

**2.4.2.1 Cabin Crew Composition**

Flights beyond Narita are normally staffed with a U.S. lead flight
attendant and the balance of the crew is composed of Asian nationals.
The latter may be based in Bangkok, Hong Kong, Narita, Seoul or
Singapore. Normally there is a mix of the various bases in order to
accommodate the numerous languages of our Pacific rim customers.
However, this cross cultural combination can cause problems with both
intra and inter cabin communications.

It was noted during the debriefing that some U.S. lead flight attendants
exercise very firm control over the interport flight attendants while
others frequently almost defer to them. This leaves the Asian flight
attendant with a lack of standards in their operation. This phenomena
is, in part, attributable to lack of standardized training for the lead
flight attendant position and/or the lack of a permanent bid lead flight
attendant position which frequently facilitates a low experience level
in that job. It can also arise out of the lack of joint training which
precludes the U.S. and Asian flight attendant groups from developing any
mutual understanding.

During the instant flight a misunderstanding arose between the lead
flight attendant and one of the Asian flight attendants over the
expectations concerned with handling the injured passengers and flight
attendants. The Asian flight attendant was expecting some instructions
from the lead as he had been trained. The lead, meanwhile, was expecting
the other flight attendants to immediately deal with the emergency as
[he]{.underline} had been trained.

Because the lead flight attendant became engrossed with directing the
cabin response he was not available at the L1 interphone when the flight
deck made its initial inquiries as to the status in the cabin. An Asian
flight attendant was the first to be contacted and, due to difficulties
with the technical questions, could not reasonably communicate with the
flight deck.

**2.4.2.2 Language**

With the exception of the lead flight attendant, English is normally the
second language of the interport cabin team. Their degree of fluency
varies widely but, in nearly all cases, they are unfamiliar with both
slang and much of the technical jargon associated with flying our
airplanes. Therefore conversation between these flight attendants and
either the lead or the flight deck crew is ripe for misunderstandings.

During the debriefing the point was made by several of the Asian flight
attendants that their role as interpreters was very difficult because
they had to listen for the words of the flight deck crew or the lead
flight attendant, often in the midst of a service or in a noisy part of
the aircraft, try to get the gist of the message and then make the
announcement in their native tongue. Lengthy PA announcements or those
dealing with technical matters such as speed, altitude, or malfunctions
can be nearly impossible to handle. Although not an issue in this
incident, there are certain situations involving unanticipated problems
with only a short time where this could be significant.

**2.4.2.3 Passenger Communication**

This flight was operated with one Japanese speaking flight
attendant/interpreter. Unfortunately, she was one of the injured
parties. This caused a good deal of confusion and misunderstanding on
the part of the cockpit and cabin crews since the entire passenger load
were Japanese nationals. The interpreter was initially unable to make
any announcements in her native language although she later proceeded to
a passenger seat from which she could use a PA handset.

All of the flight attendants were anxious for some information from the
flight crew. The captain wanted very much to provide this information to
the passengers and made several attempts to secure the interpreter\'s
presence on the flight deck believing, as he did, that there was little
point in an English language announcement to a group of passengers who
would not understand him. No one from the cabin team conveyed to him the
message that [any]{.underline} sort of announcement from the cockpit
would help to assuage both passenger and flight attendant concerns over
the turbulence encounter.

During the course of these efforts he spoke several times with one Asian
flight attendant who he assumed was the interpreter. When he finally
ascertained that she was not, he sought her assistance in bringing the
interpreter to the flight deck. When he finally discussed the issue with
the lead flight attendant he was distressed that the captain did not
initially contact him, however, he did not discuss this with the
captain.

This flight had aboard only 117 passengers, virtually none of whom were
seated in the aft (E) zone of the aircraft. However, it is interesting
to note that the two most seriously injured passengers had moved
themselves to that zone. It is also interesting to speculate on what the
situation might have been if this zone had been more fully occupied and
the services of the lone interpreter were lost.

**2.4.2.4 Crew Communication**

As noted in nnnnn and 2.4.2.3 (above) there were several areas of
breakdown in communications between the flight deck and cabin crews. In
discussing this issue with both groups it is clear that many of our
current operating practices militate against clear and effective
communications between the tow groups. Of particular import is the
declining incidence of full crew briefings.

As illustrated in the foregoing, the lack of definitive information
leads to both parties making assumptions about the other\'s actions. It
also can lead to critical [gaps]{.underline} in the information
exchange. A gap that developed in this incident was the flight
attendant\'s concern about the structural integrity of the airplane.
Moreover, the lead flight attendant was concerned with how to deploy his
remaining three able flight attendants for the landing, especially since
he was unsure of the aircraft\'s status. The exact reasons for no one on
either side of the question initiating this discussion are not certain
but, clearly, the lead flight attendant\'s assumption that the captain
had elected not to deal with him earlier must have been a factor.

**2.4.3 Fasten Seat Belt Sign Discipline**

A fundamental question arising from this investigation is, \"why were
all of these people injured when the \'Fasten Seat Belt\' sign was
illuminated?\" A corollary question must also be, \"What does the
\'Fasten Seat Belt\' sign mean to our flight attendants in terms of
service initiation or interruption?\"

The former question would seem to be the easier to answer. The flight
attendant manual states specifically that, \"Flight attendants shall
visually and individually check to ensure that all customers are seated
with their seat belts fastened whenever the seat belt sign is
illuminated and as long as it remains on.\" (Ref. p14.10.4)

However, and especially on long haul flights, it is common for the
flight crew to overlook the \"Fasten Seat Belt\" sign and allow it to
remain on for extended periods of time even in calm air. Consequently,
enforcement of the illuminated instructions has become lax. It can
reasonably be assumed that the cabin crew\'s perception of the lack of
danger during the departure was nearly the same as the captain\'s, both
of which were based on their arrival experience several hours earlier.

The second issue is far more complex and requires some research and
discussion that exceeds the scope of this report. Suffice it to say that
the \"fasten Seat Belt\" sign, of its own accord, offers little guidance
to our flight attendants unless its illumination is accompanied by
specific instructions from the flight deck. The Flight Safety division
and Inflight have undertaken a cooperative effort to reduce the level of
flight attendant turbulence injuries. Although we are only beginning to
get a data base of some significance, one salient point emerging from
the information is that a large number of the injuries occur during
flight attendant attempts to maneuver and/or stow serving carts. This
casts some doubt on the current procedures (or lack thereof) for
protecting the welfare of our flight attendants.

## FLIGHT CREW ACCIDENT AND INCIDENT HUMAN FACTORS - R. Curtis Graeber

Chief, Human Factors Engineering

Boeing Commercial Airplane Group

The following comments are offered in relation to the Workshop\'s goals
from the perspective of a Human Factors professional with both NASA and
airframe manufacturing experience. The focus is on data acquisition,
analysis, and application based on airline flight crews and does not
address other potential activities which may provide equal or greater
benefit for improved safety.

**Objective 1**

Relevant data elements or types of data required for analysis.

The aviation industry has historically encountered considerable
difficulty in attempting to analyze flight crew error in a manner which
demonstrably contributes to safety improvements. There are several
reasons for this state of affairs. These include limited data scope,
incomplete reports, inconsistency in content and format, skewed
reporting, and lack of accurate baseline error rates. For the purposes
of this effort it is worthwhile to explore each of these before offering
a \"solution.\"

The issue of *limited data scope* arises primarily from the fact that
the only data assured of collection are those related to accidents or
major incidents. While even these are often insufficient for sound human
factors analysis, they represent low probability events that usually
arise from the joint influence of multiple contributory factors of which
one or more may legitimately be designated a human factor. They provide
little beneficial insight into those human factors that are responsible
for raising the risk of aviation mishaps on a daily basis.

If anything, this overall situation is the outcome of an aviation system
that holds the flight crew accountable for safely operating the
airplane. In some cases, this practice translates into the crew being
more exposed to blame because they are the last line of defense when
unsafe situations threaten flight operations. Their positive
contribution to safety is rarely recognized except when pilots are seen
as \"superhumans\" overcoming insurmountable odds caused by other
factors (e.g., Aloha 737, United DC-10 at Sioux City).

It is well recognized that this situation also results in the reluctance
of many crew members to be forthcoming after an incident, hence the
success of ASRS\'s immunity provision for encouraging direct input from
crew members. However, such a reporting system necessarily encourages
pilots to report any perceived infraction, thereby providing a *skewed*
estimate of the actual frequency of certain types of events and an
inability to estimate *error rates*. Conversely, many events which would
be of interest to a Human Factors investigator are not reported because
pilots either don\'t see them as potential infractions or don\'t think
they are important enough to bother filling out the paperwork.

Regardless, ASRS reports can provide valuable insights into the whys and
wherefores of well described incidents; however, most of the data is
*incomplete* especially for attempting to carry out a behavioral
analysis of a set of events. Such analyses often require consistent
information about situational factors, organizational issues, and other
behavior modifiers, and not just operational facts which are easier to
gather. When in-depth data is required, it is often obtainable by the
use of the costly CALLBACK process to interview flight crews involved in
a particular type of incident. I raise these issues not because ASRS is
not a valuable tool for understanding flight crew error, but because it
is sometimes viewed as having solved the data problem. It has not.

While more accurate and consistent data is needed on incidents, major
and minor, it is my belief that in order to focus our limited safety
resources effectively, we must not limit our data search to the negative
side of the human factors equation. The industry could obtain valuable
insights from those occasions where flight crews have successfully dealt
with unexpected operational events. Crews have always been willing to
share their successes among comrades. The traditional importance of
\"hanger talk\" in promoting safety is well recognized by flight crew
oriented publications. It is perhaps the quickest, but albeit the most
limited, way to address previously unrecognized latent failures. The
industry needs to raise this proven approach to a new level which
enables wider communication and a greater consistency in information
content than that offered through anecdotes. By obtaining a balanced
view of flight crew human factors we will greatly improve our ability to
determine the critical issues that need to be addressed. It will also
encourage greater participation by the professional pilot community,
provide better insight as to why crews sometimes do the things they do,
and help us develop practical countermeasures to human factors related
issues.

At the same time we must be careful not to limit such data collection to
flight crews. Attending more to flight crew error because it is listed
more frequently as a primary cause of accidents ignores the well
recognized multiple-cause etiology of most accidents and incidents.
Similar efforts on other safety fronts have done little to change the
consistently low baseline accident rate over the past two decades. If we
are to accomplish a reduction in the overall accident rate, Human
Factors must be viewed as a system wide issue and any proposed database
must do more than is currently being done to capture the performance of
other professionals who interact with flight crews in daily operations.

The future success of any expanded effort to gather human factors data
regarding flight crews will require much more open cooperation by their
professional associations and the airlines for which they work. Recently
there have been renewed calls for the sharing of such information among
the carriers. At the IATA Human Factors Seminar in Bahrain in March,
this was one of the major conference conclusions. The IATA Flight
Operations Directorate is exploring how this might be encouraged. The
ICAO Human Factors Program has expressed similar desires. I strongly
recommend that the U.S. FAA not develop a scheme independent of the
international aviation community. Too often in the past the FAA and the
U.S. in general has chosen a \"go-it-alone\" approach. We can no longer
afford to adopt such an isolationist view. Airlines are becoming
multinational with rapidly increasing potential for mixed U.S. and
non-U.S. crews. The presence of foreign registered airplanes is rapidly
increasing within U.S. airspace while U.S. carriers continue to expand
their global operations. If we really want to understanding the human
factors issues surrounding accidents and incidents, we must take much
more of an international perspective which other authorities and
non-U.S. carriers can \"buy into\" and which will enable the sharing of
similarly structured data across boundaries.

**Objective 2**

Incorporating models of human performance and error into the analysis
process.

There are a variety of human performance and human error models that
have become popular among aviation human factors professionals. These
include Jim Reason\'s concept of latent failures and resident pathogens,
Elwyn Edwards\' SHEL model, Jens Rasmussen\'s model of skill-based, rule
based, or knowledge-based behavior, and Don Norman\'s sorting of errors
into slips, lapses, or mistakes. While both Reason\'s and Edwards\'
models can be particularly helpful after an accident to determine
whether or not all the potential areas of concern have been
investigated, they are not particularly helpful except at the most
elementary level in analyzing incident data. Of the two, Edwards\' model
is the most parsimonious and may at least provide some basic categories
for data reduction.

As part of an ATA Human Factors Task Force effort, Boeing has been
developing an analysis of accidents in an attempt to better understand
why crews do not follow procedures. The data utilized is the same as
described in our recent publication on Multiple Prevention Strategies.
The structure of this analysis is based on a cognitive approach and has
led us to include both Rasmussen\'s and Norman\'s schemes for attempting
to categorize particular errors by the crew. At the same time it
includes the following general categories: Cultural Effects, Context
Variables, Stressors, Type of Behavioral Trigger, Situational Awareness,
Information Acquisition and Processing, and Crew Communications. We have
also obtained some preliminary indications that the approach we have
developed may be beneficial in developing a structured way to analyze
air crew related incidents within an airline and hence provide a
framework for structured data collection.

Obviously, our approach is not a model but rather a behaviorally based
practical strategy for organizing data in a way that is compatible with
current cognitive science. Whether any incident database could include
such a comprehensive scheme may be a challenge, but there is no doubt
that, if a human factors database is to prove useful, it must provide
some degree of consistency across reports which will support statistical
and behavioral analysis. Human Factors is a science which requires a
richer database than most activities in aviation. Of course, the plague
of databases is that their usefulness is usually limited by their format
and completeness which can usually withstand very little tampering once
put in place.

**Objective 3**

Conceptual interface.

This is a difficult subject which ought to be approached from a Human
Factors Engineering standpoint, that is, let\'s first define and
validate the interface requirements. My greatest fear is that a great
deal of effort and resources will go into major software development
without careful examination and testing of what is really needed. We
believe that a great deal of data is already available from different
sources in a wide variety of formats and quality. While we need to
improve the quality of these data, the need to improve our ability to
share such data may be even greater.

At Boeing we currently utilize a wide variety of safety data to analyze
the factors which contribute to flight crew error. These sources include
official accident reports, ASRS, CHIRP, and airline incident reports
concerning Boeing airplanes. At times, when a particular design or
training issue arises within the company, we have contracted for
CALLBACK interviews by the ASRS staff in which we help them structure a
carefully designed set of questions targeted at understanding the issue
at hand.

We have periodically utilized these and other data obtained from our
training activities and airline support visits to assess the potential
contribution that specific design options might make to improving safety
or to assure ourselves that design features will not increase the
potential for flight crew errors. We have also used such data to help us
better understand crew factors in the course of accident investigations.
These techniques can prove extremely helpful when sufficient data is
available, but the paucity of reliable data can be extremely
frustrating. While simulator studies can provide important data on human
error potential, they take considerable effort and time and lack the
real-world validity of operational events encountered by line crews.

## HUMAN FACTORS ISSUES ASSOCIATED WITH FLIGHT CREW ACCIDENTS AND INCIDENTS - Steven D. Harper And Mark J. Detroit

**Introduction**

The Crew System ERgonomics Information Analysis Center (CSERIAC) was
established in 1988 as a national source of human factors information.
CSERIAC is a Defense Technical Information Center/Department of Defense
(DTIC/DoD) organization, hosted by the Human Engineering Division at
Wright-Patterson Air Force Base, and operated by the University of
Dayton Research Institute. CSERIAC supports all facets of Human Factors
Engineering and Ergonomics for human operated systems.

CSERIACs mission is to provide a quick and reliable source for
analytical services, topical publications, software programs, and data
bases pertaining to human factors. We collect, analyze, and disseminate
information and technologies to support all parties within the
government, industrial, and academic sectors concerned with
human-machine systems.

Historically, CSERIAC has provided its customers with a broad variety of
services and products. Products range from the *Gateway* newsletter (a
free periodical published six times per year with a readership exceeding
10,000), to State-of-the-Art Reports (SOARs), covering a variety of
areas of high interest to human factors practitioners, to sophisticated
human factors engineering and analysis support for short and long term
projects.

The aviation experience base at CSERIAC is extensive. In addition to our
traditional Air Force customer base, CSERIAC has supported NASA and FAA
with human factors analysis of aviation related projects. CSERIAC has
experienced pilots and staff (ATP rated, Flight Instructors, military)
with accident investigation experience and exposure to state-of-the-art
military and commercial flight control and navigation systems. In
addition, our staff (and the faculty at the University of Dayton) have
extensive operational and research experience with aircraft and
simulators covering general aviation, airline, and military operations.
CSERIAC is also fortunate to have a member of FAA, Dr. Mark Hofmann,
serving on our steering committee.

**Discussion**

[Objective 1: Relevant human factors data]{.underline}

The identification of human factors data relevant to the investigation,
and, more importantly, the prevention of aircraft accidents and
incidents has been an issue since the birth of aviation. However, it
wasnÕt until the late 1970s that the FAA and the NTSB began to formally
recognize the limitations and performance capabilities of human
operators as factors in accidents and incidents. Even now, after nearly
30 years of recognizing this important element, there are few
computer-based tools or models available for human factors practitioners
responsible for the identification of the causes of aircraft accidents.
The development of new databases and access to existing human factors
information is an important step toward providing tools that can
facilitate credible analysis of human performance.

In order to meet the needs of the users of the data, CSERIAC recommends
conducting analysis to identify the information needs of the user
population as well as investigating existing procedures and analysis
methods. All factors relevant to accident and incident data could then
be categorized according to a user\'s specialty (pilot, investigator,
engineer, psychologist, etc.). Potential sources for this data include:
NTSB, FAA, NASA, Military, Airline Pilots Association, Aircraft
Manufacturers, Airlines, Insurance companies, etc. This analysis effort
should also investigate other data sources and strive to uncover missing
data elements. The results of this analysis will provide an
understanding of current requirements for information and technologies
available for accessing and processing it, identify areas of common
interest, and facilitate the development of algorithms to tailor
information searches for accident/incident investigators. It is
important for the designer of any system to have a thorough grasp of the
requirements of the user before attempting to implement a solution. It
is equally important to assess the current methods for analyzing the
data in order to build upon them. CSERIAC has extensive experience in
determining and refining user requirements and analyzing technologies
which can then be synthesized into design solutions. CSERIAC methods
result in a design approach that is based on the needs of the user and
aided by technology, rather than an approach which is solely driven by
technology.

[Objective 2: Human performance and error models]{.underline}

The principles of human information processing theory (Wickens, 1992)
and some of the various mental workload assessment techniques may be
useful in the analysis of aircraft incidents and accidents. Information
processing theory provides a scientific framework for modeling
strategies and methods humans use to process information. Human
Reliability Analysis (HRA) techniques facilitate the probabilistic risk
assessment (PRA) process for predicting when humans are likely to commit
errors. Information from the human factors databases could be used to
determine the inputs and outputs of the human and system. Fault Tree
Analysis (FTA), Event Tree Analysis (ETA), and Root Cause Analysis might
be adapted to model the throughput (or processing) of the human, within
the context of system characteristics and environmental conditions. This
approach would enable various scenarios to be evaluated and yield a
higher confidence in identification of probable human causes of
accidents or incidents.

This approach could be taken further with the current state-of-the-art
of computer technology. Low fidelity (non-motion, part-task) simulations
can be developed which would allow investigators to use the human
factors data in conjunction with or without Subject Matter Experts
(SMEs) (e.g., flight crews) to recreate accident or incident scenarios.
The approach using SMEs would be much like that which is used today to
recreate an accident scenario in a motion based simulator. The approach
without the SMEs would basically use a computer optimized profile based
upon either information processing models or queuing theory and would
provide a capability to conduct some preliminary analyses before calling
in the SMEs.

A side benefit of such a development effort would be the use of tools
such as these to design of new aircraft systems. Designers could get a
better allocation of functions between the human operator and the system
than is currently capable with a static design environment. In addition,
the creation of lessons learned databases (accidents, design,
maintenance, Crew Resource Management, etc.) would also influence future
designs and greatly aid in determining flight crew error contributions
to aviation accidents and incidents.

[Objective 3: Conceptual interface]{.underline}

In order to develop an effective user interface for any system, the
needs, capabilities, and limitations of the user must be fully
understood. CSERIAC has considerable experience with Human-Computer
Interface (HCI) from designing Graphical user Interfaces (GUIs) for
accessing computer databases and to providing usability guidelines for
software tools. CSERIAC\'s experience includes the development of an
extensive internal database of human factors information, database
development guidance and support for the US Air Force, creation of a
CSERIAC \"Home Page\" on the World Wide Web, and extensive experience
with a variety of databases to support our function as an Information
Analysis Center for DTIC.

The need to access information is not much different for
accident/incident investigators than it is for many other scientific
disciplines. Most of the necessary data exists in one form or another.
What is required is an integrated, computer based solution. Users of the
data need the ability to access it from one place. A problem that has
existed in the past is that much of the data is not text based. The text
based data is relatively easy to process and store with electronic
methods. The data that is not text based (photo, video, audio, among
others) has been much more difficult to store in a central database.
Further, data collection procedures in the field do not readily support
traditional databasing methods.

The technology to create an integrated, computer based data collection,
storage and retrieval system is mature. Many of the needed elements are
grossly grouped within multi-media computer-based technology. CSERIAC
has much experience with this new technology. To take advantage of
multi-media technology, the current process for data collection,
dissemination, and usage will have to be analyzed and modeled. This
process often results in a number of changes in the process due to new
capabilities made available by the technologies. One such change might
be the streamlining of field collection procedures using electronic
checklists and data input with Laptop PCÕs. Further changes would
include the storage of photo, video, audio, etc. data on electronic
media. Access could be provided nationwide via the World Wide Web. The
technology to display and work with the data in this manner exists
today. Going a step further, human error prediction models, simulation
tools, etc. could also be made available. Levels of access to the data
would be controlled by an editable user profile that would be filled in
automatically during the initial login phase. The user would provide
background information and be granted access to information and tools at
a level appropriate for his or her needs.

**Conclusion**

CSERIACÕs charter is to provide human factors support and analysis for
projects such as the FAAs initiative to provide accident and incident
data to aviation professionals. CSERIAC is staffed with human factors
professionals with experience in aviation operations, training, and
accident investigation as well as in the access and display of computer
based information. CSERIAC is capable of tailoring solutions to meet the
unique human factors problems of our customers. In addition, CSERIAC can
create a multidisciplinary team of professionals from government,
industry, and academia to provide solutions based upon systems
engineering principles.

We believe that the intent to provide easy access to human factors data
relevant to accidents and incidents will greatly aid the analysis of
aircraft accidents. Further, such a database, coupled with human
performance models, will facilitate a better, more expedient
understanding of the role of the human operator in an aircraft accident
or incident. Finally, this initiative may be eventually provide cockpit
design personnel with information that will enable the design of
cockpits that reduce the probability of accidents and incidents. CSERIAC
would be pleased to work with FAA to further develop this initiative and
help to integrate it with the national plan for Civil Aviation Human
Factors.

**References**

Harris, W., Colson, F. J., and Donahue, G. L. (1995). [National Plan for
Civil Aviation Human Factors: An Initiative for Research and
Application]{.underline}. Federal Aviation Administration: Washington,
D.C.

Wickens, C. D. (1992). [Engineering Psychology and Human Performance 2nd
Edition]{.underline}. HarperCollins Publishers, Inc.: New York.

## HUMAN FACTORS CONSIDERATIONS TO BE USED BY CERTIFICATION TEST PILOTS IN THE FAA APPROVAL OF CERTAIN ELECTRONIC SYSTEMS IN VARIOUS AIRCRAFT - George Kaseote and George Lyddane

FAA Test Pilots

The Aircraft Certification Service approves many type of Flight
Management Systems (FMS), Navigation Systems, Radios, etc. at the
request of an applicant for a Type Certificate or a Supplemental Type
Certificate for various aircraft. The tests for these systems are
conducted by twelve different offices and over forty different
individual test pilots; however, there is very little guidance or
standardization relative to human factors issued for those individuals
to approve a particular item

The Type Inspection Authorization (TIA), is a document that outlines
various tests (among other things) that the test pilot should conduct in
order to approve an installation of a piece of electronic equipment. The
TIA generally has a \"boiler-plate\" item that loosely says \"Evaluate
the controllability and accessibility of the (FMS)\". That leaves a
large area for individual interpretation. For example, one specific
certified FMS requires 164 key punches to program one IFR approach. Some
pilots may feel that is too many key strokes and requires much too much
time. That time spent programming a system is time spent away from other
activities involved with the overall conduct and safety of the flight.

There should be studies and data available from the Human Factors
community that could be used to issue guidance for standardization
throughout the Certification Service as to a reasonable number of key
strokes required for a function, the amount of time that is reasonable
for various functions, the shape of knobs and dials, etc.

Another consideration that requires study is the annunciations required
to certificate a particular item.

These are some of the things that we would like addressed in this
workshop.

## HUMAN FACTORS IN AVIATION SAFETY DATA AND ACCIDENT INVESTIGATION - Thomas Lilli

Consultant to ALPA Human Performance Committee

On January 9th and 10th the FAA hosted an unprecedented working session
in Washington DC to address safety in the aviation industry. This Safety
Summit was comprised of six workshops one of which concentrated on
safety data collection and use. The safety data which the Workshop
addressed includes: The Safety Partnership Program data, FOQA Program
data, and Safety Data from different domestic and international data
bases. This particular workshop also addressed issues such as improving
the quality and availability of safety data, addressed ASRS\'s future
needs and addressed concerns associated with the sharing of data.

The concept in which the FAA, a commercial airline company, and a pilot
association could form a partnership to improve safety is unprecedented.
Many carriers in the US and abroad are beginning to form alliances like
these. Data collected as a result of these alliances should prove to be
a keystone to future safety enhancements and offer the opportunity to
provide immediate response to safety issues. This is due to the high
level of participation from interested groups. FOQA ( Flight Operational
Quality Assurance) should prove to add further to the benefits gained
through a program such as The Safety Partnership Program due to even
higher levels of participation in pilot labor organizations and the
amount of data being collected. The topic of FOQA can not be broached
without raising two ongoing concerns. They include anonymity and relief
from the Freedom of Information Act. The consequences resulting from
abuse of enforcement policies, and public reaction to news media
interpretation of this information could have a catastrophic effect on
this program and an even greater effect on the morale of the
participants in industry.

Historically, aircraft accidents and incidents have provided the
aviation industry with much of its safety data. Unfortunately, up until
the recent past, the identification of a specific pilot error was the
main protocol for most accident investigations. The introduction of
highly educated human factors experts taking part in accident
investigation and applying sophisticated human factors models to the
accident investigation process helped redirect probable cause
statements. Previous protocols resulted in determining *who* committed
*the* error which caused to *why* did a *chain of errors* occur which
may have ultimately lead to an accident. When we view accident
investigation in this fashion, we can begin to ask what can we do to
prevent future accidents of this nature? We must continue to support
policies that ensure the continued refinement of human factors models
and the application of their results in the accident investigation
process.

One does not have to wait for an accident, however, to see trends of
errors which could ultimately cause an accident. There is a vast
resource of aviation safety data domestically and around the world which
may have valuable information to provide evidence of increased risk and
ultimately tear the safety net. The governers of these data sources
should be encouraged to participate in a global coalition such as an
international ASRS in which a \"Single Event Issue\"[^1] could be
distributed in a high priority. An electronic network type media would
be most efficient. Along with the formation of this alliance, there
should be strict standards put in place for human factors type input.
This would help give credibility to data and provide a more accurate
assessment of specific human factors type systemic problems.

For many years, the goal of increasing safety by reducing accidents
through individual efforts has resulted in the leveling in the reduction
of accidents. For the first time the Civil Aviation Community is uniting
to decrease the number of accidents to zero through unique and
synergistic efforts. The success in the application of safety data to
help in this process can be effective only with serious commitment to
assure that there is proper protection from enforcement, and legislation
to protect data from the Freedom of Information Act. This data must
further be made available on an international basis for Single Event
Issues, and there must be a convention for collating human factors
issues when compiling human factors data. Attention paid to these issues
now will help make the quest for zero accidents a reality tomorrow.

## FLIGHT CREW ACCIDENT AND INCIDENT HUMAN FACTORS AND ANALYSIS - Ronald C. McGarry

Aviation Safety Inspector

**Subject**

Partnership for Safety Programs

**Background**

One of the most challenging responsibilities for the FAA and the air
transportation industry is to ensure that airlines operate safely and in
compliance with the Federal Aviation Regulations (FARS). The FAA, other
government agencies, trade organizations, manufacturers, airlines and
labor organizations alike are constantly looking for new and innovative
ways to make the aircraft operating environment safer. Within the past
few years a safety strategy has been developed primarily by the airline
industry in conjunction with their labor union(s) that encourages airmen
to report incidents they observe that are of a safety concern. While
that in itself is not new, what is new is these programs are in
partnership with the FAA and do not carry the threat of reprisal by the
company or legal enforcement action by the FAA provided certain
conditions are met. These programs are referred to as \"Partnership for
Safety Programs\"

Compliance with the FARS is based on voluntary actions by the airlines
and their employees. In addition, the FAA conducts inspections and
testing on a daily basis on the airline aircraft, safety equipment and
their safety personnel such as pilots, mechanics and flight attendants.
Non compliance with the FARS detected by an FAA inspector requires an
investigation and corrective action. The degree of corrective action is
dependent upon how adversely safety was affected and whether or not the
violation was intentional or inadvertent. The sanction for an airline or
airman that violates an FAR can range from civil penalties (fines),
license and certificate suspension or revocation, to administrative
action in the form of a warning notice or letter of correction. These
safety tools used by the FAA have been effective in gaining compliance
however, they do create an unwillingness (fear) for the airmen to come
forward voluntarily and report noncompliance not detected by the FAA.
This causes a gap in safety data. In other words a number of safety
issues occur without safety analysis or preventative action.

**Issue**

The theory behind partnership for safety programs is that airmen who
operate in the air transportation system on a daily basis would be more
likely to bring safety concerns forward for analysis and correction if
they were assured that there would not be reprisals taken against them
from their employer or FAA.

Since these programs are relatively new there are only a few in
operation today. Some programs are very focused on a particular safety
area such as the reporting of altitude deviations and others are more
general in scope such as the reporting of any safety of flight item that
may be of concern to a flight crew member. These programs have been
brought about through a Memorandum of Understanding (MOU) between the
FAA, an airline and its labor organization(s). The MOU is of a limited
duration and may be renewed if the program is achieving its safety
objectives.

An example of how the most complex program in existence today works is
as follows:

A FAA Flight Standards Certificate Holding District Office (CHDO)
engaged in a partnership for safety program with a major airline and its
pilot union to develop a system where the pilots could voluntarily
report any safety issues they believed were important to a review
committee. The review committee is made up of safety experts comprised
of a member from the FAA CHDO assigned oversight responsibilities for
that airline, airline management and the pilot union. Certain parameters
were established by the review committee to screen out reports that were
not related to safety and obvious, serious, safety problems that would
be detected by the FAA through normal procedures. In addition, any
intentional infraction of the FARS is not be covered by the program. The
committee would review the report and make recommendations for
corrective action to the airline, the union and/or the FAA. However, the
FAA has reserved the right to make its own decisions in the enforcement
arena. The corrective action could range from remedial training for a
pilot to the airline changing a procedure or way of doing business that
precipitated the safety problem. The committee could also recommend
enforcement action by the FAA. All flight crew members are given
feedback on the outcome of each report submitted to the committee
through an electronic bulletin board. The partnership program does not
replace any existing methods the FAA uses to ensure compliance with the
FARS and there are no provisions for immunity from FAA enforcement
action. The program is an additional safety tool for the FAA, the
airline and its pilots to use. Mitigation is given by the FAA to the
airmen that report safety concerns under this program. Most cases
handled by the FAA under this program have resulted in no action or
administrative action reports.

This program is unique because the reports made by the flight crew
members must be made within 24 hr\'s of the safety event. Each report is
reviewed and safety recommendations are made and in many cases
implemented. A number of the reports involve safety issues that are not
infractions of the FARS.

This particular program has been in existence for eleven months and over
2600 reports have been made. They range over a broad base of safety
issues including incidents that would not have been discovered and
corrected by airline management or the FAA in the past. The reporting
and analysis of these incidents have caused several positive safety
changes within the airline.

The traveling public as well as the employees and management of the
airlines have benefited from the safety changes that have occurred
because of this program. The FAA has gained additional valuable insight
into the day to day problems that occur during actual operations that
had not surfaced previously. This type of partnership for safety program
is being studied by other airlines and the FAA to see if it could
provide the same benefits on a larger scale.

**Recommendations**

**1**. It is recommended that partnership for safety programs be studied
more closely by a group of safety experts to see what safety benefits
are being achieved and how these programs can be improved.

**2**. Have the FAA prepare standardized guidance material for FAA and
industry to use in developing and administering these programs.

**3**. Develop an industry clearing house where safety information and
solutions to problems reported through partnership for safety programs
at different airlines can be shared with other airlines.

## ISSUES RELATED TO USE OF ASRS INCIDENT DATA WITH HUMAN PERFORMANCE AND ERROR MODELS - Dr. Rowena Morrison[^2]

**An Overview of the Aviation Safety Reporting System**

The Aviation Safety Reporting System (ASRS) was established in 1975
under a Memorandum of Agreement between the Federal Aviation
Administration (FAA) and the National Aeronautics and Space
Administration (NASA). ASRSÕs primary mission is to collect, analyze,
and respond to voluntarily submitted aviation safety incident reports in
order to lessen the likelihood of aviation accidents. ASRS data are used
to identify deficiencies and discrepancies in the National Aviation
System (NAS) so these can be remedied by appropriate authorities; to
support policy formulation and enhancements to the NAS; and to
strengthen the foundation of aviation human factors safety research.

Pilots, air traffic controllers, flight attendants, mechanics, ground
personnel, and others involved in aviation operations submit reports to
the ASRS when they are involved in, or observe, an incident or situation
in which aviation safety was compromised. All submissions are voluntary.
Incident reports are read and analyzed by ASRSÕs staff of aviation
safety analysts, whose experience covers the full spectrum of aviation
activity: air carrier, military, and general aviation operations; and
air traffic control in towers, TRACONS, Centers, and military
facilities. The analysts perform two primary functions. They identify
the aviation safety hazards discussed in reports and flag this
information for alerting message action; and they classify reports and
diagnose the causes underlying each reported event. Their observations,
and information from the original de-identified report, are then entered
into the ASRS database.

The ASRS database provides a foundation for specific products and
subsequent research addressing a variety of safety issues. Full-form
database records include the narratives submitted by reporters after
these have sanitized for identifying details. The narratives provide a
rich source of information for policy development and human factors
research. The database also contains coded information from the original
report which is used for data retrieval and statistical analyses.

**Background on ASRS Human Factors Taxonomy**

The ASRS had a formal taxonomy for coding human performance prior to
1986, but coding categories were too generalized (e.g., Òpilot
technique,Ó Òcontroller vigilance,Ó) for many research applications. In
1986 the formal coding of human factors diagnostics ceased as a matter
of practical necessity. This change came about as the result of several
years of sharply increased report intake, coupled with a stagnant
funding situation. ASRS no longer had the resources to process incident
reports as it had previously. Under new procedures adopted to expedite
the coding process, the amount of analyst time devoted to coding each
report was slashed from an average of 1 hour and 28 minutes per report,
to approximately 45 minutes of analyst attention. Differential report
processing was also introduced, and coding structures were streamlined.
As the result of these measures, it became easier to reliably code the
external descriptive aspects of events than their underlying causal
dynamics. It was determined that human factors coding would not be done
at all.

The decision made in 1986 still seems to be sound given the constraints
that existed then. However, the absence of hard-coded human factors
information has hampered ASRS research and decreased the utility of the
ASRS database. A significant minority of search requestsÐboth external
and internalÐseek reports that relate to human factors issues. It is
difficult to extract these from the ASRS database as it is now
configuredÐmuch less perform meaningful statistical profiles of human
factors issues in the database.

The ASRS is in the process of revamping and computerizing its report
processing procedures. Computer support is being inserted at new points
in the production process to increase efficiency and enhance quality. If
computers can be used to facilitate human factors coding, ASRS can
reintroduce hard-coded human factors diagnostics to the database. The
ASRS and research community would benefit greatly.

**Challenges in Developing a Human Factors Taxonomy for ASRS Data**

***Applying Human Error Models.*** Human error models provide taxonomic
systems based upon types of error. In the attempt to make these
taxonomies applicable to a broad range of situations (aviation, nuclear,
assembly line), they must be translated so as to apply to the area of
interest. Even if an appropriate human factors taxonomy is developed for
ASRS incident data, it would be an unrealistic expansion of ASRSÕs
mission to analyze the types of errors made in each of the more than
30,000 incident reports submitted annually. This undertaking would also
be intensely resource consumptive, since ASRS analysts currently lack
the human factors background and training necessary to make these
determinations. ASRSÕs primary role, historically, has been to provide
data as efficiently as possible. While development of a human factors
taxonomy is consistent with this role, detailed error analysis of ASRS
data belongs more appropriately to the research and user community.

***Applying Human Performance Models.*** The attempt to integrate human
performance models with incident/accident databases has a long and
difficult history. There are inherent problems in basing any database on
just one, or even a few, such models. For example, in considering just
the workload domain, one would have to choose among a variety of
performance models varying along such dimensions as single versus
multiple, and structural versus resource features. How would the model
most appropriate for ASRS data be determined? ASRSÕs main responsibility
is to provide the best possible data in the most accessible form
possibleÐhence the need for a useful human factors taxonomy. It is the
responsibility of experts in the many different areas of human
performance to apply ASRS data to models of interest.

***Selecting Terms and Creating Definitions.*** One of the most critical
issues for any proposed ASRS taxonomy is the level of aggregation at
which terms are selected. There is an obvious tradeoff between breadth
and quantity. To limit the number of human factors diagnostic terms, it
is tempting to use broad categories (e.g., Òsituation awareness,Ó
ÒworkloadÓ). However, broad categories can become meaningless in some
practical applications. For example, it may be impossible to judge, on
the basis of information supplied in an ASRS incident report, whether a
specific event reflects a breakdown in situational awareness, or a
failure to apply good decision making skills.

An effective ASRS taxonomy should employ terminology that can be used by
both ASRS analysts and human factors specialists. To achieve this
optimal level of aggregation, it is necessary to provide *operational
definitions* of traditional human factors terms. For example,
Òsituational awarenessÓ is a broad term susceptible to conflicting
expert interpretations. Rather than using such a broad term, an attempt
might be made to determine the flight deck activities that are performed
to maintain or achieve situational awarenessÐfor example, mode control
panel settings. Such operational translations would bring human factors
issues to a level that ASRS analysts could use accurately and
consistently. Parallel efforts are currently underway to translate Crew
Resource Management (CRM) terms in this fashion for the Advanced
Qualification Program (AQP).[^3]

Regardless of the level of aggregation employed in an ASRS taxonomy, the
research and user community must agree on standard definitions for the
terminology employed. Without such standardization, it will be
impossible for users to select salient variables for human factors
investigation.

**Proposed Steps Toward Developing an ASRS Human Factors Taxonomy**

Several problems must be solved if a human factors taxonomy is to be
tailored to ASRS needs: (1) a taxonomy must be defined that is
meaningful to the human factors community; (2) report processing tools
and techniques must be developed that will enable ASRS analysts to
rapidly code reports for human factors information, at an acceptable
level of quality. *No matter what method is devised, its application by
ASRS analysts must be formally tested to ensure that the taxonomy is a
practical tool, and that quality goals are actually being met by
analysts during routine report processing.*

***Adopt a Structured Research Approach.*** Development of an ASRS human
factors taxonomy should consist of the following tasks[^4]:

**Task 1** Review existing human factors taxonomies and lexicons,
including those developed by ICAO, NTSB, British Airways[^5], and other
organizations.

**Task 2** Synthesize and enhance pre-existing taxonomies to create a
new ASRS human factors taxonomy for database coding.

**Task 3** Run tests to determine whether ASRS analysts can code reports
at acceptable levels of speed and quality using the new taxonomy.

**Task 4** If the coding tests fail, develop alternate approaches to
accomplish the second research objective.

***Develop Computerized Tools.*** The ASRS reporting form currently asks
reporters to provide information on the chain of events (how the problem
arose; how it was discovered; contributing factors; corrective actions)
and human performance considerations (perceptions, judgments, decisions;
actions or inactions; factors affecting the quality of human
performance). Reporters do not always, or even usually, give equal
attention to these human performance topics. Additional structure
imposed on narrative descriptions might help some reporters provide
information of value to the human factors researcher. If ASRS report
submission *via* the Internet becomes a reality, a simple expert system
with guided queries could support the collection of human factors
information. Such a system could be set up (in theory) as a BBS or web
page to which reporters could connect.

As various elements of report processing become computerized, coding of
human factors information in ASRS reports may be accomplished by use of
an electronic thesaurus of search terms. This feature would be
particularly useful with a taxonomy that incorporates operational
ÒtranslationsÓ of human factors concepts (i.e., both top-level and
lower-level terms). An electronic thesaurus would allow ASRS analysts to
insert one or more words and access a set of matching candidate terms. A
specific term could then be selected from the candidate set.

**References**

Mangold, S. M. and D. Neumeister. 1995. CRM and the Model AQP: A
Preview. In *Proceedings of the Eighth International Symposium on
Aviation Psychology*. Columbus, Ohio: The Ohio State University Press
(in press).

Mangold, S.M. Fax communication to R. Morrison, 16 May 1995.

Rosenthal, L.J. 1993. The Practical Use of Approximate Data When Coding
ASRS Reports. Paper presented at the International Data Exchange on
Aviation Safety (IDEAS) Conference, 27-29 September, Stockholm, Sweden.

Rosenthal, L.J., L.J. Connell, B.H. Kantowitz, and S.M. Frank. 1993.
Human Factors Taxonomy: A Research Proposal. The Aviation Safety
Reporting System.

Wilson, P. Fax communication to R. Morrison, 25 May 1993.

## NEEDED: A SYSTEMATIC APPROACH FOR A COCKPIT AUTOMATION PHILOSOPHY - Hop Potter

Air Carrier Training Branch

Flight Standards Service, FAA

**Background**

At just about any meeting involving aviation human factors, cockpit
automation is one of the hottest topics. It is a hot topic for a number
of reasons. Perhaps the most important single reason is that technology
is advancing too fast for us humans to assimilate its new products. This
assimilation problem is even greater in respect to the FAA\'s ability to
make appropriate regulatory policy.

There is an uneasy feeling among many in the aviation world that
automation has a dark side. Anecdotal evidence points insistently at the
automation element, in hangar-flying conversations about close calls and
in safety articles and newsletters. Accident reports issued by the
National Transportation Safety Board (NTSB) are also pointing at
automation.

**The Cockpit Automation Philosophy Gap**

Still, the FAA has been unable to take a clear position in respect to
cockpit automation. And there are many voices on this complex topic.
What is desperately needed is a unifying cockpit automation philosophy,
one that is universally accepted and faithfully applied in every
activity of every aviation stakeholder. Those stakeholders include
aircraft designers and manufacturers, aircraft customers, pilots, the
travelling public, and the FAA.

**One Example Of The Problem**

Consider just one unresolved issue concerning cockpit automation,
selecting an appropriate level of cockpit automation near a terminal
area:

¥ Keystroke (input) errors generally run about 1 in 10 (10%) regardless
of age, proficiency, and most other variables.

¥ One variable does affect keystroke error rate: WORKLOAD. Keystroke
errors increase as workload increases.

¥ Workload is linked to confusion. If an FMS input operation is
interrupted, confusion results, and crews typically START OVER AGAIN
with the entire FMS input routine.

¥ Seems to say that use of the FMS for flight guidance is not desirable
when workload is already heavy \-- heads-down, more keystroke errors.

¥ Workload is heaviest in the terminal area. Seems to say that FMS use
is not desirable in the

terminal area.

¥ A further reasonable inference seems to be the reverse of bullet #2:
If keystroke errors start to pile up, it is probably caused by workload
and confusion.

¥ An effective \"human-centered\" automation system would recognize when
erroneous keystrokes are starting to signal that the human is getting
overloaded (confused) and would fail \-- drop off \-- to a low- or
zero-automation state.

¥ At present, automation systems fail into a kind of passive-aggressive
state where they simply fail to accept appropriate input until the pilot
unscrambles the egg. Meanwhile, clearances are busted \-- altitudes,
airspeeds, intercepts, holding clearances, etc.

¥ The overriding message seems to be: THE MORE THE CONFUSION, THE LESS
SHOULD BE THE AUTOMATION IN USE BY THE PILOT.

¥ IF THE PILOT CAN\'T RECOGNIZE THE MESS THAT\'S DEVELOPING (and drop
back to a lower level of automation by choice), THE SYSTEM SHOULD (and
drop back to a lower level of automation on its own).

The line of reasoning flows. Anecdotal evidence seems to support the
line of reasoning. Safety wags (writers) and safety experts (NTSB) are
starting to point to automation\'s dark side. But there the remedial
process bogs down.

**The Missing Piece**

What is missing is a way to get to an overarching cockpit automation
philosophy that is based not on inference but on hard data \-- a
philosophy that is simple and irrefutable. The data gathering process
needs to be broad enough to comprise human errors associated with
specific features of cockpit automation hardware and software, and with
the integration of automation with other cockpit processes.

Once appropriate data are collected, analysis should follow which
culminates in practical principles regarding cockpit automation,
principles for every aviation stakeholder. Those principles may then be
packaged into a simple, usable cockpit automation philosophy statement
that will show the way into the next century.

## BASELINE ASSESSMENT OF THE NATIONAL AIRSPACE SYSTEM: AN APPROACH TOWARD MODELING OPERATIONAL ERRORS - Mark D. Rodgers and Carol A. Manning

Federal Aviation Administration\'s Civil Aeromedical Institute

Oklahoma City, OK

**Introduction**

In the history of the Federal Aviation Administration, no aircraft have
collided while under positive control in En Route airspace. However,
aircraft have violated prescribed separation minima and approached in
close proximity. This event can occur as a result of either a pilot
deviation or an air traffic control (ATC) operational error (OE). In Air
Route Traffic Control Centers (ARTCCs), also called En Route facilities,
Air Traffic Control Specialists (ATCSs) primarily handle aircraft
traveling between the terminal facilities across the nation. A system
(SATORI: Systematic Air Traffic Operations Research Initiative) was
recently developed that utilizes routinely collected radar and computer
data from these facilities to re-create the air traffic situation,
allowing for the graphic representation of situational dynamics (Rodgers
& Duke, 1993). This paper discusses a method for utilizing SATORI input
data with the POWER (Performance and Objective Workload Evaluation
Research) system (Rodgers, Manning, & Kerr, 1994) to investigate the
relationship between OE occurrence and measures of sector dynamics,
aircraft proximity, and sector activity.

An ATC OE can occur as a result of either an equipment malfunction or an
error on the part of an ATCS. ATCSs are required to maintain certain
separation minima between aircraft under their control. Standards for
separation minima are described in the Air Traffic Control Handbook
(7110.65H), and associated supplemental instructions. While there is
considerable complexity in those standards, at flight levels between
29,000 and 45,000 feet, ATCSs at En Route facilities are required to
maintain either 2,000 feet vertical separation or 5 miles horizontal
separation between aircraft. At flight levels below 29,000 feet with
aircraft under Instrument Flight Rules (IFR) conditions, En Route ATCSs
are required to maintain either 1,000 feet vertical separation or 5
miles horizontal separation. An ATCS OE takes place when a controller
allows less than these prescribed minimum separation distances between
aircraft (or between an aircraft and an obstruction).

In order for Quality Assurance (QA) specialists who investigate OEs to
make more definitive determinations of the factors involved in OEs, they
require an accurate representation of the dynamics associated with a
given error situation (i.e., using SATORI), and access to objective
(machine-based) measures of the ATC situation. Within the context of the
current investigation/reporting procedure, it is not possible to
determine the extent to which OEs are a result of traffic density,
airspace configuration, local operating procedures, workstation design,
poor technique, or any of a number of other possible factors. In fact,
since normative information is not available for many relevant
variables, as mentioned above, one must settle for a descriptive view of
the factors associated with the error process. However, the POWER system
was developed to allow for the analysis of routinely collected radar and
computer data. The POWER system objectively measures factors that may be
related to the incidence of operational errors. POWER computes several
sets of numerical measures, including airspace characteristics, aircraft
dynamics, aircraft proximity, and sector activity.

POWER allows for the calculation of facility level normative data, and
several measures currently made subjectively by QA specialists. The
addition of objectively computed quantitative (i.e., number of aircraft
worked, extent of separation loss) and qualitative (i.e., workload,
complexity) measures to the current error investigation process may
prove invaluable in determining causal relationships associated with the
occurrence of OEs. This study addresses the impact of aircraft dynamics,
aircraft proximity, and sector activity on situations in which an OE
occurred. It is hypothesized that indices of aircraft dynamics, aircraft
proximity, and sector activity vary systematically during the time
periods before, during, and after an OE. In this study the period
immediately preceding the occurrence of an OE is compared to the time
period during which the error occurred. With POWER it should be possible
to model the error process in ATC. OE modeling depends on the ability to
develop measures sufficiently sensitive to distinguish between periods
when OEs occur and when they do not. However, in this study, there were
too few observations and too many variables to achieve this result.
Therefore, these analyses were considered exploratory and thus were used
to provide general indications about variables of interest that might be
explored further.

**Method**

[Routinely Collected ATC Data]{.underline}. En Route facilities record
all data associated with air traffic control operations on a System
Analysis Report (SAR) tape. Several data types on a SAR tape can be
extracted on the ATC Host Computer System (HCS) using the National Track
Analysis Program (NTAP) and the Data Analysis and Reduction Program
(DART). Three DART files were extracted for this analysis: Log, Track,
and Conflict Alert. One NTAP file was extracted: Beacon. The Log file
contains information on all ATCS interactions with the HCS and
information displayed in the aircraft data blocks (i.e., altitude,
speed, and handoff messages). The Track file contains information about
the number and type of aircraft tracks (i.e., free, flat, coast, block,
hold, unknown) in each sector. The Conflict Alert file contains
information regarding the conflict alert notices (notifications of
potential conflicts between pairs of aircraft occurring within one\'s
sector of control) sent to each radar display in the facility. The
Beacon file contains information about aircraft location within the
facility\'s airspace.

[Apparatus]{.underline}. The calculation of dependent measures was
performed by the POWER software routines which were written to analyze
the extracted DART and NTAP data used by the SATORI system. The POWER
workstation is a DEC 3000-600 Alpha using the DEC/AXP 6.1 operating
system with 44 gigabytes of striped (RAID Level 5) storage and 128
Megabytes of RAM.

[Design and Procedure]{.underline}. Data were analyzed for 12 OEs that
occurred at Atlanta Air Route Traffic Control Center between 1992-1994.
Each of the 12 errors involved only one sector. Eight of the sectors
were low altitude sectors, and four were high altitude sectors. In six
of the errors, a single controller was working the sector; in five
errors, two controllers worked the sector; and in one error, three
controllers were working the sector. Seventeen minutes of data were
associated with each error. The data for each error were split into two
parts: the first 8 1/2 minutes preceding the error, and the second 8 1/2
minutes during which the error occurred. The error occurred
approximately 2 1/2 minutes after the start of the second time period.

A within-subjects design was used to analyze the data for the two time
periods. MANOVAs were performed on the 2 sets of sector activity
measures, 3 sets of aircraft proximity measures, and 3 sets of aircraft
dynamics measures. The 2 sets of sector activity measures included 4
measures associated with handoff activity (Number of Aircraft Transiting
Sector, Average Sector Transit Time, Number of Handoffs Accepted, and
Latency to Accept Handoffs), and 2 measures associated with HCS
interactions (Number of HCS Inputs, and Number of Input Errors). The 3
sets of aircraft proximity measures included 2 measures of conflict
alert (Number of Conflict Alerts, and Duration of Conflict Alerts), 4
measures of proximity (Average Horizontal Separation, Average Vertical
Separation, Number of Aircraft Pairs Within a Criterion Distance of Each
Other, and the Average Time Aircraft Pairs Spent Within a Criterion
Distance of Each Other), and Number of Aircraft in the Sector by track
type. The 3 sets of aircraft dynamics measures included speed, heading,
and altitude dynamics. Four measures were calculated for each: Average
Change, Number of State Changes (A state is defined as a continuous
change in position. For instance, if an altitude update for an aircraft
increases for 5 updates, then remains stable for 5 updates, and then
decreases for 5 updates, a total of 3 state changes occurred; however,
average change over the time period could be zero.), Number of Aircraft
Changing State, and the Number of Changes Over a Criterion Level. The
criterion values chosen for this analysis were 50 knots for speed
changes, 1000 feet for altitude changes, 15 degrees for heading changes,
and 10 miles and 1000 feet for the proximity criterion.

**Results**

Each set of measures was calculated by POWER for each OE. The resulting
data files were appended into a single data set for each type of
measure. These measures were then examined using MANOVA to evaluate any
change that might be attributable to the within-subjects factor of time
period. Since this was an exploratory study, an alpha level of p \< .1
was selected to protect against the commission of Type II error.

[Sector Activity Measures]{.underline}. Table 1 details the means,
standard deviations, and significance levels of the sector activity
measures associated with handoffs. The Average Sector Transit Time was
significantly longer during the error period, [F]{.underline} (1,11) =
57.86, [p]{.underline} \<.001, as was the Average Latency to Accept
Handoffs, [F]{.underline} (1,11) = 23.74, [p]{.underline} \<.001. The
Number of Handoffs Accepted was significantly less during the error
period, [F]{.underline} (1,11) = 26.95, [p]{.underline} \<.002. The
Number of Aircraft Transiting the Sector was not significantly different
across time periods. Neither of the measures associated with HCS
Interactions (Number of Inputs and Input Errors) was significant.

**Table 1**

**Sector Activity**

+--------------+----+----------+------------+--------------------------+
| *            |    |          |            | **Overall**              |
| *[Handoff]{. |    |          |            | [F]                      |
| underline}** |    |          |            | {.underline}(4,8)=17.16, |
|              |    |          |            | [p]{.underline}\<.001    |
+--------------+----+----------+------------+--------------------------+
| **Average    | t1 | [M]{.un  | [SD]{.     | [F]{.underline}(1,11) =  |
| Sector       |    | derline} | underline} | 57.86, [p]{.underline}   |
| Transit      | t2 | = 263.41 | = 79.30    | \< .001                  |
| Time**       |    |          |            |                          |
|              |    | [M]{.un  | [SD]{.     |                          |
|              |    | derline} | underline} |                          |
|              |    | = 547.27 | = 151.83   |                          |
+--------------+----+----------+------------+--------------------------+
| **Average    | t1 | [M]{.un  | [SD]{.     | [F]{.underline} (1,11) = |
| Latency to   |    | derline} | underline} | 23.74, [p]{.underline}   |
| Accept HO**  | t2 | = 44.57  | = 32.42    | \< .001                  |
|              |    |          |            |                          |
|              |    | [M]{.un  | [SD]{.     |                          |
|              |    | derline} | underline} |                          |
|              |    | = 116.17 | = 65.16    |                          |
+--------------+----+----------+------------+--------------------------+
| **Number AC  | t1 | [M]{.un  | [SD]{.     | F (1,11) = 2.57,         |
| Transiting   |    | derline} | underline} | [p]{.underline} \> .13,  |
| Sector**     | t2 | = 3.58   | = 2.11     | N.S.                     |
|              |    |          |            |                          |
|              |    | [M]{.un  | [SD]{.     |                          |
|              |    | derline} | underline} |                          |
|              |    | = 4.42   | = 2.71     |                          |
+--------------+----+----------+------------+--------------------------+
| **Number of  | t1 | [M]{.un  | [SD]{.     | [F]{.underline}(1,11) =  |
| HO           |    | derline} | underline} | 26.95, [p]{.underline}   |
| Accepted**   | t2 | = 5.00   | = 2.67     | \< .001                  |
|              |    |          |            |                          |
|              |    | [M]{.un  | [SD]{.     |                          |
|              |    | derline} | underline} |                          |
|              |    | = 2.67   | = 1.88     |                          |
+--------------+----+----------+------------+--------------------------+

[Aircraft Proximity Measures]{.underline}. Table 2 details the means,
standard deviations, and significance levels of the aircraft proximity
measures. For the aircraft proximity measures, average vertical
separation was greater for the error period [F]{.underline} (1,11) =
6.51, [p]{.underline} \< .027, as was the Average Time Aircraft came
within a Criterion Distance of Each Other [F]{.underline} (1,11) =
17.24, [p]{.underline} \<.002. Neither of the measures of conflict alert
nor the measures of Aircraft Tracks were significantly different across
time periods.

**Table 2**

**Aircraft Proximity**

+----------------+---+------------+-----------+-----------------------+
| **[Proximity]  |   |            |           | **Overall**           |
| {.underline}** |   |            |           | [F]{.                 |
|                |   |            |           | underline}(4,8)=7.51, |
|                |   |            |           | [p]{.underline} \<    |
|                |   |            |           | .008                  |
+----------------+---+------------+-----------+-----------------------+
| **Average      | t | [M]{.      | [SD]{.u   | [F]{.underline}       |
| Horizontal**   | 1 | underline} | nderline} | (1,11) = 2.74,        |
|                |   | = 4.18     | = 4.41    | [p]{.underline} \>    |
| **Separation   | t |            |           | .12, N.S.             |
| (mi)**         | 2 | [M]{.      | [SD]{.u   |                       |
|                |   | underline} | nderline} |                       |
|                |   | = 6.33     | = 2.31    |                       |
+----------------+---+------------+-----------+-----------------------+
| > **Average    | t | [M]{.      | [SD]{.u   | [F]{.underline}(1,11) |
| > Vertical**   | 1 | underline} | nderline} | = 6.51,               |
| >              |   | = 391.69   | = 427.26  | [p]{.underline} \<    |
| >              | t |            |           | .027                  |
| **Separation** | 2 | [M]{.      | [SD]{.u   |                       |
| >              |   | underline} | nderline} |                       |
| > **(ft        |   | = 701.01   | = 265.67  |                       |
| > x100)**      |   |            |           |                       |
+----------------+---+------------+-----------+-----------------------+
| **Average Time | t | [M]{.      | [SD]{.u   | [F]{.underline}(1,11) |
| AC within 10   | 1 | underline} | nderline} | = 17.24,              |
| Miles & 1000   |   | = 33.88    | = 60.46   | [p]{.underline} \<    |
| Ft. (Sec)**    | t |            |           | .002                  |
|                | 2 | [M]{.      | [SD]{.u   |                       |
|                |   | underline} | nderline} |                       |
|                |   | = 68.99    | = 52.94   |                       |
+----------------+---+------------+-----------+-----------------------+
| **Number of AC | t | [M]{.      | [SD]{.u   | [F]{.underline}(1,11) |
| Pairs within   | 1 | underline} | nderline} | = 3.37,               |
| 10 Miles &     |   | = 2.42     | = 4.32    | [p]{.underline} \<    |
| 1000 Ft.**     | t |            |           | .09                   |
|                | 2 | [M]{.      | [SD]{.u   |                       |
|                |   | underline} | nderline} |                       |
|                |   | = 4.08     | = 3.50    |                       |
+----------------+---+------------+-----------+-----------------------+

[Aircraft Dynamics Measures]{.underline}. None of the overall MANOVAs
for the three sets of dynamics measures (speed, heading, and altitude
changes made by aircraft) indicated differences between time 1 and time
2.

**Discussion**

It should be stated at the outset that this analysis was an exploratory
study that cannot be interpreted as defining the causes of OEs. However,
it was a first attempt using the POWER system to compare situations in
which OEs occur with situations in which they did not occur. While
several measures were significant in the expected direction, many failed
to reach significance. Those that did reach significance suggest that
the controller was busier during the error period than the period prior
to it. For instance, there were longer sector transit times, longer
latencies to accept aircraft, and fewer aircraft accepted during the
error period. In fact, it took nearly twice as long for approximately
the same number of aircraft to transit the sector during the error
period. The question as to whether these characteristics caused the
errors or resulted from the errors can not be ascertained from this
analysis.

Another interesting finding was related to aircraft density. As
expected, during the error period more aircraft pairs were within the
criterion distance (10 miles & 1000 feet) for a significantly longer
duration. However, mean vertical separation between aircraft was also
significantly greater during the error period, and mean horizontal
separation appeared greater during the error period, but the difference
failed to reach statistical significance. That is, although more
aircraft pairs were within the criterion distance from each other for a
longer duration during the error period, in general, those aircraft had
greater vertical and horizontal separation during the error period. This
indicates that aircraft density was temporarily increased during the
error period. It would be interesting to look at the distribution of
aircraft density within a sector for several criterion distances and
track the density transitions to determine their relationship to losses
in separation.

The events associated with this analysis occurred within the same sector
over two closely spaced time periods. The focus of further analyses will
be to look at different time periods, using the same data set to better
understand the temporal sensitivity of these measures. By analyzing the
temporal characteristics of these measures, it is hoped that it will be
possible to more accurately identify the time period that defines the
error. Many of these measures may also be highly correlated; by reducing
the number of measures, namely those less sensitive in identifying error
situations, a more predictive set of measures should become available.

POWER has many potential applications, including performance and
taskload measurement, description of factors that influence complexity
in ATC, evaluation of alternative ATC display designs and
configurations, and evaluation of the productivity of controllers
utilizing new automated systems. In order to model the operational error
process using POWER, further analyses will be required, and larger data
sets and fewer variables will be requisites. Once profiles of selected
variables have been developed and applied to operational data sets it
may be possible to predict the occurrence of an operational error, aside
from the fact that separation between aircraft has been lost. Such
models, if validated and appropriately applied to conflict resolution
algorithms, should eliminate/reduce the occurrence of operational
errors. It is to this end that this research is directed.

**References**

Rodgers, M. D. & Duke, D. A., (1993). SATORI; Situation Assessment
Through Re-creation of Incidents. [The Journal of Air Traffic
Control]{.underline}, October-November.

Rodgers, M. D., Manning, C. A., Kerr, C. S. Demonstration of POWER:
Performance and Objective Workload Evaluation Research. [Proceeding of
the Human Factors and Ergonomics Society 38th Annual
Meeting]{.underline}. Nashville, TN, October 1994.

## THE QUEST FOR AVIATION SAFETYÕS HOLY GRAIL: FINDING UNDERLYING CAUSES OF ACCIDENTS AND INCIDENTS or IF YOU REALLY WANT TO IMPROVE AVIATION SAFETY, YOU MUST FIRST IDENTIFY SYSTEMIC PROBLEMS - Captain Robert L. Sumwalt, III

Working Group for Human Factors in Accident Investigation

Air Line Pilots Association

The goal of the FAAÕs Human Factors Data Project is to identify causes
of human error that contribute to accidents and incidents in order to
develop prevention strategies. The Air Line Pilots Association,
representing 42,000 professional airline pilots flying for 37 airlines,
has long supported efforts to improve flight safety by addressing and
resolving the real and potential threats to safety. Many sources of data
indicate that human error is the largest single cause of aviation
accidents across all types of flight operations. For example, the
*FAA\'s National Plan for Civil Aviation Human Factors: An Initiative
for Research and Application* states that human error has been
identified as a causal factor in 60 to 80% of air carrier, commuter, and
general aviation accidents and incidents (FAA, 1995). Because such a
large percentage of accidents involve human factors, ALPA believes that
proper and thorough investigation of aviation human factors is an
essential element in preventing future accidents.

ALPA is committed to thoroughly investigating all areas of potential
significance in an accident investigation, including human factors.
Recently we formed a Working Group for Human Factors in Accident
Investigation, a committee comprised of experienced airline pilots who
have received advanced training in human factors in accident
investigation techniques. *ICAO Human Factors Digest Number 7* states
that combining an understanding of human factors with an accident or
incident investigation can provide Òa greater opportunity to identify
root causes which may not have been recognized previouslyÓ (ICAO, 1993).

**Why Investigate Accidents and Incidents?**

A personÕs answer to this rhetorical question may depend on his or her
perspective. A lay person, for instance, may simply want general answers
to what caused the accident. For a trail lawyer the goal of an
investigation may be to find fault or assign blame. However, to those
persons truly dedicated to improving aviation safety there is only one
overriding objective of each investigation: identifying and correcting
causal factors to prevent future mishaps.

Although the goal of Òidentifying and correcting causal factors to
prevent future mishapsÓ may sound a bit obvious, the outcome of some
investigations reveal that this objective is not clearly understood.
ICAO says, ÒAccident investigation reports usually depict clearly
**what** happened and **when**, but in too many instances they stop
short of fully explaining **how** and **why** the accidents occurred.Ó
Until the industry consistently addresses the ÒhowÓ and ÒwhyÓ, the
watch-words for accident prevention, efforts will be ÒreactiveÓ rather
than Òproactive.Ó It is ALPAÕs intention to continually improve the
processes that lead to more proactive accident prevention. This
proactive approach includes a systemic review of human factors in
accidents and incidents. This is approach is akin to a person who
discovers a slow leak in one of his car tires. There are essentially two
options to resolve the dilemma: continue indefinitely to fill the empty
tire, or find the nail that caused the leak and correct the problem.
Accident reviews that do not look at systemic issues are simply Òfilling
the tireÓ rather than repairing it.

**Looking Beyond the Obvious**

Indeed, several investigations have concluded that the mishapÕs cause
was some performance failure or error by a Òfront line operatorÓ such as
a pilot, air traffic controller or mechanic. While the cited individuals
may have been the last person(s) to have made an error prior to the
accident, it must be clearly understood that people do not make errors
in a vacuum; they are merely one element in a very complex system. To
really understand what caused a particular mishap the investigation
[must]{.underline} dig beneath the surface - look beyond the obvious -
to determine what systemic factors may have caused or led an
individual(s) to act inappropriately. An investigation that looks
beneath the surface Òhelps identify the underlying deficiencies that
might cause other incidents or another accident to happen,Ó according to
ICAO. Developing a framework for systemic review provides a roadmap (a
process and a focus) for unfamiliar territory. The complexity and unique
context inherent in every accident suggests that each review is
uncharted territory.

A Òsystems approachÓ to accident and incident investigation acknowledges
that *all* elements of the system may have a role to play in the
occurrence. These elements include, but are not limited to, airline
management philosophies and practices, regulatory influence and
corporate culture. Adopting a systems approach helps to identify the
underlying causes through a better understanding of how various
components of the system interacted and integrated to result in a
mishap.

ALPA strongly believes that all accident and incident investigations
must strive to review systemic factors leading to the accidentÕs root
cause(s). Because a systems approach aids in the discovery of underlying
system deficiencies, it allows energies to be clearly focused when
offering recommendations for corrective actions. Without a clear
understanding of the mishapÕs root cause(s), recommendations are likely
to be Òband aidÓ solutions, not true solutions.[^6] For example, two
often cited ÒsolutionsÓ are *increased training* and a call for *added
vigilance*. While these items may be necessary and appropriate in some
cases, oftentimes they compensate for, rather than correct underlying
system deficiencies. Resource management training teaches crews how to
leverage available resources. While a highly resourceful and well rested
crew may not fall prey to a systemic deficiency, another crew may. The
more important question is ÒAre the resources sufficient or appropriate
for the operational requirements?Ó For example, a review of operator
action without a review of available resources and/or situation context
is a deficient process that will likely lead to a band aid solution. It
is necessary to define methodologies that focus investigative efforts
beyond the surface indicators. The development of systemic human factors
investigative methodologies will focus review activities on a deeper
level.

A system as complex as the air transportation system requires real
solutions. While band aid solutions may provide short term fixes, they
are faulty by virtue of their failure to actually correct the real
problems. In order to make real improvement in aviation safety, it is
necessary to first identify systemic problems. Unless the underlying
systemic deficiencies are identified, it is not likely that
corresponding corrective actions and recommendations will prevent future
mishaps. To illustrate, in February 1979 an Nord 262 air carrier crashed
at Clarksburgh, WV. The National Transportation Safety Board (NTSB)
determined that the probable cause of the accident was Òthe captainÕs
decision to takeoff with snow on the aircraftÕs wings and empennage
surfaces \...Ó (NTSB, 1979). By pointing their finger at the crew, what
did the NTSB accomplish? Did their probable cause statement eliminate
takeoff icing-related accidents? Sadly, it should be no surprise that in
the 13 years that succeeded this accident, there were 10 air carrier
accidents worldwide attributed to takeoff attempts with ice on wings.

In March 1992 a Fokker F28 air carrier crashed during attempted takeoff
from New YorkÕs LaGuardia Airport. The focus of this investigation
centered on wing icing, but unlike others, it also looked at the entire
system in which the flightcrew operated. The NTSB then determined that
the probable cause of the accident was Òthe failure of the industry and
the Federal Aviation Administration to provide flightcrews with
procedures, requirements, and criteria compatible with departure delays
on conditions conducive to airframe icing\...Ó (NTSB, 1993). It must be
noted that the NTSB also cited the crewÕs performance, but by citing the
[system]{.underline} as the foremost cause, the catalyst for change was
set. The systemic investigative approach that surrounded this accident
directly influenced widespread improvements to Federal Aviation
Regulations pertaining to takeoff in icing conditions, as well as
massive overhauls to air carrier operating procedures. Had the NTSB
merely pointed the finger of blame at the flightcrew, what corrective
action, if any, could have been gleaned from this investigation? Based
on the fact that 10 air carrier accidents occurred worldwide due to
icing in the previous 13 years, it is suspected that very little safety
improvement would have been realized had the NTSB failed to take a
systemic approach. Remember: in order to improve aviation safety, you
must first identify systemic problems.

A systems approach to investigation must be clearly defined and
methodically used in each and every investigation. While the NTSB may
have used a systems approach in determining probable cause of the 1992
New York icing accident, it failed to use this methodology when
determining the cause of a 1994 windshear accident at Charlotte, NC
(NTSB, 1995). Had the crew been made aware of the severe windshear
conditions that existed that evening, they would have not initiated
their approach for landing. However, a series of systemic failures
prevented the crew from acquiring this information: a sophisticated
terminal Doppler weather radar (TDWR), a weather-alerting system with
proven results, should have been installed in Charlotte, but was plagued
with FAA bureaucratic problems including land acquisition difficulties;
air traffic controllers were required, but failed to provide the crew
with timely weather information; and, the aircraftÕs on-board windshear
detection system had a design flaw that prevented the flightcrew from
receiving a timely windshear alert. In spite of these weather
dissemination deficiencies, the NTSB cited crew errors as accidentÕs
foremost probable cause. It was speculated that upon entering the
windshear, the crew could have successfully flown through the encounter
had the aircraft been flown differently. While that may have been the
case, a systems approach would have revealed that the crew probably
would have never initiated the approach had they been warned of the
serious nature of the weather. In essence, had the crew been properly
forewarned, the accident opportunity would have been eliminated.

The National Center for Atmospheric Research states that there have been
30 air carrier windshear-related accidents from 1964-1989. Following the
Charlotte windshear accident the NTSB had an excellent opportunity to
highlight the need for improvements in weather dissemination practices.
Because they failed in that endeavor, no real change is likely to
significantly reduce windshear-related accidents. As indicated in the
above-cited icing analogy, until investigations focus on the underlying
causal factors that contributed to the mishap, it is difficult to
appropriately focus corrective actions.

**How Do We Get There?**

To conduct a systemic investigation of a mishap, it is necessary to
properly collect and analyze human factors data. Human factors cognitive
models provide a framework for the scope and focus for review efforts.
Two well known models are EdwardÕs SHEL model and ReasonÕs model. Below
is a very brief description of these models. *ICAO Human Factors Digest
Number 7* contains a more extensive explanation of each.

The SHEL model is useful for human factors data collection because it
allows exploration of many (hopefully all) factors that may have acted
upon the individual in question. This model removes focus from the
individual, instead highlighting that individuals do not operate on
their own, but rather interact directly with many other components in
the system.

A systemic analysis follows the data collection process. Dr. James
ReasonÕs model is helpful because it establishes a framework for
investigators to use when looking beyond the actions of front line
operators. Imagine for a moment the task of trying to get to the core of
an onion. Envision peeling back layer after layer of the onion, all the
while getting closer and closer to core. To apply this analogy back to
the context of an aviation mishap investigation, ReasonÕs model helps
investigators peel back the various layers of the system in order to
find the root cause of the event. As one works through all layers of the
system it is possible to identify how latent failures[^7] interacted
with the front line operators to create the Òwindow of accident
opportunity.Ó Reason states that front line operators are the
Òinheritors of a systemÕs defects.Ó

Because the quality of data collected and analyzed may be limited by the
models and protocols used, and because these tools are limited in
number, ALPA challenges the industry to advance development of human
factors investigation protocols and models.

**Needed: Emphasis On Incident Investigation**

The value of investigating incidents must not be underestimated. It is
widely believed that incidents are precursors of accidents. To support
this notion, industry-wide estimates suggest that there are
approximately 360 minor safety incidents that underlie each fatal
accident (Ganse, 1994). Therefore, it is important to investigate
incidents and analyze trends that may point to potential accidents.

It is commendable that many air carriers are developing in-house
incident reporting systems. However, this implementation is not without
danger. If each air carrier develops its own incident reporting system,
how does this affect the utility of a national or centralized incident
reporting system such as NASAÕs Aviation Safety Reporting System (ASRS)?
Is it possible that encouraging airline employees to submit incident
reports to their own in-house incident reporting systems may effectively
decrease the emphasis on submitting these reports to ASRS?

To realize an inherent danger of decreasing emphasis on ASRS usage, one
only must look at an event that triggered formation of ASRS. In December
1974, a Boeing 727 airliner descended prematurely and crashed into a
mountainside while on approach to WashingtonÕs Dulles International
Airport. The investigation uncovered widespread ambiguity concerning
flightcrew actions following receipt of approach clearances (NTSB,
1975). Ironically, only six weeks earlier a United Air Lines crew very
narrowly escaped the same fate while conducting the same approach to the
same airport (Hardy, 1990). The United crew reported the event to their
company Flight Safety Awareness Program, a program where crew members
were encouraged to report to the company any incidents they felt
involved a safety problem. This report resulted in safety bulletins
being issued to United crews to warn them of this trap; however there
was no clear avenue to disseminate this information to crews of other
air carriers.

There is tremendous, undisputed value of having air carriers collect and
analyze their own incident data. However, philosopher George Santayana
stated, ÒThose who cannot remember the past are condemned to repeat it.Ó
The reasons for forming ASRS must not be forgotten, lest accidents and
incidents of the past will be repeated. It is essential that a system be
developed to encourage reporters to dual report incidents to company
incident reporting systems, as well as to ASRS. ASRS has capability to
quickly disseminate safety concerns to appropriate authorities to ensure
timely corrective actions. Also, ASRS offers Òpower in numbersÓ when
looking for hazard trends. For instance, if each carrier received only
one report of a particular problem it may be difficult, perhaps
impossible, to determine the level of threat posed by this seemingly
isolated event. However, if ASRS also received a copy of each of these
reports, then it is considerably more likely that the hazard would be
detected. Obviously it is easier to detect trends with an increased
number of data points.

**Conclusions**

The goal of the FAAÕs Human Factors Data Project is to identify causes
of human error that contribute to accidents and incidents in order to
develop prevention strategies. It must be clearly understood that it is
not possible to prevent humans from making errors. What is possible,
however, is better management of human error. By conducting
investigations that focus on finding and correcting systemic problems,
it is possible to reduce the frequency of these errors and to limit
their adverse consequences.

In order to make an appreciable improvement in aviation safety, it is
essential that systemic problems be identified and corrected. Knowledge
of aviation human factors and an understanding of how to apply this
knowledge in accident and incident investigations will offer a greater
opportunity to accomplish this goal.

**References**

Federal Aviation Administration (FAA, 1995). *FAA\'s National Plan for
Civil Aviation Human Factors: An Initiative for Research and
Application.* March 1995.

Ganse, L.R., (1994). *Flight Safety Organizations: A Value Added
Functions.* Presentation to ATA Ops Forum. September, 1994.

International Civil Aviation Organization (ICAO, 1993). *ICAO Human
Factors Digest Number 7: Investigation of Human Factors in Accidents and
Incidents.* 1993 (This document is available through ICAO via credit
card order. Phone number is (514) 285-8219.)

Hardy, R., (1990). *Callback.* Washington, DC: Smithsonian Press
International.

National Transportation Safety Board (NTSB, 1979). *Allegheny Airlines,
Inc. Nord 262, Mohawk/Frakes 298, N29824, Clarksburgh, WV. February 12,
1979.* Aircraft Accident Report, NTSB/AAR-79-12. August 1979.

National Transportation Safety Board (NTSB, 1993). *Takeoff Stall in
Icing Conditions. USAir Flight 405, Fokker F28, N485US, LaGuardia, NY.
March 22, 1979.* Aircraft Accident Report, NTSB/AAR-93-02. February
1993.

National Transportation Safety Board (NTSB, 1995). *USAir Flight* *1016,
DC9-31, N954VJ, Charlotte/Douglas International Airport. July 2, 1994.*
Aircraft Accident Report, NTSB/AAR-95-XX. April 1995.

National Transportation Safety Board (NTSB, 1975). *Trans* *World
Airlines, Inc, Boeing 727-231, N54328, Berryhille, VA. December 1,*
*1974*. Aircraft Accident Report, NTSB/AAR-75-16. November 1975.

## ASSESSING THE QUANTITY AND QUALITY OF AVIATION SAFETY INCIDENT REPORTS: TWO STUDIES - Michal Tamuz

University of Texas

School of Public Health

Houston Health Science Center

In this paper, I draw on the findings of two studies that relate to one
of the workshop goals, assessing the quality and quantity of information
provided by aviation safety information systems. These studies were
conducted as a part of a larger, ongoing research project that seeks to
understand how organizations collect information about (Tamuz, 1988) and
learn from near accidents. The main argument is that while the number of
incident reports does not provide a reliable indicator of aviation
safety, the contents of the reports can provide valid safety-related
information.

The first is a study of pilot reports filed with the Near Midair
Collision System (NMAC System) operated by the Federal Aviation
Administration, as part of the National Airspace Information Monitoring
System (NAIMS). It explores how the midair collision over Cerritos
affected the reporting of near midair collisions (NMACS) by pilots in
California (Tamuz, 1988). The study examines the content validity of
NMAC reports filed in the wake of heavily-publicized accidents.

The second, a study of the Aviation Safety Reporting System, examines
changes in pilot incident-reporting behavior following the
implementation of the automated surveillance of air traffic controllers
(Tamuz, 1987). The research focuses on some of the consequences of
offering limited immunity in exchange for information on potential
dangers.

The studies examine changes over time in the reporting of incidents or
near accidents to safety information systems. The definition of a near
accident is drawn from the National Research Council\'s definition of an
air traffic incident; it is defined as an event in which \"\...no
significant damage or injuries occurred, but which, under other
circumstances, could have been an accident\....\"(NRC, 1980). In this
paper, the terms \"incident\" and \"near accident\" are used
interchangeably.

The quantity and quality of data is affected by how the information was
collected. Because safety reporting systems rely on individuals as the
primary source of information, the factors that influence individual
reporting behavior can also confound efforts to use incident data to
monitor safety conditions. Indeed, safety analysts often have observed
that counting the number of potentially hazardous events reported by
individuals does not provide a reliable safety indicator.

**Observations on the Quantity of Incident Reports**

As aviation safety experts have long realized, variations in the number
of incident reports are not only affected by changes in the underlying
safety conditions but also by factors, often unrelated to safety, that
influence individual reporting behavior. The Aviation Safety Reporting
System (ASRS) routinely cautions data-users about the reporting bias
that can confound their data analysis. Indeed, an increase in the
quantity of reports may reflect improvements in reporting rather than an
erosion in safety conditions. For example, an aviation safety
information system experienced an abrupt increase in the number of
incident reports following the distribution of new reporting forms. The
increase in reporting can also result from the influence of external
events or internal changes in the safety information system. While
monitoring near midair collisions in California, FAA safety analysts
observed a dramatic increase in the number of NMACs reported after the
Cerritos midair collision, on August 31, 1986 (FAA, 1987b). Similarly,
in a footnote to NMAC statistics, FAA safety analysts explain that the
remarkable increase in the number of NMAC reports filed during 1968-71
corresponds with the time period in which the FAA experimented by
offering immunity to those who filed NMAC reports (FAA, 1987a; Tamuz,
1994).

The evidence from aviation suggests that the number of incidents
reported to a safety reporting system is not a reliable safety
indicator. Drawing on the cumulative experience of aviation safety
experts, a series of studies confirmed this conclusion. The research
findings indicated that the number of near accidents reported varied
with changes in publicity, politics, and perceived incentives (Tamuz,
1988).

Although the confounding effects of reporting biases on
incident-reporting trends is widely-recognized by aviation safety
analysts, less is known about the relationship between the quantity of
reports filed and the quality of the data provided by them.

**Questions about the Quality of Information**

Questions remain about the validity of information contained in the near
accident reports. Consider the near midair collision (NMAC) reports
submitted to the Federal Aviation Administration. If a pilot perceives
that a near midair collision occurs and files a report with the FAA, the
FAA must gather and investigate each report, even if after further
investigation, the report is found to be mistaken. It is commonly
thought that since the number of NMAC reports filed is sensitive to
external events, such as highly-publicized accidents, that these
\"extra\" reports are likely to be mistaken or trivial. In the ASRS,
limited eligibility for immunity is offered in exchange for providing
information about potential dangers. Understandably, a common assumption
is that most incident reports are filed in order to benefit from the
immunity provisions. The following studies address these two sets of
common assumptions.

**A Study of Near Midair Collision Reporting before and after the
Cerritos Midair Collision**

Drawing on safety analysts\' observations of the effect of the Cerritos
accident on NMAC reporting, this longitudinal study monitors whether
highly publicized accidents influenced the number of NMACs reported in
California (Tamuz, 1988). The findings tentatively suggest a \"geography
of risk\", in which pilots perceptions of risk were increased by
accidents close to home but were left unaffected by accidents that
occurred in far away places. Thus, as noted earlier, tracking the number
of incidents did not appear to reflect changes in underlying safety
conditions.

Although the number of NMAC reports filed was influenced by publicity
over recent local accidents, the contents of most of these reports
provided valid information about safety concerns. Using data provided by
FAA investigators, the study examined the validity of the near accident
reports filed in response to publicity over the Cerritos accident and
other California-based accidents. As part of their investigation, FAA
flight safety investigators categorized each NMAC report as
\"critical\", \"potential\", or \"no hazard\". If, as one might assume,
the accident encouraged individuals to file an NMAC report over any
trivial event, one would expect a greater proportion of non-hazardous
reports filed after a highly-publicized accident than before it.

The findings of this study suggest the contents of incident reports
provide valid information about potentially hazardous events despite
their lack of reliability as a safety indicator. The findings indicate
that there is no significant difference in the proportion of valid to
non-hazardous NMAC reports submitted in California, before and after the
Cerritos accident, and other highly-publicized, local accidents.

**Perceived Changes in Reporting Incentives: A Natural Experiment**

This study of the Aviation Safety Reporting System (ASRS) examined some
of the effects of immunity provisions on the voluntary reporting of
potentially hazardous situations (Tamuz, 1987). One of common concerns
about offering immunity from punishment in exchange for information
about potential dangers is that individuals will focus their attention
on rule violations and report only those events for which they can
benefit from the immunity provisions (e.g., Lawler and Rhode, 1976).

In 1983, a natural experiment occurred when the FAA, as part of its
Quality Assurance Program, began to install a computerized system for
the surveillance of controllers working in air traffic control centers.
Although the system was designed to monitor the performance of
controllers, one of its indirect effects was an intensified surveillance
of pilots, albeit under limited conditions. With the implementation of
this new surveillance system, pilots apparently perceived that the
chances of detection had increased; and consequently, they perceived
greater incentives to seek immunity by reporting possible air traffic
violations to the ASRS.

Building on this natural experiment, the study examines pilot reports to
the ASRS before and after the implementation of the QAP computerized
surveillance of controllers. The study compares the reporting of
\"monitored events\" (i.e., possible air traffic violations that could
be detected by surveillance, such as altitude deviations, heading or
track deviations) to the reporting of \"non-monitored events\" (i.e.,
safety-related incidents, unrelated to air traffic violations and the
quest for immunity). With the increase in the perceived incentives for
seeking immunity, it was expected that pilots would focus their
attention solely on protecting their interests by filing reports of
possible air traffic violations to the ASRS. Thus, it was predicted that
the reporting of monitored events would increase while the reporting of
non-monitored events would decrease.

Contrary to expectations, the findings demonstrated that pilot reporting
of incidents to the ASRS could not be explained solely as a means of
seeking immunity. As expected, following the implementation of the QAP
computerized surveillance, the findings indicated a dramatic increase in
the number of pilot reports of monitored events, or possible violations.
Surprisingly, there also was an increase in the number of reports of
non-monitored events, or safety-related incidents, which could not be
detected by the surveillance system. Despite the perceived increase in
the incentives for seeking immunity, pilots continued to file reports of
safety-related incidents that were unrelated to rule violations and
thus, unrelated to gaining benefits from the immunity provisions.

**Conclusions**

While counting the incident reports gathered from individuals does not
produce a reliable measure of aviation safety, pilot reports of
potential dangers can provide a valid source of information about
potential hazards. Consider the Near Midair Collision System operated by
the FAA. Although variations in the number NMAC reports can be affected
by highly-publicized local accidents, the increase in the number of
reports also yields valid descriptions of actual and potentially
hazardous near midair collisions. Similarly, while the possibility of
gaining immunity gives pilots an incentive to file reports with the
Aviation Safety Reporting System, it is not the only motivating factor.
Pilots reported many safety-related incidents to the ASRS that did not
involve immunity provisions. Thus, in both systems, counting incident
reports does not provide a reliable safety indicator, but the contents
of the reports can contain valuable descriptions of potential dangers.

**References**

Federal Aviation Administration, Office of Aviation Safety (1987a).
Unpublished statistics.

Federal Aviation Administration, Office of Aviation Safety (1987b). An
analysis of pilot-reported near midair collisions, 1985-1986, for
California. Unpublished report dated June 12, 1987.

Lawler, E. E., & Rhode, J. G. (1976). *Information and Control in
Organizations*. Pacific Palisades, CA: Goodyear.

National Research Council, Committee on FAA Airworthiness Certification
Procedures. (1980). *Improving Aircraft Safety: FAA Certification of
Commercial Passenger Aircraft.* Washington, D.C.: NAS.

Tamuz, M. (1987). The impact of computer surveillance on air safety
reporting. *Columbia Journal of World Business. 22,1:69*-*77.*

Tamuz, M.(1988). Monitoring Dangers in the Air: Studies in Information
and Ambiguity. Ph.D. Dissertation. Stanford University.

Tamuz, M. (1994). Developing organizational safety information systems
for monitoring potential dangers. In G.E. Apostolakis and J.S. Wu (Eds.)
*Proceedings of PSAM II,* An international conference devoted to the
advancement of system-based methods for the design and operation of
technological systems and processes, San Diego, California, March 20-25,
1994, University of California, Los Angeles. Volume 2, Pages 71:7-12.

## IN SEARCH OF ANSWERS WITHOUT QUESTIONS: OR, HOW MANY MONKEYS AT TYPEWRITERS WILL IT TAKEÉ - John A. Wise and Mark A. Wise

Embry-Riddle Aeronautical University University of Central Florida

Daytona Beach, FL Orlando, FL

**Abstract**

It is argued that it will be difficult to identify potential problem
areas in new technology areas *a priori* because of the lack of theory
and/or hypotheses to tell one were to look. Random searches, or searches
based on theory relevant to old technology, will be of limited value.
Some non-standard approaches to the problem area are discussed.

ÒAll men by nature desire to know.Ó

Aristotle, *Metaphysics*, (p.1)

**Introduction**

Rapid advances in technology and software have provided the capability
to develop very complex systems that have highly interdependent
components. While this has permitted significant increases in system
efficiency and has allowed the development and operation of systems that
were previously impossible (e.g., negative stability aircraft), it has
also brought the inherent danger of system induced catastrophes. Perrow
(1984), in his book *Normal Accidents*, has demonstrated that
highly-complex systems with highly interdependent components have an
inherent disposition toward massive failure.

Systems that have high intradependence often create new types of
failures. Interrelated components that were previously independent can
cause unpredicted failures in each other. For example, the tests of
wide-bodied aircraft initially used the same criteria for cabin
depressurization as older narrow bodied aircraft. As a result, when the
DC-10 lost a cargo door in flight (an unskilled ground crew could not
lock it properly because of the complexity of the procedure), the
insufficient means of depressurizing the cabin caused the floor to
buckle jamming the controls. The unpredicted intradependence between
ground personnel skill, procedure complexity, cabin pressure, and flight
controls resulted in a crash and the loss of many lives.

As complex systems become more intradependent, interdisciplinary issues
become more critical. Nowhere is this more true than in the
person-machine interface. It is likely that new operational interface
problems will reside in locations where disciplines (and the system
components relevant to their domain) meet and interact. It is in these
intellectual intersections that most new compromises and
cross-discipline trade-offs are made. And it will be in these
intersections, that new interface induced failures will emerge that will
probably not have been anticipated. It is also in these intersections
that it will be the most difficult to determine where to look for
potential faults *a priori*.

Given the above issues, it is difficult to argue against the goal of
this meeting: to identify ways of predicting where problems will exist Ð
before they occur Ð so as to mitigate their consequences. However, we
will argue that the odds of such an undertaking being successful are
low. The history of science is ripe with debates on this issue, but it
is generally accepted today that any search is always guided by some
theory or hypothesis. Pirsig (1974) in his classic book *Zen and the Art
of Motorcycle Maintenance* debated the origin and value of knowledge and
quality. He noted that there has been a long debate within the
philosophy of science of how science and scientific theories develop.
Pirsig was describing the arguments of the French philosopher and
mathematician Henri PoincarŽ when he wrote:

Which facts are you going to observe? he \[PoincarŽ \] asked. There is
an infinity of them. There is no more chance that an unselective
observation of facts will produce science than there is that a monkey at
a typewriter will produce the LordÕs prayer. (Pirsig, 1974, p. 237)

The equivalent in the current discussion would be that Òthere is no more
chance that the unselective observation of aviation data will produce
insight into future accidents than there isÉ Ó So one needs to ask the
following basic questions/. How do we develop selective observations in
areas where little theory or other insight exists? How do we know where
to selectively look when we are using a new technology or when we are
using a procedure that has never been applied?

**How About Science?**

The scientific model, as PoincarŽ has noted, is particularly weak at
making predictions without an underlying theoretical base. As a matter
of fact many philosophers of science have argued that it is impossible
for science to work at all without theory to guide the search. The
scientific method itself is rooted in the empiricist philosophy of the
John Locke, George Berkeley, and David Hume. These philosophers maintain
that all knowledge is gained through empirical, *a posteriori*
experience or observation. These scholars have taught us that step one
of the scientific method is the statement of the problem and step two
involves developing hypothesis. Scientific research involves the testing
of the hypotheses, without a statement problem or hypothesis, inquiry
can not proceed within the scientific method.

The analysis and prediction of future errors or events is based
primarily on the ability to infer from those which have previously
occurred, and a belief that there is justification for believing that
these events will or could happen again (Hume, 1748). However, to
predict those events which have no historical correlate, requires
something much more Ð foresight. If, as the old adage states Òhindsight
is 20/20,Ó then foresight would be at least functionally blind. History
suggests that no matter how diligently and methodically a system is
tested *a priori,* there still exists a possibility for failure. Take
for example ÒThe Unsinkable Ship,Ó the Titanic Ð it sank. Thus, the
question becomes, ÒHow do we form reasonable hypotheses about future?Ó

Even if it were possible to identify the hypotheses, the scientific
method would still be extremely difficult to apply. To address the
current inquiry within the scientific paradigm, the following problem
statement might be established: ÒWhat events will lead to undesirable
situations within the aviation system?Ó However, further attempts to
break this problem down into more manageable tasks, proves laborious, if
not futile. The complexity of the interface between human and machine
provides a number options which, when analyzed mathematically approach
infinity, especially when coupled with time and highly unpredictable
environmental factors (both physical and psychological).

Hume (1748), Mason and Mitroff (1973), and others have also argued that
science is basically a consentual method of inquiry. That is, the
ÒtruthÓ of a scientific theory is based on the amount of prevailing
opinions for the theory exists among the scientists of that field (i.e.,
does it win a referendum vote?). If the topic area is new (e.g., new
technologies or new procedures) there will not be many prevailing
opinions, there will be much disagreement and debate. Mason and Mitroff
observed that if the consentual position is at all suspect (i.e., there
is not a large majority supporting a given theory) then something other
than traditional science may be needed.

**How About Simulation Or Modeling?**

Unfortunately modeling and simulation, while they are extremely powerful
tools in structured problem spaces, have problems when it comes to new
technologies and/or procedures. Modeling and simulation use a formal
(often mathematical) representation of the problem space by starting
from a set of basic ÒtruthsÓ and then systematically build a network of
more formal propositional truths.

Modeling and simulation normally have high value because they can be
tested by the exacting tests of logic and mathematics, such as internal
consistency, completeness, comprehension. Mason & Mitroff (1973) have
argued that simulation and modeling are extremely powerful when working
on well structured problems for which an analytic solution exists. Their
weakness however, overlay the area of the proposed inquiry: The future
of relatively unknown technologies. For example, if one had used a
formal model to investigate the impact of the automobile on pollution
when it was originally introduced, the result no doubt would have been
that the automobile would drastically reduce pollution. (The principal
pollutant at the time was horse manure which was ubiquitous and a
serious health hazzard in most large cities. ) The modelÕs basic truths
would have probably not included anything issues associated with air
pollution, and thus it would not have been predicted.

**Is There A Solution?**

Several non-traditional methods of inquiry have been suggested in the
literature. For example, some (e.g., Mitroff & Betz, 1972) have argued
that dialectical inquiry may be the best method of investigation into
future events. Dialectical inquiry (which involves an dynamic argument
over common data by persons holding two polar worldviews) has the
advantage that the participants will always attack the assumptions on
which the oppositionÕs predictions are made. The attacks on assumptions
are critical when addressing the future. Assumptions about the future
can often be treated dogma (e.g., they will never change), because it is
often difficult to imagine alternative models of the world based on not
yet evolved systems behaviors. For an example of this phenomenon, one
only need remember a few years ago, when the real estate purveyors were
telling everyone that real estate prices could only increase. A
dialectical inquiry would have attacked that assumption, and at least
made the public aware that a fall was possible and should be considered.

Dialectical inquiry does not guarantee any better conclusions, but it
does maximize the chance that the results obtained are held with
caution. As a matter of fact, dialectic inquiry would demand that an
antithesis to the conclusion be immediately formed and begin to attack
its weaknesses.

An other approach is to Ògo with what you know.Ó Use data and knowledge
that is *generalizable to all systems.* Following the path of
experience, examine possibilities that can be reasonably inferred from
the past, even if they fall upon the outer edge of the path.
Additionally, it often helps to examine systems, however new or complex
in a top-down manner (Wise & Wise, 1994). That is go from the systems
level to the directly applied. Both of these approaches allow one to
build on relevant experiences from other domains (e.g., computer
science, space exploration) and form reasonable hypotheses and identify
successful methods of inquiry.

**Conclusions?**

Hume (1748) has argued that:

All inferences from experience suppose as their foundation, that the
future will resemble the past, and that similar powers will be conjoined
with similar sensible qualities. If there be any suspicion, that the
course of nature may change, and that the past may be no rule for the
future, all experience becomes useless, and can give rise to no
inference or conclusion (p. 24).

If Hume is correct and human knowledge about the probability of future
aviation accidents is based on experience or data (e.g., Aviation Safety
Reporting System) will be limited to current systems and procedures.
However, if there is a situation, such as the development of highly
complex and intradependent systems, which have no correlate to data that
we have on the past, then no conclusion can possibly reached which would
lead one to discover or predict future events.

The underlying problem, with the sort of inquiry desired by the aviation
community (i.e., predicting future problems), is that it is a process of
searching for an answer where no question has been (or can be) defined.
Additionally, the complexity and intradependence of current and future
systems, coupled with the unpredictable nature of a human operator,
creates an near infinite number of contingencies, making a search of all
possibilities impossible. The desired inquiry is analogous to a
pathologist trying to predict diseases that will occur twenty years from
now. Who would have predicted the genesis and spread of HIV 30 years
ago? All of the leads and clues were there, but they were impossible to
perceive, to piece together. In much the same way that the clues which
would lead to the prediction of future pilot and system errors do exist,
however they will be virtually impossible to see without the gift of
hindsight.

It should be noted that the arguments being posed here are NOT arguing
that the proposed effort not be undertaken Ð in fact, the authors would
argue in its favor (see Wise & Wise, 1993; 1994). The aim of this paper
has been to point out the weaknesses of traditional inquiring methods in
dealing with future events. Especially, when it is anticipated that
these events may have no necessary connection to past or previously
examined events.

In order not to fall into an inquiry trap, in which the search for an
answer is limited or defeated by using an inappropriate inquiry method,
the approach that will be taken during this probe is equally as
important (or more so) than the probe itself.

The cautious words of Confucius may best summarize the proposed inquiry.

ÒTo know that you know what you know;

and that you do not know what you do not know Ð

that is true knowledge.Ó

Confucius

**Addendum**

A major question at the workshop was ÒWhat data should be included in an
accident prevent database?Ó Given the above arguments about not being
able to predict future questions *a priori*, it follows that one cannot
predict *a priori* the data will be needed to answer those questions.
There will always be a need for some set of fundamental data (e.g.,
basic flight parameters), but new technologies and new operational
paradigms (e.g., free flight) will generate their own problems that will
have specific data needs that will not be obvious until the problems
surface. Thus, the most important data goal for the system must be
flexibility! Flexibility to quickly adapt its data acquisition tools to
collect new types of data in an effective and efficient manner.
Flexibility to acquire relative large amounts of data on a new topic in
a short period of time. Flexibility to develop new tools to collect data
that has no current acquisition device. If the system is not flexible in
its data acquisition capabilities, it will only help solve problems for
which we already have solutions.

**References**

Aristotle, *Metaphysics*, (p.1)

Hume, D. (1748/1977). *An Enquiry Concerning Human Understanding.*
Indianapolis: Hackett Publishing Co.

Mitroff, I.I. & Betz, F. (1972). Dialectical decision theory: A
meta-theory of decision making. *Management Science, 19*, B-634 -B-648.

Perrow, C. (1984). *Normal Accidents: Living with High-Risk
Technologies*. New York: Basic Books.

Pirsig, R.M. (1974). *Zen and the Art of Motorcycle Maintenance.* New
York: Bantam New Age Books. (May 1982 Edition)

Wise, J. A. & A. Debons (Eds.). (1987). *Information Systems: Failure
Analysis.* Berlin: Springer-Verlag.

Wise, J. A., V. D. Hopkin, & M. L. Smith (Eds.). (1991). *Automation and
Systems Issues in Air Traffic Control.* Berlin: Springer-Verlag.

Wise, J.A., & Wise, M.A. (1993). Basic considerations in verification
and validation. In: J.A.

Wise, V.D. Hopkin, & P. Stager (eds.) *Verification and Validation of
Complex Systems: Human Factors Issues.* Berlin: Springer-Verlag.

Wise, M.A., & Wise, J.A. (1994). On the systems approach to certify
advanced aviation technologies. In: J.A. Wise, D.J. Garland, & V.D.
Hopkin (eds.) *Human Factors Certification of Advanced Aviation
Technologies.* Daytona Beach, FL: Embry-Riddle Aeronautical University
Press.

## LEARNING FROM ACCIDENTS: THE NEED TO LOOK BEHIND HUMAN ERROR - David D. Woods and Nadine B.Sarter

Cognitive Systems Engineering Laboratory

The Ohio State University

Human error is often cited as the cause or a contributing factor to
accidents and incidents in domains such as aviation where operators
perform their duties in cooperation with advanced technological systems.
The implication is that problems reside either in the human or in the
engineered side of the equation (Woods, 1990). The perception of a human
error problem then leads to proposals to minimize people\'s involvement
in overall system operations through the introduction of more and more
automation of functions.

The situation is far more complex, however, as indicated by the results
of research on human-automation interaction in a variety of domains
(Wiener, 1989; Sarter and Woods, 1994, 1995; Moll van Charante et al.,
1992). These studies show that most incidents and accidents are the
result of breakdowns in the communication and coordination between human
and machine agents \-- breakdowns which are related to design flaws and
inappropriate training, both of which fail to support the formation of a
human-machine team. This research illustrates that the label human error
is prejudicial and unspecific, and that it retards rather than advances
our understanding of how complex man-machine systems fail and how the
same operator behavior can contribute to both success and failure.
Labeling undesirable actions and faulty assessments as human error
identifies a symptom, not a cause; the symptom should call forth more
in-depth investigations of how a system comprising people,
organizations, and technologies both functions and malfunctions
(Rasmussen et al., 1987; Reason, 1990; Hollnagel, 1993).

Most incidents and accidents involve a conjunction of several failures
or factors such as breakdowns in crew communication and coordination,
organizational pressures, or the introduction of clumsy technology (see
Sarter and Woods, 1995). It is not possible nor helpful to single out
one individual agent or aspect of the event as the critical causal
factor.

The Cognitive Systems Engineering Laboratory (CSEL) approach to
analyzing incidents and accidents emphasizes this inseparability of
human, machine, and environment in their contribution to overall system
performance (Woods et al., 1994). The focus of this approach is not on
any one individual system component but on the interaction and
(in)compatibility of the various elements. The goal is to identify,
analyze, and eliminate mismatches between human, machine, and
operational environment in order to prevent the (re-)occurrence of
observed problems and to correct predictable problems before they occur.

The CSEL approach is future- and prevention-oriented as it goes beyond
phenotypical descriptions of processes involved in the evolution of an
incident and instead looks for genotypical patterns in incident and
accident data (Hollnagel, 1993). We attempt to identify lawful factors
that govern different types of erroneous actions or assessments in order
to be able to predict and prevent problems. These lawful factors involve
cognitive factors \-- e.g., knowledge, attention or mindset, and goal
conflicts, clumsy use of technology, mode awareness, and organizational
pressures. These patterns are verified and further explored in the
context of empirical research. Based on these patterns, generic
conceptual frameworks are being developed that allow for the application
of findings in one domain to other fields of practice that involve
similar forms of human-machine interaction.

One important step in this approach is to build a model of how operators
behave in a locally rational way given the knowledge, attentional, and
strategic demands at work in that particular field of activity. It is
critical to adopt the position of the practitioner in a given context
rather than that of a detached omniscient observer in order to
understand and not condemn behavior. Normative models of practitioner
behavior usually are not helpful in this context.

In summary, it seems important to view incident and accident
investigations as an opportunity to learn, not as a means to assign
blame and responsibility. To learn we must avoid the biasing effect of
knowledge of outcome (the hindsight bias) and we must go beyond the
search for absolute single causes. If the goal of accident
investigations is to improve the reliability of distributed
human-machine systems, we must try to learn about vulnerabilities to
failure, about effective strategies for change, and about the
prioritization of investments. The potential for constructive change
lies behind the label human error \-- it is the starting point, not the
ending point of any investigation.

**References**

Hollnagel, E. (1993). Human reliability analysis: Context and control.
London: Academic Press.

Moll van Charante, E., Cook, R.I., Woods, D.D., Yue, L., and Howie, M.B.
(1992). Human-computer interaction in context: Physician interaction
with automated intravenous controllers in the heart room. In
IFAC/IFIP/IFOEA/IEA (Eds.), Proceedings of the Fifth Symposium on
Analysis, Design, and Evaluation of Man-Machine Systems. The Hague,
Netherlands.

Rasmussen, J., Duncan, K., and Leplat, J. (Eds.) (1987). New technology
and human error. Chichester, England: Wiley.

Reason, J. (1990). Human Error. Cambridge, England: Cambridge University
Press.

Sarter, N.B. and Woods, D.D. (1994). Pilot interaction with cockpit
automation II: An experimental study of pilotsÕ model and awareness of
the Flight Management System. International Journal of Aviation
Psychology.

Sarter, N.B. and Woods, D.D. (1995). Ò ÔStrong, silent, and
out-of-the-loopÕ: Properties of advanced (cockpit) automation and their
impact on human-automation interactionÓ. CSEL Technical Report TR-95-01,
February.

Wiener, E.L. (1989). Human factors of advanced technology (Ôglass
cockpitÕ) transport aircraft. Technical Report 177528, NASA-Ames
Research Center, Moffett Field, CA.

Woods, D.D. (1990). Modeling and predicting human error. In J.L. Elkind,
S. Card, J. Hochberg, and B. Huey (Eds.), Human performance models for
computer-aided engineering. New York: Academic Press.

Woods, D.D., Johannesen, L., Cook, R.I. and Sarter, N.B. (1994). Behind
human error: Cognitive systems, computers, and hindsight. Crew Systems
Ergonomic Information and Analysis Center (CSERIAC), Dayton, OH.

[^1]: ^?^A \"Single Event Issue\" is the issue of an event that, if
    discovered, could result in the avoidance of an accident. Such an
    event might be the discovery that an airport Minimum Descent
    Altitude was not appropriate on a chart. For example, if the MDA at
    airport XYZ was stated as 863\' MSL instead of 1863\' on an airport
    with a field elevation of 1463\' MSL, it is obvious this discovery
    could save some one from possibly having a controlled flight into
    terrain accident.

[^2]: This paper is based on conceptual materials provided by the
    following individuals: Loren J. Rosenthal, the Battelle Program
    Manager of ASRS, and Vice President of the Aviation Operational
    Safety and Efficiency Business for BattelleÕs Transportation
    Division; Susan J. Mangold, a Principal Research Scientist and
    aviation human factors specialist with BattelleÕs Transportation
    Division ; and Linda J. Connell, a NASA Ames Research Psychologist
    and ASRS Staff Scientist. Dr. Morrison is a Battelle Principal
    Research Scientist and ASRS Research Coordinator.

[^3]: The FAA has tasked the Battelle Memorial Institute with
    development of a Model AQP Program which will identify innovative
    ways in which scenario-based training can be applied to LOFTs, LOEs,
    ground school curricula, field training devices, and simulator
    training (Mangold & Neumeister, 1995).

[^4]: This task definition was excerpted from a research proposal,
    ÒHuman Factors Taxonomy,Ó prepared by Loren J. Rosenthal in June
    1993.

[^5]: ^?^ British Airways has tested several human factors coding
    schemes, including one developed by John Chappelow of the RAF
    Institute of Aviation Medicine.

[^6]: By suggesting that investigations look at root causes, ALPA is not
    suggesting that front line operators are never to blame. However,
    the Association recognizes that if any appreciable improvements are
    to be made in aviation safety, they must be done through a
    comprehensive systems approach directed towards getting at the root
    cause of mishaps. We see little benefit in investigating accidents
    and incidents without a commitment to investigate all elements of
    the system.

[^7]: ^2^ Latent failures are flaws in the system that are lying
    dormant, waiting to become manifested. An example of a latent
    failure is the design problem with the on-board windshear detection
    system that was cited in the Charlotte accident. The problem existed
    for many years but did not become manifested until combined with a
    series of other flaws, such as FAA problems with implementing TDWR,
    ATCÕs non-adherence to prescribed weather dissemination practices,
    and crew procedural errors.