Human error prediction in Air Traffic Management:

A comparison of two approaches

**Steven Shorrock**^1^**, Barry Kirwan**^2^**, Ed Smith**^3^

^1^DNV Consulting, High Bank House, Exchange Street, Stockport,
Cheshire, SK3 0ET.

^2^EUROCONTROL Experimental Centre, Centre du Bois des Bordes, BP15,
F-91222, Brétigny-sur-Orge, Cedex, France.

^3^DNV Consulting, Palace House, 3 Cathedral Street, London, SE19 9DE.

**Introduction**

Air Traffic Management (ATM) is in a period of major change, with a
variety of new concepts under consideration to help air traffic
controllers cope with projected increases in traffic levels. New
technology will enable considerable changes to older methods of
operation, and automation is now widely acknowledged to be one of the
only ways to meet future capacity demands. Automation, or cognitive
support tools, in aviation has been subject to increasing investigation
and analysis over recent years, particularly since the introduction of
glass cockpit systems. In ATM, full automation is highly unlikely in the
foreseeable future, but automation of some functions has been introduced
since the 1960s, for example flight data gathering and processing (e.g.
Radar Data Processing and Flight Data Processing) and conflict alerting.
This trend is continuing with tools such as Electronic Flight Strips and
Final Approach Spacing Tools. In future, communication tools such as
datalink and decision support tools such as 'risk' display tools and
conflict resolution tools may be implemented, and perhaps changes to the
way that ATM goals are achieved, such as free flight, time-based
separation and increased delegation of controller tasks to the cockpit.

One of the key issues that has emerged from these studies is the effect
that technology might have on 'human error'. 'Human error reduction' was
once seen as a natural consequence of automation; system designers
thought that if human operators could be moved to the fringes, the risk
of human error would decrease. But this premise was questioned (Wiener
and Curry, 1980), and the real situation is still being revealed. On the
basis of experimental studies and operational experience, many
commentators asserted that computers, and automation in particular,
produce new error forms or additional sources of errors (Wiener, 1988;
Sarter and Woods, 1995b). Another view is that "Computers do not produce
new sorts of errors. They merely provide new and easier opportunities
for making the old errors" (e.g. Kletz, 1988). It is also a widely held
view that the consequences of errors that do occur are likely to be more
serious (Weiner, 1985; Billings, 1988; Leroux, 2000). Whatever the case,
it is clear that human error is an issue that still needs to be managed.

Various means exist to assist in this process. One family of approaches
called 'Human Error Analysis' (HEA) has seen increasing use since the
1980s. These methods involve two types of approaches. First,
*group-based* approaches utilise a multi-disciplinary team of
individuals to help brainstorm and analyse potential 'failures'. These
approaches stem from the Hazard and Operability (HAZOP) and 'What-if'
query-based methods. Second, *analyst-led* approaches generally utilise
a task analysis and a classification system to probe potential errors
and their psychological and contextual origins. These approaches include
SHERPA (Embrey, 1986), CREAM (Hollnagel, 1998) and TRACEr (Shorrock and
Kirwan, 1999, 2002). Such methodologies, when supplemented with other
techniques, can help to provide an integrated approach to 'designing for
safety' via explicit consideration of human involvements.

Two HEA approaches for 'human error analysis' that can support the
process of designing for safety are:

-   **(Human) HAZOP** - an established, group-based approach to human
    hazard identification based on the HAZOP study method, developed in
    the chemical industry.

-   **TRACEr-lite** - a relatively new, single analyst-led approach to
    human error analysis developed for ATM, analogous to the
    engineering-based 'Failure Modes and Effects Analysis'.

These approaches were compared in via independent analyses of three
EUROCONTROL concepts: **Co-space**, **Time-Based Separation** and **CORA
2**. These represent varying degrees of change in the way that ATM is
performed, and in the level of computer support involved. The objective
of the project was to assess the relevance of the techniques, show what
they can deliver in terms of safety and design insight, and to show the
relative advantages of each for human error analysis purposes. The
project did not attempt to provide a full safety assessment, but rather
to provide an illustration of how human error could be considered
prospectively.

**The Systems**

***Co-space***

The Co-space project aims to increase controller availability though a
reorganisation of tasks between controller and pilot, and thereby
achieve a more effective task distribution that is beneficial to all
parties[^1]. It is expected that increased controller availability could
lead to improvements in safety, efficiency and/or capacity. Delegation
of the aircraft spacing task from the controller to the flight crew is
envisaged as a possible option to help achieve this. For aircraft within
an aircraft arrival stream, the delegation could consist of tasking the
flight crew to maintain a given spacing value to a lead aircraft, as
defined by the controller.

Within the Co-space concept, spacing tasks are delegated to flight crews
upon controller initiative; the controller decides to delegate if
appropriate and helpful. The delegation is limited since the controller
can only delegate 'low-level' tasks (monitoring and implementation) as
opposed to 'high-level' tasks (conflict detection and resolution). The
delegation is flexible since the controller has the ability to select
for each situation the level of task to be delegated from monitoring up
to implementation. The delegation takes advantage of emerging
technologies in pre-operational state along with additional avionics
such as Cockpit Display of Traffic Information (CDTI) or an Airborne
Separation Assistance System (ASAS). This project focuses on near-term
applications being considered by certain ATC organisations for both
en-route airspace and terminal areas.

In the scope of defining a new task distribution between controllers and
flight crews, from the onset of the project, two key constraints were
identified and adopted. The first one is related to human aspects and
can be summarised as 'minimise change in current roles and working
methods of controllers and flight crews'. The second one is related to
technology and can be expressed as 'keep it as simple as possible'.

The project is well established having just completed a third cycle of
user simulations and has already incorporated human factors and
operational expertise. A set of delegation procedures exists, which was
used as the basis for analysis by HAZOP and TRACEr-lite.

***Time Based Separation***

The Time-Based Separation (TBS) project aims to define and investigate
the relevance of a new concept of operation applied to the arrival phase
of flights. This concept involves replacing actual distance based
separations with time intervals.

More specifically, the project will investigate the possibilities of
preventing loss of runway capacity under strong wind conditions while
maintaining required levels of safety performance. The project will:

-   Assess a new concept of separation based on time interval as opposed
    to Radar or ICAO Wake Vortex separation criteria.

-   Investigate the use of lateral separations of less than 3 NM (or
    less than 2.5 NM if this is in use).

-   Explore possibilities of compensation of wind effect by aircraft
    speed adjustment and required ATC techniques.

    This project is in the very early stages of background research and
    concept development, with no outline procedures at the time of the
    study.

***CORA 2***

The Conflict Resolution Assistant (CORA) provides computer-based support
for air traffic controllers in the detection, identification,
prioritisation and resolution of predicted conflicts in the en-route
flight phase. Conflict identification and resolution is a core ATC task
today, carried out by Planner Controllers (longer-range) and Tactical
Controllers (shorter-range) by scanning the radar display and paper or
electronic strips. Without change, increased traffic will naturally
bring more conflicts, leading to higher workload for controllers and
more complex conflict resolution problems. CORA aims to improve planning
and anticipation processes through earlier conflict notification and
resolution decisions with the introduction of new computer-based ATM,
and the associated evolution of ATC procedures, roles, tasks and working
methods. This improvement should also help to smooth peaks in controller
activity, redistribute workload between Planner and Tactical
Controllers, and improve level and quality of service for airlines by
minimising deviation from airline optimal trajectories via earlier and
more strategic resolutions.

The 'CORA 1' system will identify conflicts for controllers and support
the planning and decision making process by helping to test the impact
of tactical clearances on the traffic situation. 'CORA 2' will provide a
set of ranked, conflict-free resolution advisories for the controller,
who can directly select and implement one of the suggested advisories,
or employ a different, self-generated resolution. The CORA 2 project is
defining and developing operational requirements and prototype enhanced
concepts for conflict resolution.

**The Techniques**

***HAZOP***

Hazard and Operability (HAZOP) studies provide a formal, systematic and
critical examination of the process and engineering intentions of a
design. HAZOP assesses the potential for hazard and identifies
mal-operation or malfunction of individual items of equipment and the
consequences for the whole system. This examination of the design is
structured around a set of guidewords, which ensure comprehensive
coverage of possible problems. HAZOP studies normally involve a team who
have experience of the system or design to be studied, including design,
engineering and operational personnel, often also including training
specialists, human factors specialists, and independent safety
specialists.

There are generally four overall objectives addressed by HAZOP:

-   to identify all deviations from the way the design is expected to
    work; their causes, and all the hazards and operability problems
    associated with these deviations;

-   to decide whether action is required to control the hazard, or the
    operability problem, and if so to identify the ways in which the
    problem can be solved;

-   to identify cases where a decision cannot be made immediately and to
    decide on what information or action is required;

-   to ensure that actions decided upon are followed through.

The (Human) HAZOP study method can also be applied to the study of a
task, human-machine interface or operating procedure. A 'task' can be
seen as a set of things including a system goal, resources for
accomplishing the system goal, such as information and controls, and a
set of constraints on how the goal may be achieved using these resources
(Shepherd, 2001). A 'procedure', meanwhile, is defined here as a set of
instructions whose aim is to direct an operator to make changes to the
state of a system in a safe manner, so that a particular objective is
achieved.

The first process is to identify the intent of the task or procedure,
including the system state at the beginning of the task/procedure, and
the required system state at the end, and how it will be achieved. The
detailed study is then carried out in a step-by-step manner and for each
task or task step the team agree an objective or intent. They will then
view the step as requiring an action at a time in a sequence. If
procedural documentation is available, they will also review the wording
of the instruction. To assist this process, a set of guidewords is used.
For a Human HAZOP study this might include some or all of the following:

+-----------------------------------+-----------------------------------+
| *Basic Guidewords*                | -   More information              |
+-----------------------------------+-----------------------------------+
| -   No action                     | -   Less information              |
+-----------------------------------+-----------------------------------+
| -   More action                   | -   No information                |
+-----------------------------------+-----------------------------------+
| -   Less action                   | -   Wrong information             |
+-----------------------------------+-----------------------------------+
| -   Wrong action                  | *Additional Concepts*             |
+-----------------------------------+-----------------------------------+
| -   Part of action                | -   Purpose                       |
+-----------------------------------+-----------------------------------+
| -   Extra action                  | -   Clarity                       |
+-----------------------------------+-----------------------------------+
| -   Other action                  | -   Training                      |
+-----------------------------------+-----------------------------------+
| -   More time                     | -   Abnormal conditions           |
+-----------------------------------+-----------------------------------+
| -   Less time                     | -   Maintenance                   |
+-----------------------------------+-----------------------------------+
| -   Out of sequence               | -   Safety                        |
+-----------------------------------+-----------------------------------+

A HAZOP study meeting will follow through a series of steps repeatedly.
There are seven stages, which are repeated many times during a HAZOP
(see Figure 1). Examination sessions are structured with the team leader
facilitating the discussion, and a secretary recording the deliberations
on log sheets, detailing tasks, deviation guidewords, causes,
consequences, safeguards, risk ranking (if used) and recommendations.
The discussions are recorded on logsheets (see Table 1 for an example
extract).

![](media/image1.wmf)

[Figure 1:]{.underline} Process for carrying out a HAZOP Study

[Table 1:]{.underline} Extract of HAZOP worksheet for Co-space.

  ---------------------------------------------------------------------------------------
  **Project:                                                        
  Co-space,                                                         
  Delegation of                                                     
  "Merge                                                            
  Behind"**                                                         
  ------------- ------------ ------------------ ------------------- ---------------------
  **Guideword / **Causes**   **Consequences**   **Safeguards**      **Recommendations**
  Deviation**                                                       

                                                                    

                                                                     

  2\. Wrong     2.1.         2.1. Loss of       2.1. Target         1\. Anti Overlap
  action        Controller   separation         positioning by      (display
                identifies                      pilot               decluttering) as it
                wrong target                                        currently exists
                                                                    needs some
                                                                    improvement. Review
                                                                    how this software
                                                                    tool can be used to
                                                                    support the
                                                                    controller during
                                                                    delegation and what
                                                                    improvements are
                                                                    required for it to be
                                                                    effective.

                                                2.2. Confirmation   2\. Consider making
                                                of target           target positioning by
                                                                    pilot a compulsory
                                                                    subtask in target
                                                                    selection.

                                                2.3. Read back of   3\. Explore how data
                                                target from pilot   link technology could
                                                to controller       be used to support
                                                                    both controller and
                                                                    pilot when selecting
                                                                    a target during
                                                                    delegation.

                                                2.4. The pilot may   
                                                question the target 
                                                selection if he has 
                                                enough supporting   
                                                information         

                                                2.5. Controller      
                                                monitoring of the   
                                                aircraft may        
                                                identify that the   
                                                pilot has the wrong 
                                                target later in the 
                                                task.               

                                                2.6. The pilot\'s    
                                                TCAS (visual and    
                                                audible alarm)      

                                                2.7. The             
                                                controller\'s STCA  
                                                (short term         
                                                conflict alert)     
                                                will sound a couple 
                                                of minutes before   
                                                separation          
                                                infringement        

                                                2.8. The use of      
                                                Anti-Overlap        
                                                software tool on    
                                                the controller\'s   
                                                interface           
  ---------------------------------------------------------------------------------------

***TRACEr-lite***

TRACEr (Technique for the Retrospective and Predictive Analysis of
Cognitive Errors) (Shorrock and Kirwan, 1999, 2002) was developed from
literature findings and operational experience for predictive and
retrospective human error analysis in ATM. TRACEr was represented as a
set of decision-flow diagrams containing human error modes and
mechanisms, intended for use by human factors specialists[^2]. TRACEr
was applied to a number of ATM projects, retrospectively and
predictively. In its predictive mode, TRACEr was applied in the UK to a
new ATM situation display, electronic flight strips and new controller
tools (Shorrock, *et al*, 2001), a Final Approach Spacing Tool for use
by Heathrow Approach Control (Evans *et al*, 1999), and an analysis of
reduced separation minima in unregulated airspace (Shorrock and Kirwan,
2002). These trial applications delivered useful insights for the
projects, and led to new design decisions. Significantly, some of the
errors predicted were observed in subsequent simulations and trials, and
most of the errors observed had been predicted previously, thus
providing an indication of good predictive validity[^3]. Whilst these
early studies were encouraging, it was realised that for the potential
of TRACEr to be realised more fully, a simpler, reduced-scope version
was required. This version of the technique was called 'TRACEr-lite'
(Shorrock, 2002a, b).

For predictive use of TRACEr-lite, the analyst first scopes the
analysis, and then conducts a task analysis, e.g. using Hierarchical
Task Analysis (HTA) (see Shepherd, 2001). Using TRACEr-lite and the task
analysis, the analyst determines what could go wrong. There are four key
components to the TRACEr-lite toolkit (see Figure 2).

Prior to the TRACEr-lite analysis, during the task analysis process,
Performance Shaping Factors (PSFs) are analysed to set the scene for the
analysis. PSFs are those factors, either internal to the controller or
pilot, or relating to the task and operational environment, that affect
performance positively or negatively, directly or indirectly. The PSFs
are used to prepare a general '**Context Statement**' - a set of
statements about the performance conditions under which the controller
will be working. PSFs may also be analysed separately for particular
task steps, relating to specific errors, if the analyst so chooses. Each
PSF takes the form of a question, eliciting a 'yes'/'no' type response
and a statement of justification. These questions occupy eight
categories, such as traffic and airspace, procedures and documentation,
training and experience, workspace design, HMI and equipment, etc.
TRACEr-lite does not predetermine the links between PSFs and error
modes/mechanisms because of the many-to-many mapping relationships
involved, as well as the uncertainty in making such specific links.
However, some general guidance is provided on how particular types of
PSF affect cognitive processing.

[Figure 2:]{.underline} TRACEr-lite light prototype interface and
predictive task steps.

![](media/image2.png){width="4.9215277777777775in"
height="3.6909722222222223in"}

Once a context statement is prepared the analyst engages in a cycle of
activities, applying the TRACEr-lite taxonomies to the detailed task
steps within the task analysis. **External Error Modes** are first used
as prompts to enable the identification of the observable manifestations
of potential errors, based on logical outcomes of erroneous actions, in
terms of timing, sequence, selection and quality. Examples include
'Omission', 'Wrong action on right object', 'Mis-ordering', and
'Information not sought / obtained'. EEMs are context-free and
independent of cognitive processes (e.g. intention). However, when used
on a task step from a task analysis, the error mode is converted to a
contextual 'external error' (e.g. 'Controller fails to issue instruction
to select target').

The cognitive aspects of the error are analysed using a set of Internal
Error Modes and Mechanisms. These are structured around four error
domains, with an associated question prompting further classification:

-   **Perception** - Does the controller/pilot have to see or hear
    something during the task step?

-   **Memory** - Does the controller/pilot have to recall information or
    remember to perform actions in the future during the task step?

-   **Decision making** - Does the controller/pilot have to project
    required separation, or make a plan or decision during the task
    step?

-   **Action** - Does the controller/pilot have to perform a manual
    action or say something during the task step?

**Internal Error Modes** describe how the controller's/pilot's
performance failed to achieve the desired result. For instance, Internal
Error Modes within the 'Perception' error domain include 'mishear',
'mis-see', 'no detection (visual)' and 'no detection (auditory)'. One or
more Internal Error Mode is used for each error identified in the
report. **Internal Error Mechanisms** describe in greater depth the
psychological underpinnings of an Internal Error Mode and can better
enable the consideration of measures to reduce or mitigate errors.
Example error mechanisms within the 'Perception' domain include
'expectation', 'confusion', 'discrimination failure', 'perceptual
overload' and 'distraction/preoccupation'. Internal Error Mechanisms
would normally be identified only for critical errors (e.g. errors with
low Recovery Success Likelihood). The TRACEr-lite error modes and
mechanisms are shown in Table 2.

[Table 2:]{.underline} TRACEr-lite Internal Error (Modes and Mechanisms)
taxonomy

  -----------------------------------------------------------------------
  **Error Mode**                      **Error Mechanism**
  ----------------------------------- -----------------------------------
  ***Perception***                    

  Mishear                             Expectation

  Mis-see                             Confusion

  No detection (auditory)             Discrimination failure

  No detection (visual)               Perceptual overload

                                      Distraction / Preoccupation

  ***Memory***                        

  Forget action                       Confusion

  Forget information                  Memory overload

  Misrecall information               Insufficient learning

                                      Distraction / Preoccupation

  ***Decision Making***               

  Misprojection                       Misinterpretation

  Poor decision or poor plan          Failure to consider side- or
                                      long-term effects

  Late decision or late plan          Mind set / Assumption

  No decision or no plan              Knowledge problem

                                      Decision overload

  ***Action***                        

  Selection error                     Variability

  Unclear information                 Confusion

  Incorrect information               Intrusion

                                      Distraction / Preoccupation

                                      Other slip
  -----------------------------------------------------------------------

Following the analysis of Internal Errors, **Initial Consequences** are
determined by a process of analysis. Consequences are stated as free
text, and are normally restricted to the more immediate and likely
effects and consequences. Error likelihood and severity are not rated as
part of the standard TRACEr-lite approach, but can be rated by a team of
individuals if appropriate, for instance using data (e.g.
simulation-derived), or expert judgement.

The next in the TRACEr-lite process involves considering **Recovery**
from the error. This may involve stating a future step in the task
analysis, or stating the 'detection means', i.e. cues from the work
context, e.g. RT readback, radar monitoring, other controller, etc. The
analyst may also at this point rate the 'Recovery Success Likelihood
(RSL)'. This is a subjectively rated likelihood of recovering the task
successfully without adverse consequences, assisted by the use of an
anchored rating scale contained within TRACEr-lite.

On the basis of this analysis, **Comments** or **Questions** may be made
or **Recommendations** may be proposed. However, rather than propose
recommendations in a reactive fashion, these are normally proposed
during the synthesis of the data to address common themes within a
manageable set of recommendations or requirements. The process above is
illustrated in Figure 3. An example of TRACEr-lite analysis output is
shown in Table 3.

![](media/image3.wmf)[Figure 3:]{.underline} Process of Using
TRACEr-lite

# 

[Table 3:]{.underline} Extract of TRACEr-lite analysis worksheet for
Co-space[^4].

**Study Method**

Following initial scoping interviews and high-level task definition for
each project, the HAZOP and TRACEr analyses were conducted separately
and led by different analysts. The time permitted for each analysis was
roughly equivalent - about 10 person-days for each of the three
projects. In order to maintain objectivity, the results of each analysis
were only pooled when each analysis was complete. The general approach
is described below.

***HAZOP Approach***

The HAZOP studies for the three test projects proceeded according to the
methodology described previously. For each project, the HAZOP considered
one or two high level tasks taken from a task analysis (see Table 4).
The HAZOP considered both controller- and pilot-related errors, since
all were judged relevant and useful to consider. Due to restrictions in
the project team's availability, each HAZOP was conducted over one day,
and was attended by three project team members (e.g. operational
specialists, designers, human factors specialists). The sessions were
facilitated and recorded by independent DNV safety engineers or DNV
human factors specialists.

***TRACEr-lite Approach***

The TRACEr-lite analysis began with the development of a Hierarchical
Task Analysis (HTA) for each project (see Table 4). Following initial
consultations, various documents were reviewed, including procedures,
operational concept documents, human-machine interaction protocols and
diagrams, operational scenario documents, and simulation briefing
documents. For each project, the task steps were represented in detail
in an HTA, to the 'keystroke' level where necessary. The draft HTAs were
presented to each project team member, who helped to shape and revise
the HTAs until agreed versions were formed. The TRACEr-lite analyses
were conducted on the HTAs according to the methodology described
previously.

[Table 4:]{.underline} Tasks represented in the Hierarchical Task
Analysis, and indication of tasks further analysed by HAZOP (H) and
TRACEr-lite (T)

  -----------------------------------------------------------------------
  **Co-space HTA Top      **TBS HTA Top Level     **CORA 2 HTA Top Level
  Level Tasks**           Tasks**                 Tasks**
  ----------------------- ----------------------- -----------------------
  **1. Conduct task of    **1. Control terminal   **1. Resolve conflict
  Extended TMA controller and approach air        situation using CORA
  using Co-space**        traffic using TBS**     2**

  *Plan: Do 1.1 at start  *Plan: Do 1.1 at start  *Plan: Do 1.1 to 1.5 in
  of shift. Do 1.2, 1.3   of shift, then do 1.2.  order. Do 1.6
  and 1.4 as appropriate. Do 1.3 to 1.9 as        throughout as
  For delegation aircraft appropriate depending   appropriate.*
  do 1.5, 1.6 and 1.7 in  on controller position. 
  order. Do 1.8 near      Do 1.10 at end of       
  sector boundary. Do 1.9 shift.*                 
  at end of shift.*                               

  1.1 Take over from      1.1 Take over from      1.1 Detect conflict
  off-going controller    off-going controller    situation (T)
  (T)                     (T)                     

  1.2 Receive aircraft    1.2 Receive aircraft    1.2 Prioritise conflict
  (T)                     (T)                     situation(s) (T)

  1.3 Maintain traffic    1.3 Maintain traffic    1.3 Analyse focussed
  separation within       separation within       situation (TH)
  sector (T)              sector (T)              

  1.4 Form sequence plan  1.4 Hold aircraft (T)   1.4 Act on focussed
  / Follow sequence                               situation (T)
  formed by AMAN (T)                              

  1.5 Conduct             1.5 Sequence aircraft / 1.5 Check resolution
  identification Phase    Follow AMAN sequence    progress (T)
  (TH)                    (TH)                    

  1.6 Issue delegation    1.6 Turn aircraft onto  1.6 Monitor situation
  instruction (TH)        base leg (T)            (T)

  1.7 End delegation (T)  1.7 Turn aircraft onto  
                          intercept ILS (T)       

  1.8 Transfer to next    1.8 Establish aircraft  
  sector (T)              on ILS (T)              

  1.9 Handover control to 1.9 Transfer to next    
  relief controller (T)   sector / tower /        
                          controller (T)          

                          1.10 Handover control   
                          to relief controller    
                          (T)                     
  -----------------------------------------------------------------------

**Comparison of HAZOP and TRACEr-lite Findings**

The comparisons presented some difficulties, which need to be borne in
mind:

-   TRACEr-lite analysed more tasks than HAZOP. For the three projects,
    TRACEr considered between six and 10 of the tasks represented in the
    HTA (See Table 4), while HAZOP considered one or two tasks for each
    project. This was because HAZOP is time- and resource-intensive, and
    only a one-day meeting was possible for each HAZOP.

-   HAZOP considered both pilot and controller errors, as well as a
    limited number of information/equipment problems relating to the HMI
    for example. TRACEr-lite considered only controller errors. This was
    because the HTAs concentrated only on controller tasks.

-   The HAZOP and TRACEr-lite studied analysed errors at different
    levels. TRACEr-lite analysed errors at a more detailed level than
    HAZOP.

Since HAZOP analysed fewer tasks, it is only possible to compare HAZOP
and TRACEr-lite findings for those tasks analysed by HAZOP. A comparison
was performed for each study of errors and issues identified by HAZOP
and TRACEr-lite (jointly and separately). An example extract of the
comparison for Co-space is shown in Table 5.

[Table 5:]{.underline} Errors and issues identified by HAZOP and
TRACEr-lite for Co-Space Task 1.5 'Conduct Identification Phase'.

+-------------------------------------------------------+------+------+
| **Error/Issue**                                       | **In | **In |
|                                                       | -sco | -sco |
|                                                       | pe** | pe** |
|                                                       |      |      |
|                                                       | **   | **TR |
|                                                       | HAZO | ACEr |
|                                                       | P?** | -lit |
|                                                       |      | e?** |
+=======================================================+======+======+
| **Issues identified by both TRACEr-lite and HAZOP**   |      |      |
+-------------------------------------------------------+------+------+
| Controller instructs pilot to select wrong target     | Yes  | Yes  |
+-------------------------------------------------------+------+------+
| Controller goes straight to delegation without        | Yes  | Yes  |
| confirmation from pilot (fails to detect / query      |      |      |
| missing readback or target identification)            |      |      |
+-------------------------------------------------------+------+------+
| Controller goes straight to instruction of delegation | Yes  | Yes  |
| omitting to identify target (fails to instruct pilot  |      |      |
| to select target)                                     |      |      |
+-------------------------------------------------------+------+------+
| Controller issues correct target selection            | Yes  | Yes  |
| instruction to wrong aircraft                         |      |      |
+-------------------------------------------------------+------+------+
| Controller gives wrong target to correct aircraft     | Yes  | Yes  |
+-------------------------------------------------------+------+------+
| *Etc.*                                                |      |      |
+-------------------------------------------------------+------+------+
| **Issues identified by HAZOP only**                   |      |      |
+-------------------------------------------------------+------+------+
| Controller may give other instruction along with      | Yes  | Yes  |
| delegation instruction                                |      |      |
+-------------------------------------------------------+------+------+
| Pilot doesn\'t confirm he has heard the target \*     | Yes  | No   |
+-------------------------------------------------------+------+------+
| Pilot doesn\'t read back the target reference \*      | Yes  | No   |
+-------------------------------------------------------+------+------+
| Pilot selects wrong target \*                         | Yes  | No   |
+-------------------------------------------------------+------+------+
| Pilot identifies correct target and goes straight to  | Yes  | No   |
| next action (anticipates controllers instruction)     |      |      |
| \*\*                                                  |      |      |
+-------------------------------------------------------+------+------+
| *Etc.*                                                |      |      |
+-------------------------------------------------------+------+------+
| **Issues identified by TRACEr-lite only**             |      |      |
+-------------------------------------------------------+------+------+
| Controller fails to detect / query erroneous readback | Yes  | Yes  |
+-------------------------------------------------------+------+------+
| Controller fails to select aircraft on radar          | Yes  | Yes  |
+-------------------------------------------------------+------+------+
| Controller selects unintended aircraft (not part of   | Yes  | Yes  |
| delegation) when selecting aircraft on radar          |      |      |
+-------------------------------------------------------+------+------+
| Controller fails to detect / query failure to         | Yes  | Yes  |
| position                                              |      |      |
+-------------------------------------------------------+------+------+
| Controller fails to query spurious position           | Yes  | Yes  |
+-------------------------------------------------------+------+------+
| *Etc.*                                                |      |      |
+-------------------------------------------------------+------+------+

\* Failure(s) of associated ATCO hearback identified

\*\* Failure(s) of associated ATCO monitoring identified

In order to facilitate the discussion and comparison of HAZOP and
TRACEr-lite as applied to EUROCONTROL projects, a range of criteria for
the evaluation of human error analysis methods is used (based on
Shorrock, 2002a).

***Comprehensiveness***

HAZOP and TRACEr-lite performed differently on the issue of
comprehensiveness. Both HAZOP and TRACEr-lite identified critical errors
that could occur for all projects. HAZOP identified both controller and
pilot errors when considering each task, and also examined potential
information problems. However, in the time available, HAZOP could only
examine two tasks for Co-space, and one task for TBS and CORA 2.
TRACEr-lite examined the whole range of controller tasks for each
project, and provided a highly comprehensive and detailed 'register' of
potential errors. However, TRACEr-lite examined only controller errors
(because controller tasks were the focus of the HTA).

TRACEr-lite tended to analyse errors in a more detailed and systematic
fashion than HAZOP. This is largely because TRACEr-lite uses a detailed
HTA, while HAZOP is not normally conducted at such a fine level of task
modelling, and instead used the high level tasks in Table 4 along with a
brief description of what was involved. However, one of the potential
dangers of detailed TRACEr-lite analysis is in getting 'lost in detail'
and failing to identify more fundamental issues. While this did not seem
to occur in the current analysis, it is a potential problem to be aware
of.

The analysis showed that TRACEr-lite identified approximately 91% of the
errors/issues identified by HAZOP that were within the scope of the
TRACEr analysis (i.e. controller errors). For issues and errors
identified by HAZOP that were outside the scope of the TRACEr-lite
analysis (such as pilot errors and general performance conditions),
TRACEr-lite was able to predict related errors, such as failures in ATCO
responses to pilot errors for approximately 36% of the issues. HAZOP
identified approximately 42% of the errors identified by TRACEr-lite for
Co-space, TBS and CORA 2. However, as previously stated, HAZOP
identified other issues such as pilot errors and controller performance
conditions that were not analysed by TRACEr-lite.

HAZOP and TRACEr-lite identified a similar range of consequences for the
three studies, though the TRACEr-lite consequences tended to be limited
to the more immediate, short-term consequences while HAZOP sometimes
focussed only on the 'bottom line' consequences. HAZOP, however,
identified significantly more safeguards than TRACEr-lite, reflecting
the experience of the HAZOP team members. This may suggest that TRACEr
can produce more 'pessimistic' analyses than HAZOP.

The Recovery Success Likelihood (RSL) rating used by TRACEr-lite was
useful in filtering the errors predicted, and the ratings were moderated
by members of the Co-space team to help ensure that they were realistic.
Only around 12% of the total number of RSL ratings for all errors
predicted was modified in this review, thus suggesting that the initial
RSL ratings were, on the whole, realistic. The RSL concept may be useful
to consider incorporating into a group-based technique such as HAZOP. It
would also have been useful to rate error likelihood within the
TRACEr-lite analysis using expert judgement (see Kirwan, 1994).

***Life cycle applicability***

Both HAZOP and TRACEr-lite demonstrated that they could be used
throughout the formative and summative phases of system design
lifecycle. TBS was in an early conceptual stage, while CORA 2 was in the
design stages, and Co-space was in the simulation phases. However,
TRACEr-lite's need for a more detailed task analysis would prompt the
conclusion that HAZOP is more suited to projects in the conceptual and
pre-design stages of development. A useful approach would be to conduct
a preliminary HAZOP initially, to focus the analysis, identify the
primary sub-tasks of interest, and identify the fundamental errors, and
then follow this up with a detailed TRACEr-lite analysis to help ensure
that errors are captured at a detailed and comprehensive level.

***Theoretical validity***

TRACEr-lite is based on a model of human performance, with a
theoretically plausible internal structure (Wickens, 1992). While HAZOP
does not have this foundation, the guidewords are valid human
performance outcomes.

***Contextual validity***

HAZOP and TRACEr-lite account for context in different ways. HAZOP
primarily uses the expertise of the HAZOP team. This is an established
method of ensuring that such analyses are contextually relevant.
TRACEr-lite uses project personnel and documentation to construct and
review the Context Statement and Task Analysis. The project personnel
would also then review the TRACEr-lite analysis. This is an important
post-analysis step, which would involve face-to-face discussions between
the TRACEr analyst and project personnel, perhaps also considering error
likelihood and severity, but was not possible during this study due to
time and resource constraints[^5]. Some of the TRACEr-lite-predicted
errors may, therefore, appear somewhat 'naïve'. However, equally, there
are positive arguments in favour of this! Project team members who are
very 'close' to the project' may consider certain errors 'incredible' in
a HAZOP analysis, and prefer not to propose or record them. From a
TRACEr-lite point of view, on the other hand, such errors would be
logical possibilities. Overall during this study, considering the errors
and issues predicted, consequences, safeguards and recommendations,
HAZOP outperforms TRACEr-lite on this criterion.

***Flexibility***

Both HAZOP and TRACEr-lite are flexible in that they allow different
levels of detail in the analysis. HAZOP has the advantage that early in
the concept development/selection process, a preliminary HAZOP can be
performed, using the creative brainstorming and knowledge of the group.
The full HAZOP technique can then be performed later. TRACEr-lite,
however, requires a more developed task analysis, since the individual
analyst is not qualified to perform such a 'brainstorming' approach.
TRACEr-lite does, however, provide flexibility in the use of Internal
Error Modes and Mechanisms -- the use of Mechanisms in the analysis can
be omitted or used only for critical errors. TRACEr-lite can also be
used in a small group-based format, using the Error Domains Perception,
Memory, Decision, and Action as prompts. Indeed, this method has
previously been incorporated into the Human HAZOP method to better
account for cognitive aspects of errors in other HAZOP studies.

***Usefulness***

Both HAZOP and TRACEr-lite helped to produce error reduction or
mitigation measures. The HAZOP analysis produced more traditional
recommendations while TRACEr-lite tended to produce 'performance
requirements', which could be fulfilled in a number of ways. The HAZOP
method, however, was clearly much more productive in this respect. This
productivity can be traced to the contribution of the project teams. The
project team could, in future, participate similarly in generating
recommendations based on TRACEr-lite findings.

The TBS project team found the HAZOP a very useful exercise even at the
very early stages of the project, and felt that the HTA helped to
identify tasks that would be affected by TBS, and where tasks still
needed to be specified or considered.

***Resource efficiency (training)***

This criterion has not been formally tested, but training to become a
HAZOP facilitator would normally involve three to four days training,
and prior experience as a recorder. No training is required to act as a
participant but awareness or familiarisation training is very useful to
speed up the process. HAZOP's training requirement is largely associated
with the ability to facilitate a group-based process. The role of the
HAZOP leader or TRACEr-lite analyst may not suit every individual, and
the HAZOP leader role requires different skills to the TRACEr-lite
analyst role. Training in the use of the TRACEr-lite technique would
normally involve 1-2 days, plus a further 1-2 days to learn how to use
Hierarchical Task Analysis. Hence, the training demand for each
technique is quite similar.

***Resource efficiency (usage)***

Overall, TRACEr-lite was the more resource efficient technique,
analysing more tasks in a similar amount of time. When HAZOP and
TRACEr-lite are compared for an equivalent analysis of all controller
high-level tasks in the CORA HTA, it is estimated that HAZOP would
require a total of 26.5 person days, while TRACEr-lite would require 16
person days -- about 60% of the HAZOP time. These figures cover
background familiarisation, task modelling/analysis, analysis, data
formatting/tidying, reporting, as well as review and revision at each
stage.

#### Usability

Both HAZOP and TRACEr-lite are usable techniques. HAZOP has stood the
test of time, and does not demand complex analysis from participants.
However, the process can prove frustrating, particularly where several
unbroken days of analysis are performed. In this respect, it is wise to
break up sessions that occur over several days (though care should be
taken to ensure that a 'flow' is maintained in the study). TRACEr-lite,
similarly, can be frustrating to the analyst due to the repetitive
nature of analysis.

***Auditability***

Again, both HAZOP and TRACEr-lite provided a fully auditable process,
with worksheets demonstrating the reasoning behind the analysis. HAZOP,
additionally, visually projected each worksheet during the session so
that all of the participants could verify the findings 'on-line'.

**Conclusions**

Overall, both Human HAZOP and TRACEr-lite proved useful methods to
support designing for safety. It would be useful to use a hybrid
approach by performing a preliminary Human HAZOP to identify the core
tasks and critical, high-level errors followed by a detailed TRACEr-lite
analysis. This preliminary Human HAZOP could also identify the relevant
safeguards and consequences for use in the detailed TRACEr analysis, to
improve the contextual relevance of the technique. Additionally, HAZOP
can be modified to assess Human-Machine Interfaces (e.g. Kennedy, *et
al*., 2000) when interface design mock-ups are available.

These two approaches are variants of already successful approaches in
other industries. Therefore this chapter shows the adaptability and
usefulness of these approaches in another new field, that of Air Traffic
Management. The emphasis has been on enriching design, making it more
safe, rather than independent assessment of design, and it is intended
to have these approaches seen as the province of 'safe designers', as
well as safety assessors.

**Acknowledgements**

The authors gratefully acknowledge the contribution to the study of the
following people: Roisin Johnson (DNV Co-space HAZOP facilitator), Emma
Noble and Joanne Ramsay (DNV HAZOP recorders); Isabelle Grimaud,
Laurence Rognin and Karim Zeghal (EUROCONTROL Co-space team); Peter
Crick and Antoine Vidal (EUROCONTROL TBS team); Mary Flynn, Sophie
Dusire and Nicholas Dardenne (EUROCONTROL CORA 2 team). The views
expressed in this paper are those of the authors; they are not
necessarily those of EUROCONTROL or DNV.

**References**

Billings, C.E. (1988). Toward human centred automation. In S.D. Norman
and H.W. Orlady (Eds.), *Flight Deck Automation: Promises and
Realities*. Moffet Field, CA: NASA-Ames Research Center, pp. 167-190.

Embrey, D.E. (1986) SHERPA: A systematic human error reduction and
prediction approach. *Paper presented at the International Topical
Meeting on Advances in Human Factors in Nuclear Power System*s,
Knoxville, Tennessee.

Isaac, A., Shorrock, S.T. and Kirwan, B. (2002). Human error in European
air traffic management: the HERA Project. *Reliability Engineering and
System Safety*, **75 (2)**, 257-272.

Hollnagel, E. (1998). *Cognitive Reliability and Error Analysis Method:
CREAM*. Oxford, UK: Elsevier.

Kennedy, R., Jones, H., Shorrock, S. and Kirwan, B. (2000). A HAZOP
analysis of a future ATM system. In P.T. McCabe, M.A. Hanson and S.A.
Robertson, *Contemporary Ergonomics 2000*. London: Taylor and Francis.

Kletz, T.A. (1988). *What Went Wrong?: Case Histories of Process Plant
Disasters*. US: Gulf Publishing.

Kirwan, B. (1994). *A Practical Guide To Human Reliability Assessment*.
London: Taylor and Francis.

Leroux, M. (2000). Cognitive aspects and automation. In N.B. Sarter and
R. Amalberti (Eds.), *Cognitive Engineering in the Aviation Domain*.
Laurence Erlbaum Associates: London.

Sarter, N. and Woods, D.D. (1995b). *Strong, Silent, and
Out-of-the-Loop: Properties of Advanced (Cockpit) Automation and their
impact on Human-Automation Interaction*. Technical Report CSEL 9-TR-01.
Columbus, OH: Cognitive Systems Engineering Laboratory, Ohio State
University.

Shepherd, A. (2001). *Hierarchical Task Analysis.* London: Taylor and
Francis.

Shorrock, S.T. (2002a). The two-fold path to human error analysis:
TRACEr-lite retrospection and prediction. *Safety Systems* (Newsletter
of the Safety-Critical Systems Club), Vol. 11 No. 3, September 2002.

Shorrock, S.T. (2002b). Error classification for safety management:
finding the right approach. Proceedings of the *Workshop on the
Investigation and Reporting of Incidents and Accidents*, 17th to 20th
July 2002, The Senate Room, University of Glasgow.

Shorrock, S.T. and Kirwan, B. (1999). TRACEr: a technique for the
retrospective analysis of cognitive errors in ATM. In D. Harris (Ed.)
*Engineering Psychology and Cognitive Ergonomics: Volume Three -
Transportation Systems, Medical Ergonomics and Training.* Aldershot, UK:
Ashgate.

Shorrock, S.T. and Kirwan, B. (2002). The development and application of
a human error identification tool for air traffic control. *Applied
Ergonomics*, **33**, 319-336.

Shorrock, S.T., Kirwan, B., MacKendrick, H. and Kennedy, R. (2001).
Assessing human error in Air Traffic Management systems design:
methodological issues. *Le Travail Humain*, **64 (3)**, 269-289.

Wickens, C.D. (1992). *Engineering Psychology and Human Performance*.
2^nd^ Edition. New York: Harper Collins.

Wiener, E.L. (1985). Cockpit automation: in need of a philosophy.
*Proceedings of the 1985 Behavioural Engineering Conference*,
Warrendale, PA, 1985, pp. 369-375.

Wiener, E.L. (1988). Cockpit automation. In E.L. Wiener and D.C. Nagel
(Eds.), *Human Factors in Aviation*. San Diego: Academic Press.

Wiener, E.L. and Curry, R.E. (1980). Flight-deck automation: promises
and problems. *Ergonomics*, **23**, 995-1011.

[^1]: See
    [www.eurocontrol.fr/projects/freer/publications.htm](http://www.eurocontrol.fr/projects/freer/publications.htm)
    for a list of relevant publications.

[^2]: TRACEr was adapted for retrospective use in Europe in the
    EUROCONTROL 'HERA' project - human error in ATM (see Isaac, et al.,
    2002), which, in collaboration with the Federal Aviation
    Administration (FAA), was further developed in a joint project
    resulting in the HERA-Janus technique.

[^3]: In one application, 92% of the errors that were observed during
    extensive observations of simulations sessions were previously
    predicted by TRACEr. Those errors that were not predicted were not
    represented in the associated task analysis.

[^4]: 'Comments' column not shown in this extract.

[^5]: The TRACEr lite synthesis was reviewed by the Co-Space project
    team.