A Formalized Approach 

for the Collection of HRA 

Data from Nuclear Power 

Plant Simulators

Office of Nuclear Regulatory Research

NUREG/CR-7163

INL/EXT-12-26327
NRC Reference Material 

As of November 1999, you may electronically access 

NUREG-series publications and other NRC records at 

NRC’s Public Electronic Reading Room at 

http://www.nrc.gov/reading-rm.html. Publicly released 

records include, to name a few, NUREG-series 

publications; Federal Register notices; applicant, 

licensee, and vendor documents and correspondence; 

NRC correspondence and internal memoranda; bulletins 

and information notices; inspection and investigative 

reports; licensee event reports; and Commission papers 

and their attachments. 

NRC publications in the NUREG series, NRC 

regulations, and Title 10, “Energy,” in the Code of

Federal Regulations may also be purchased from one 

of these two sources. 

1. The Superintendent of Documents

 U.S. Government Printing Office Mail Stop SSOP 

Washington, DC 20402–0001

Internet: bookstore.gpo.gov

Telephone: 202-512-1800

Fax: 202-512-2250 

2. The National Technical Information Service 

Springfield, VA 22161–0002

www.ntis.gov

1–800–553–6847 or, locally, 703–605–6000 

A single copy of each NRC draft report for comment is 

available free, to the extent of supply, upon written 

request as follows:

Address: U.S. Nuclear Regulatory Commission

 Office of Administration 

Publications Branch

 Washington, DC 20555-0001

E-mail: DISTRIBUTION.RESOURCE@NRC.GOV

Facsimile: 301–415–2289 

Some publications in the NUREG series that are 

posted at NRC’s Web site address 

http://www.nrc.gov/reading-rm/doc-collections/nuregs 

are updated periodically and may differ from the last 

printed version. Although references to material found on 

a Web site bear the date the material was accessed, the 

material available on the date cited may subsequently be 

removed from the site.

Non-NRC Reference Material 

Documents available from public and special technical 

libraries include all open literature items, such as books, 

journal articles, transactions, Federal Register notices, 

Federal and State legislation, and congressional reports. 

Such documents as theses, dissertations, foreign reports 

and translations, and non-NRC conference proceedings 

may be purchased from their sponsoring organization. 

Copies of industry codes and standards used in a 

substantive manner in the NRC regulatory process are 

maintained at—

The NRC Technical Library

Two White Flint North

11545 Rockville Pike

Rockville, MD 20852–2738 

These standards are available in the library for reference 

use by the public. Codes and standards are usually 

copyrighted and may be purchased from the originating 

organization or, if they are American National Standards, 

from—

American National Standards Institute

11 West 42nd Street

New York, NY 10036–8002

www.ansi.org

212–642–4900 

AVAILABILITY OF REFERENCE MATERIALS

IN NRC PUBLICATIONS

Legally binding regulatory requirements are stated 

only in laws; NRC regulations; licenses, including 

technical specifications; or orders, not in NUREGseries publications. The views expressed in 

contractor-prepared publications in this series are 

not necessarily those of the NRC.

The NUREG series comprises (1) technical and 

administrative reports and books prepared by the 

staff (NUREG–XXXX) or agency contractors 

(NUREG/CR–XXXX), (2) proceedings of 

conferences (NUREG/CP–XXXX), (3) reports 

resulting from international agreements 

(NUREG/IA–XXXX), (4) brochures (NUREG/BR–

XXXX), and (5) compilations of legal decisions and 

orders of the Commission and Atomic and Safety 

Licensing Boards and of Directors’ decisions

under Section 2.206 of NRC’s regulations 

(NUREG–0750).

DISCLAIMER: This report was prepared as an account of work sponsored by an agency of the U.S. Government. 

Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or 

assumes any legal liability or responsibility for any third party’s use, or the results of such use, of any information, 

apparatus, product, or process disclosed in this publication, or represents that its use by such third party would not 

infringe privately owned rights.
A Formalized Approach 

for the Collection of HRA 

Data from Nuclear Power 

Plant Simulators

Manuscript Completed: September 2013 

Date Published: May 2014

Prepared by: 

B. Hallbert, T. Morgan, J. Hugo, J. Oxstrand, J. J. Persensky

Idaho National Laboratory

2525 Fremont Avenue

Idaho Falls, ID 83415

Erasmia Lois, NRC Project Manager

Office of Nuclear Regulatory Research

 

NUREG/CR-7163

INL/EXT-12-26327
NOTICE

This report was prepared by the Idaho National Laboratory as an account of work sponsored by an agency of 

the U.S. Government. Neither the U.S. Government nor any agency thereof, nor any of their employees,

makes any warranty, express or implied, or assumes any legal liability or responsibility for any third party’s

use, or the results of such use, of any information, apparatus, product, or process disclosed herein, or 

represents that its use by such third party would not infringe privately owned rights. The views expressed 

herein are not necessarily those of the U.S. Nuclear Regulatory Commission.

Prepared for the U.S. Nuclear Regulatory Agency Office of Regulatory Research and 

for the U.S. Department of Energy

Under DOE Idaho Operations Office Contract DE-AC07-05ID14517
iii

ABSTRACT

Data from nuclear power plant (NPP) simulator environments reflect important aspects of human 

performance and potentially the causes of human error. However, the information collected is

constrained by the uniqueness of each particular plant and design of each particular simulator 

study. Thus, the ability to apply the findings of a given simulator study to other studies of human 

performance at NPPs is often limited. A well-defined approach to experimental

design, a common language or measurement technique for describing and classifying errors 

and human actions, and an associated theoretical underpinning for the design and resulting 

statistical analyses are required to allow extrapolation of the results of simulator studies across 

plant designs, crew make-ups, and scenarios.

The purpose of this report is: to propose an approach for collecting human performance data 

from NPP simulators and employing the reliability engineering (RE) concept of limit state, to 

describe the process for collecting data, and to present illustrative examples of data analyses. 

This approach borrows from techniques applied in traditional RE while using experimental 

techniques derived from the behavioral sciences.

v

CONTENTS

ABSTRACT ...................................................................................................................................... iii

CONTENTS ....................................................................................................................................... v

LIST OF FIGURES.......................................................................................................................... vii

LIST OF TABLES............................................................................................................................ vii

EXECUTIVE SUMMARY ........................................................................................................ix

ACKNOWLEDGMENTS.................................................................................................................xiii

ABBREVIATIONS........................................................................................................................... xv

1. INTRODUCTION ..................................................................................................................... 1

1.1 Background ...................................................................................................................... 1

1.2 Purpose ............................................................................................................................ 3

1.3 A Framework for Human Reliability Analysis Data Collection ......................................... 4

1.4 Document Organization ................................................................................................... 5

2. PRINCIPLES FOR COLLECTING HRA DATA FROM NUCLEAR POWER PLANT 

SIMULATORS WITH AN EMPHASIS ON THE LIMIT STATE CONCEPT ........................... 7

2.1 Introduction....................................................................................................................... 7

2.2 Principles of the Data Collection Approach ..................................................................... 7

2.2.1 Performance Measures Should Relate to Success and Failure as Defined 

in the PRA................................................................................................................. 7

2.2.2 Simulator Contexts Should be Representative of PRA Events................................ 8

2.2.3 Data Collection Should Include the Opportunity to Observe Human Failure 

 Events of Interest...................................................................................................... 8

2.2.4 Data Collection Should Include Objective Human Performance Measures of

 Success and Failure ................................................................................................. 9

2.2.5 Data Collected Should Describe Relevant Aspects of Performance and the

 Factors that Affect that Performance...................................................................... 10

2.3 Categories of Information............................................................................................... 10

3. COLLECTING DATA FROM TRAINING SIMULATORS FOR HRA ................................... 13

4. PREPARATION FOR DATA COLLECTION ........................................................................ 19

5. DATA COLLECTION ............................................................................................................ 27

6. DATA ANALYSIS.................................................................................................................. 33

7. REPORTING.......................................................................................................................... 43

8. SUMMARY AND CONCLUSIONS........................................................................................ 45

9. REFERENCES ...................................................................................................................... 51

APPENDIX A – SCENARIO DATA ANALYSIS GRAPHS ....................................................... 55

APPENDIX B – SIMULATOR SCENARIO DESCRIPTIONS.................................................... 67

APPENDIX C – PERFORMANCE CRITERIA ........................................................................... 73

APPENDIX D – PERFORMANCE DRIVERS DATA COLLECTION ........................................ 77

APPENDIX E – EXAMPLE OF LOG SHEET ............................................................................ 79

APPENDIX F - EXAMPLE OF POST-SCENARIO DEBRIEFING FORM................................. 83

vii

FIGURES

Figure 1 -- Recommended human reliability analysis simulator data collection process...............18

Figure 2 -- Performance criteria and limit state for feed and bleed action..................................21

Figure 3 – Example summary of crew response characterization plot......................................36

Figure 4 -- Example of numerical ratings summary of performance drivers...............................38

Figure 5 – Distribution of percentage of available time used by crews to initiate feed and bleed....39

Figure 6 – Crew response times from loss of Component Cooling Water to tripping the 

Reactor Coolant Pumps...................................................................................40

TABLES

Table 1 – Simulator data collection procedure....................................................................13

Table 2 -- Typical debriefing questions..............................................................................31

ix

EXECUTIVE SUMMARY

Data from nuclear power plant (NPP) simulator environments reflect important aspects of human 

performance and potentially the causes of human error. However, the information collected is

constrained by the uniqueness of each particular plant and design of each particular simulator 

study. Thus, the ability to apply the findings of a given simulator study to other studies of human 

performance at NPPs is often limited. A well-defined approach to experimental

design, a common language or measurement technique for describing and classifying errors 

and human actions, and an associated theoretical underpinning for the design and resulting 

statistical analyses are required to allow extrapolation of the results of simulator studies across 

plant designs, crew make-ups, and scenarios.

The purpose of this report is: to propose an approach for collecting human performance data 

from NPP simulators and employing the reliability engineering (RE) concept of limit state, to 

describe the process for collecting data, and to present illustrative examples of data analyses. 

This approach borrows from techniques applied in traditional RE while using experimental 

techniques derived from the behavioral sciences.

The ultimate aim of the proposed approach is to achieve greater consensus, consistency and 

convergence of human reliability analysis (HRA) methods by stimulating simulator data 

exchange, greater communication, and review within the HRA research and practitioner 

community. The overall objective of this work is in accordance with SRM-M090204B.

HRA exists at a nexus between three distinct disciplines: (1) the behavioral sciences (i.e., human

factors, ergonomics, cognitive sciences, and psychology); (2) reliability engineering; and (3) 

probabilistic risk assessment (PRA). Each discipline has a tradition of scientific development and 

theoretical underpinnings that reflect the formal requirements for, and use of information (e.g., 

data, knowledge, etc.) to address issues to which the discipline is applied. This tradition includes

the use of formal experimental design, protocols for data collection, methods for the analysis and

interpretation of data, development of models to integrate results, and

development of codes and standards.

This presents challenges to the HRA discipline because, in several important ways, the 

methodological approaches of these disciplines are quite different. In principle, RE analysis 

assesses the performance of discrete system elements, having well-characterized governing 

dynamics (e.g., mechanical, electrical, chemical, etc.), interacting with other discrete elements 

under well-described conditions. The interactions among components in a mechanical system 

are straightforward to characterize. Mechanical system interactions are observable, 

measurable, quantifiable, and predictable.

Human performance is difficult to subject to the same form of reliability analysis that is applied

to electro-mechanical systems. The body of behavioral sciences literature and knowledge is not 

as easy to employ in studies of human reliability analysis as engineered system characteristics 

are in the analysis of systems reliability. Human capabilities are not as easily understood as the 

capabilities of electro-mechanical devices. Because of the complexity of the human systems, our 

state of knowledge of human cognition and behavior is not as mature as it is for mechanical 

systems. Therefore, developing functional models, as well as testing and refining theories,

about these human related processes is difficult.
x

One of the hallmark characteristics of the RE discipline is the ability to “test to failure” by

creating system failure conditions which can be observed and to collect data that can be used to 

establish failure rates for various failure types. Validated models using such data can describe 

the physics of failure of electro-mechanical devices or structural systems, including nuclear 

energy systems.

Studies of humans that enable observations of failure are needed to develop and validate 

models for human reliability. Failures in human performance that would be of interest to HRA 

are unlikely to be observed. The HFEs modeled in a PRA, are events that result from unique 

combinations of failures (i.e., initiating events) and plant conditions and are very plant specific. 

Few studies of human failures designed to manifest these conditions in nuclear power plant 

control rooms have been conducted and reported such that the ability to draw general insights 

about failure rates and types is limited.

As postulated above, the ability to collect data on the reliability of human performance in 

complex environments is much more difficult than it is to collect data for many of electromechanical devices employed in nuclear energy production and other industrial systems. What is

needed are opportunities to collect data about human performance in which success and failure

are defined and measured and theories of human performance and human reliability that are

sufficiently mature to support HRA model and method development.

The approach proposed in this report could be used to provide a common language for human 

failures (limit state) and human performance within HRA studies that would allow data from one 

HRA context to be extrapolated to a new context.

Collecting simulator data to support HRA requires robust experimental design and process. 

This report describes five essential experimental design characteristics based on paradigms 

from the behavioral sciences that should be considered for robust experimental design when 

preparing to collect data for application in HRA. The five experimental design characteristics 

are:

• Performance Measures Should Relate to Success and Failure as Defined in the PRA

• Simulator Contexts should be Representative of PRA Events

• Data Collection Should Include the Opportunity to Observe Human Failure Events of

Interest

• Data Collection should include Objective Human Performance Measures of Success or

Failure

• Data Collected Should Describe Relevant Aspects of Performance and the Factors that

Effect that Performance

HRA methods consider conditions that may be found in NPP contexts to predict the kinds of 

errors (e.g., slips, lapses, or mistakes) and the likelihood of those errors that can occur. There is 

substantial experimental and experiential evidence about the many factors that may be “drivers” 

of human reliability. These include factors such as stress, workload, human-system interface 

technologies, training and experience, crew resource management, intra-team dynamics, and
xi

others. An analysis of crew performance data should permit a determination regarding success 

of the crew and to minimize the potential for subjectivity in evaluating crew performance.

The report describes in detail the specific process steps for performing an experiment following 

a robust design that focuses on collecting HRA relevant data from an NPP simulator running

risk-significant events. The process of conducting an HRA simulator data collection effort can be 

described in four main phases: preparation, data collection, data analysis, and reporting. These 

four phases are further defined by 19 sub-steps. Some of the sub-steps are discussed in detail. 

For example, in the preparation section, legal contractual mechanisms are discussed, as well as 

important items to consider when designing and validating the scenarios to be used in the data 

collection effort. This includes the determination of success and failure criteria and human 

performance measures. The data collection section presents different methods to be used for 

data collection and also describes how to structure the collection effort. In the data analysis 

section, examples of how to develop synopses and how to document deviations and errors are 

provided. There is also a description of how to analyze performance drivers in this section. This 

report discusses in detail the concept of limit state as a generic approach to define success and 

failure criteria so that the data can be used across HRA methods. Limit state from an RE and 

systems safety perspective denotes a region of performance that distinguishes between 

performance success and failure.

Two important approaches for analyzing the quality of crew response include (1) analyzing the 

distribution of crew performance on the criteria related to the limit state and (2) analyzing the 

variability in crew response relative to the limit state. The performance criteria for human actions 

are specific to the event sequence and scenario. Even though the same human actions are 

required in different event sequences, the specific criteria that define success for the human 

action is scenario specific. Identifying the criteria in each scenario context is the first step of 

establishing the measure of success and failure for the critical human action. Using observed 

crew performance during the simulator runs of the scenario and the defined threshold for 

performance, the data show the response of individual crews against the maximum available 

response range. This method of analysis produces a distribution of crew performance relative to 

the limit state and is an indicator of how near the crew was to failing to meet the performance 

criteria for the scenario.

The second analysis approach uses crew performance data to characterize the variability in 

crew response. Variability in crew performance reflects intercrew differences in task execution, 

procedure use, and decision making, all of which are interrelated. It also reflects uncertainty in 

the results of crew performance that may have implications for the reliability and consistency in 

crew response to the initiating event(s).

Performing these analyses results in normalization of the raw performance measurement in 

terms of the limit state. This means that the raw performance measures are adjusted using their 

relationship to the limit state on a notionally common scale. This permits insights to be drawn 

regarding three properties related to performance reliability:

• Whether the actions meet the success criteria for the defined HFE

• The amount of margin available between their performance and the limit state for the 

action as defined by the HFE

• Variability among crews in performing the action(s)
xii

The process of normalization whether using raw data or transformed data relates

observed performance to the system limit state and provides a reference that can be

used to show performance relative to success and failure criteria alone. There may be

many differences between success and failure criteria for accident sequence criteria

among different plants and plant types. Normalization and appropriate transformations

are ways that data collected from different facilities may be treated and analyzed to

permit comparison and potentially aggregation across data collection sites.
xiii

ACKNOWLEDGMENTS

The authors gratefully acknowledge the assistance of several individuals at the Nuclear 

Regulatory Commission, especially Dr. Amy D’Agostino, Dr. Julie Marble, and Dr. James 

Chang. The authors acknowledge the assistance of a number of individuals at the Idaho 

National Laboratory, the Organization for Economic Co-Operation and Development Halden 

Reactor Project, and the participating utility who have shared their information and insights 

regarding human reliability and human performance data collection. Finally, the authors thank 

the peer review members for their thorough and constructive contributions to our report. In 

particular, our thanks go to Professor Ali Mosleh (University of Maryland), Dr. Barry Kirwan 

(Eurocontrol), Dr. Vinh Dang (Paul Scherrer Institute), Dr. Wondea Jung (Korea Atomic Energy 

Research Institute), Mr. Andreas Bye (Halden Reactor Project) and Dr. Susan Cooper, Dr. Jing 

Xing and Dr. James Chang (USNRC).

xv

ABBREVIATIONS

AFW auxiliary feedwater

CCW component cooling water

CSNI Committee on the Safety of Nuclear Installations

EDG emergency diesel generator

EOP emergency operating procedures

ESF engineering safety feature

F&B feed and bleed gpm

gallons per minute

HFE human failure event

HHP head safety injection pump

HRA human reliability analysis

IDHEAS A Method of the Integrated Human Event Analysis System

LER license event report 

LOCA loss-of-coolant accident 

LOFW loss of feedwater

MFW main feedwater

MSIV main steam isolation valve 

NEA Nuclear Energy Agency 

NPP nuclear power plant

NRC Nuclear Regulatory Commission

OECD Organization for Economic Cooperation and Development

PORV power-operated relief valve 

PRA probabilistic risk assessment 

PSF performance shaping factor 

PZR pressurizer

RCP reactor coolant pump 

RCS reactor coolant system 

RE reliability engineering 

RO reactor operator

Rx reactor

S/G steam generator

SGTR steam generator tube rupture

SI safety injection

SM shift manager

SSC structure, system, or component

STA shift technical advisor 

US unit supervisor 

WGRISK Working Group on Risk

1

1. INTRODUCTION

1.1 Background and Technical Basis

Data from nuclear power plant (NPP) simulator environments reflect important aspects of human 

performance and potentially the reasons for and causes of human error. However, the 

information collected is constrained by the uniqueness of each particular plant and design of 

each particular simulator study. Thus, the ability to apply the findings of a given simulator study 

to other studies of human performance at NPPs is often limited. A well-defined approach to 

experimental design, a common language or measurement technique for describing and 

classifying errors and human actions, and an associated theoretical underpinning for the design 

and resulting statistical analyses are required to allow extrapolation of the results of simulator 

studies across plant designs, crew make-ups, and scenarios.

In addition to the NRC research efforts, the need for HRA data has been part of international 

cooperative activities sponsored by the Organization for Economic Co-Operation and 

Development (OECD), Nuclear Energy Agency (NEA), Committee on the Safety of Nuclear 

Installations (CSNI) which has been pursuing simulator studies at the Halden Reactor Project. 

(Bye et al. 2012, Forester et al. 2012, Hendrickson et al. 2012, & Marble et al. 2012).

Report NEA/CSNI/R(2008)9 proposes a standardized HRA data collection approach, focusing

on data generated in plant training simulators. Specifically, the report describes a pilot study that 

was conducted with the objective of establishing a framework for collection and exchange of 

human performance data from training and research simulators, including the establishment of 

good practices for performing these activities.

Since its inception, the research activity described in the current report was informed by these 

other efforts to promote the possibility of standardizing the method for use by other organizations.

Adding to the complexity of the study of human performance in nuclear power is that many of the

causes for human error cannot be observed because they occur in the realm of human cognition.

This is one of the reasons why data from human failures have not been used to derive failure

estimates for human reliability assessment (HRA) in the same way that equipment failures and

unavailability are used to estimate hardware failure parameters for probabilistic risk assessment

(PRA). Further, the underlying theoretical treatment of the human as another component of the

NPP system that can fail predictably if all the critical parameters can be defined is difficult at best

to achieve. Estimating failure probabilities from counts of human error alone ignores relevant

aspects of the performance context and overlooks the interdependency between cognition and

context. This interdependency is critical given that context is well understood to be a driver for

cognition (Sternberg 1996, Wickens 1984, Wickens 2008). The ability to empirically test and

verify aspects of operational demand and crew capacity as they relate to success and failure and

to develop systematic insights about human reliability would be valuable to predict error, as well

as to better estimate the likelihood of its occurrence.

One of the hallmark characteristics of the reliability engineering discipline is the ability to “test to 

failure” by creating system failure conditions which can be observed and used to collect data to 

establish failure rates for various failure types. For example, subjecting structures and structural 

materials to various design loads, including those design loads that could produce structural 

failures, is one method used to qualify new building materials and construction methods, as well 

as to establish engineering codes and standards. Validated models using such data can

describe the physics of failure of electro-mechanical devices or structural systems. Such models
2

also can be developed for new SSCs employed in safety critical applications, such as those that 

ensure the safety or integrity of nuclear energy systems are observable, measurable, 

quantifiable, and predictable.

Human performance cannot yet be subjected to a form of reliability analysis comparable to that 

of mechanical and electrical systems. Although the behavioral sciences are comprised of an 

extensive body of literature and knowledge, some believe that this knowledge is not as easy to 

employ in HRA as is performance data obtained from engineered systems for traditional RE 

applications. This belief is perhaps supported by the view that human capabilities are far more 

extensive, complex, and interconnected and therefore, not understood as well as the capabilities

of electro-mechanical devices. The notion that our state of knowledge of the functioning of

human cognition and behavior is not as advanced as for mechanical systems, has potentially

inhibited direct adaptation of reliability engineering (RE) concepts in HRA, although HRA is part

of the reliability discipline. Furthermore, until lately, there was little focus in the behavioral

science on developing empirical basis for HRA whose objective is to analyze human 

performance dealing with rather extreme situations. However, developing functional models, as 

well as testing and refining theories, about human related processes involved in accident 

conditions has been recognized as vital to shaping our knowledge and expectations about 

human reliability (Williams 1992).

Failures in human performance that would be of interest to HRA are rare in opportunity and, 

thus, unlikely to be observed except by design. In the real world, the expected plant conditions 

to which plant staff would have to respond rarely occur – and occur even more seldom for PRA 

events and their precursors. The HFEs modeled in a PRA are events that result from unique

combinations of failures (i.e., initiating events) and plant conditions. Reports of human failures in 

behavioral science studies in non-nuclear domains have shown how humans fail in complex 

environments (Vicente and Tanabe 1993). However, analysis of reported failures cannot always 

distinguish between failures due to variability in human response (e.g., a simple ‘slip’ in

activating a control device) and failures due to systemic, underlying causes such as real limits in 

human capabilities (e.g., an overload of mental attention that interferes with an operator’s 

activation of a control device). This underscores the need for more directly relevant sources of 

data about human performance that can be used to derive information about the limits of operator

performance or capabilities in NPPs. In this report the concepts discussed above are being used

to develop an approach for using simulators to produce human performance data suitable to

HRA. The approach is using some fundamental features of RE engineering allowing the mining

and dissemination of the data in an efficient manner reducing the amount of effort needed to be

documented, analyzed and shared. A particular emphasis of the proposed approach is the

sharing of such data across organizations.

A key aspect of the proposed approach is the use of a standardized measurement technique that

could be used by anyone conducting a simulator-based study. The proposed measurement 

technique employs the notion of comparing crew objective performance with the limiting 

condition for success as used in the plant PRA, which we characterize as the limit state. The 

intention of suggesting this particular standardized measure is to support inferences about crew 

performance and the reliability (or unreliability) of that performance in simulated exercises built 

around PRA contexts. Other purposes of the proposed approach are to minimize the efforts 

needed to interpret human performance data collected and to provide an objective measure of 

human performance that can be directly related to success in the PRA context.
3

1.2 Purpose

The purpose of this report is to propose a formalized approach for collecting human 

performance data employing the concept of a limit state, to describe the process for collecting 

data, and to present illustrative examples. This approach borrows from techniques applied in 

traditional reliability engineering (RE) while using experimental techniques derived from the 

behavioral sciences.

The ultimate aim of the proposed approach is to achieve greater consensus, consistency and 

convergence of HRA methods by stimulating simulator data exchange, greater communication, 

and review within the HRA research and practitioner community. The proposed approach is 

intended to yield data that strengthens the technical bases for HRA methods and their 

applications.

To obtain such data when dealing with human performance requires consideration of the 

principles of experimental design derived from behavioral and psychological research. This 

research proposes an approach that can be employed in a standardized manner to consistently 

collect data that measures aspects of human performance relevant to human reliability analysis. 

To do so, principles of reliability engineering were investigated, particularly the principle of 

demand and capacity and the associated notion of a limit state. This latter notion is central to

this effort. A limit state is a formally identified region of performance that distinguishes 

successful from unsuccessful performance. In this effort, the notion of a limit state was applied 

to human performance in order to define success and failure of actions performed by operators 

in response to a PRA design basis event.

The approach to design scenarios and measure human performance is derived using a robust 

experimental design. As such it should ensure consistency of results, and application of the 

findings from PRA scenarios to address general issues of human reliability as well as support 

the eventual goal of exchanging data among organizations that collect such data. The 

characteristics of such a robust experimental design include:

1. Objective performance measures for operating crew performance in PRA scenarios.

2. Plant parameters that correspond directly to operator actions related to human failure 

events (HFEs) of interest in PRA.

3. Summaries and descriptions of crew performance that account for crew decision making 

and performance related to the human actions of interest for each PRA scenario.

4. Structured measurement of performance drivers that can be related to operator and 

crew performance.

5. Crew feedback using structured debriefings to collect retrospective observations from 

crewmembers that provide insights about the factors that influenced their performance.

Ultimately, HRA practitioners may wish to develop probability estimates from the empirical data 

collected. However, this is beyond the scope of the current effort. This effort is not intended to 

serve as an end-to-end method for generating probability estimates of human reliability. The 

current research is intended to identify and describe some of the research characteristics that 

can ensure that studies of human performance produce data that are directly related to human 

reliability issues, as used in PRA. Further, the results of the research can support the eventual 

exchange of information or data about human performance among organizations that collect and

use such data.
4

1.3 A Framework for Human Reliability Analysis Data Collection

Because of its relationship with other disciplines, an HRA data generation approach must reflect 

critical aspects of the underlying scientific and methodological requirements of the scientific 

disciplines upon which it is based (Hallbert et al. 2007). HRA exists at a nexus between three 

distinct disciplines: (1) the behavioral sciences (i.e., human factors, ergonomics, cognitive 

sciences, and psychology); (2) RE; and (3) PRA. Each discipline has a tradition of scientific 

development and theoretical underpinnings that reflect the formal requirements for, and use of 

information (e.g., data, knowledge, etc.) to address issues to which the discipline is applied.

This is seen in efforts that generate data to support HRA to either develop and validate HRA

models or to the use performance data to predict human reliability parameters (Hallbert et al.

2011, Groth 2009).

The nature of HRA is interdisciplinary. Behavioral sciences draw from the disciplines of 

psychology, cognitive science, human factors and offer a variety of theoretical models to model 

human performance dealing with unexpected conditions. As a result several HRA methods are 

currently available, each handling human performance differently. An issue dealt with in the last 

few years is the variability of results in HRA; i.e., recognition that analysis of a given human 

action may produce different results (estimations and insights) using different methods as well as

different analysts using same method. Addressing this issue has been the focus of the HRA 

community. This led recently to collaborative studies sponsored by OECD (Lois et al, Bye et al, 

Dang et al.) as well as domestically (Forester et al.). The NRC also undertook the initiative to 

perform a thorough literature search to update the technical basis for HRA (Whaley et al. 2012 a,

Whaley et al. 2012b) as well as to develop a hybrid HRA method called A Method of the 

Integrated Human Event Analysis System (IDHEAS) (Xing et al). Data that are used to inform 

(i.e., develop, validate, test aspects of, etc.) modern HRA methods must also be based on solid 

technical and methodological foundations. This includes approaches to human performance 

measurement and experimental design based on both behavioral science and reliability 

engineering.

RE analysis concepts have been developed from an equipment and physical system 

performance perspective. Accordingly, they are used to assess the performance of discrete 

system elements, having well-characterized governing dynamics (e.g., mechanical, electrical, 

chemical, etc.), interacting with other discrete elements under well-described conditions. Hence, 

the interactions among components in a mechanical system are straightforward to characterize. 

This includes the manner in which they receive and process inputs from other components, the 

manner in which they produce outputs, and the specific outputs they produce. The PRA

provides a clear structure to identify the tasks needed to be performed as well as the conditions 

under which the tasks should be performed. From that perspective, PRA analyzes equipment as 

well as human failures. To assess equipment failures PRA analysts are relying on information 

and data developed with RE principles, including the test-to-failure principle on the basis of

which system capability is characterized for the various conditions assumed to be functioning 

when needed. Similarly, HRA needs information related to test-to-failure information for the

various conditions.

The latest focus of the HRA community is to use simulator experiments to advance HRA. The 

current effort employs existing capabilities in both behavioral sciences and RE to develop a 

structured approach for performing experiments focusing on HRA; that is, experiments that 

produce human performance data which satisfy both the behavioral sciences dealing with
5

human capabilities needed for tasks related to responding to unaccepted events, and

associated drivers of performance determined by both external situations (e.g., design of control 

room, severity of event etc.) as well as cognitive and physical demands. The previously 

mentioned simulator studies (Lois, Bey, Dang, Forester) for HRA demonstrate the feasibility and 

practicality to use simulator facilities to improve the robustness of HRA. In fact the NRC has 

recently undertaken an effort to collect human performance data from the plant’s routine training 

using their plant simulators (Chang & Lois, 2012; Criscione et al, 2012).

Today’s field of HRA should benefit from a formalized approach analogous to that used in the 

behavioral sciences to guide the collection of data that then can be used to develop insights and 

improve models used for HRA as well as address needs in a reliability engineering domain. For 

example, collecting data about the performance context of a pump may not be important to 

estimate a parameter of pump reliability. However, performance context data may be very 

important for human reliability where performance and context are known to be very 

interdependent. Also, raw data about operator performance varies from plant to plant, owing to 

differences in their design and corresponding thermal-hydraulic behavior. The proposed 

approach attempts to make possible to share human performance data regardless of such 

differences. Using the PRA concepts of success criteria, it is a formalized approach using 

performance measures that can be applied to any study regardless of individual plant

differences and can relate directly to the matter of human reliability (i.e., the success or failure of 

human action in PRA contexts).

1.4 Document Organization

The document first describes the theoretical background and technical basis for the proposed 

standardized HRA simulator data collection approach. Then it describes the five tenets of the 

data collection approach, followed by a step-by-step process for conducting an HRA simulator 

data collection study. It is followed by a description and examples from a study where the 

proposed standardized approach was applied. A conclusion section summarizes the documen

7

2. PRINCIPLES FOR COLLECTING HRA DATA FROM NUCLEAR 

POWER PLANT SIMULATORS WITH AN EMPHASIS ON THE LIMIT 

STATE CONCEPT

2.1 Introduction

Collecting simulator data to support HRA requires robust experimental design and a process. 

This section describes five essential experimental design characteristics based on paradigms 

from the behavioral sciences that should be considered when preparing to collect data for 

application in HRA. The section includes recommendations on categories of information to 

collect when employing training simulators.

HRA methods consider conditions that may be found in probabilistic safety analysis contexts to 

predict the kinds of errors (e.g., slips, lapses, or mistakes) and the likelihood of those errors (i.e.,

probability) that can occur. There is substantial experimental and experiential evidence

about the many factors that may be “drivers” of human reliability in PRA contexts. These include 

factors such as stress, workload, human-system interface technologies, training and

experience, crew resource management, intra-team dynamics, and others.

2.2 Principles of the Data Collection Approach

The following are proposed as principles to guide collecting human performance data. They are 

intended to provide information about the conditions, qualitative aspects of HFE, and data on the 

occurrence of the HFEs to support their use in HRA research and related applications.

2.2.1 Performance Measures Should Relate to Success and Failure as Defined 

in the PRA

In all simulator-based activities in which crews of operators demonstrate their performance in 

PRA-based simulated events, success and failure needs to be clearly defined. The definition of 

success will differ across PRA events since the required operator actions needed to mitigate the 

effects of initiating events also differ. It may also differ within a single class of a PRA scenario 

due to differences in the postulated initial conditions, timing of the initiating event(s), and any 

additional complications and failures included for training or testing. Notwithstanding such 

differences, success criteria exist for each PRA context based on plant response and the need

to prevent damage to the nuclear fuel and release of radioactive materials.

The measure of success should be capable of characterizing performance on a continuum so 

as not to limit the ability to draw more useful insights from such a measure. For example, it is 

possible that all crews in a training sample complete needed actions to mitigate a design basis 

event. This would indicate one hundred percent success on the critical mitigation action(s). 

Upon closer inspection, systematic variability may exist indicating that some crews did so with 

substantial ‘margin’ to spare while others did not. Employing a continuum permits

characterization of performance with greater contextual sensitivity to issues such as available or 

remaining margin to failure and variability in performance amongst crews. It also permits 

estimation of measures of central tendency and statistical analysis that can be used in HRA.

For PRA purposes, an absolute, event-specific limit exists for design basis events. Its use

should inform data collection efforts, for example, to identify the extent to which it is exceeded or
8

challenged in some other way during simulator based crew performance exercises. Its use 

would assist in reducing subjectivity when interpreting the results of crew performance.

2.2.2 Simulator Contexts Should be Representative of PRA Events

Scenarios and contexts should be representative of the needs for PRA and the different 

performance conditions that operating crews may encounter. For purposes of HRA research, this 

necessitates understanding and incorporating important conditions in data collection situations to

ensure the environments sampled are representative of PRA events and performance contexts in

PRA events. In particular, differences in scenarios that arise within classes of PRA events should

be taken into account. This includes such features as the magnitude of challenge to plant safety,

the pace of an event, complications that arise from equipment failures and others, which vary

across instances of a class of design basis events. Such features result in differences in the 

tasks to be performed by the control room crew, available time for crew response, the degree of 

interpretation and problem solving required, and ancillary demands (e.g., emergency and offsite 

notifications or use of ex-control room plant staff and coordination of these resources). Human 

reliability issues may thus arise from differences

in how scenario demands manifest themselves within a class of design basis events.

Characterizing a particular scenario for the purposes of HRA on the basis of the initiating event 

and event sequence is an important part of describing a representative range of demand 

conditions. Additional information is also needed to characterize human performance demand 

features in order to relate observed variability in crew performance or failures back to potential 

causal demand features. This requires describing, with specificity, the kinds of cognitive and 

human performance demands present in simulator scenarios that are used for data collection. It 

also requires that demands be described consistently, following an approach that may be 

standardized and applying an associated, accepted model of human performance. 

Characterizing scenario and context demands systematically, according to valid and

consistently applied criteria, is an important part of understanding the data. Furthermore, these 

characterizations may serve as a means of identifying and selecting data for future use and 

study.

Data collected from well-designed scenarios and run on plant training simulators or research 

simulators, should be representative of human performance and reliability. This includes the 

ways that crews are expected to behave and perform in the simulator environment, and the 

availability of data from plant-specific simulators that can be used to characterize the 

performance of crews.

2.2.3 Data Collection Should Include the Opportunity to Observe Human

Failure Events of Interest

The analysis of a PRA event (i.e., event sequence) encompasses performance conditions and 

specific needed human actions. The human failure events of interest in an event sequence 

represent human actions that, if not performed or not performed to a prescribed standard, will 

result in a functional failure of a system or safety criteria and may lead to degraded plant 

conditions. Data on these human actions are of primary interest and constitute the main unit of 

measure. The result is that error information is typically defined in terms of specific plant 

parameters from an individual plant. Observations based on plant-specific data are not easily 

generalized to other plants especially where the success criteria for a human action differs 

across plants.
9

Observations of failure under varying demand conditions are needed to provide data about one 

of the most important aspects of HRA methods: the ability to predict performance failure. It is 

vital to the development of human performance reliability theory and methods that data 

collection efforts include the opportunity to observe an unrestricted range of human 

performance, including success and failure. HRA research should be carried out under 

conditions similar to those modeled in PRAs. It should include those human actions both 

implicitly and explicitly modeled in PRAs as needed for recovery and mitigation of plant 

conditions. Limiting the range of demand in data collection to those that minimize opportunities 

to observe HFEs, errors, or deviations (i.e., partial failures) restricts the ability to formulate or 

validate theories of human reliability.

Data for performance success and failure are needed for a number of reasons. First, a data 

collection approach that is capable of yielding a representative range of results from the success

and failure regions of interest can form the basis for collaborative research. Because it employs

methods and measures that can provide insights about human performance that transcend

plant-specific issues, the proposed approach may serve as a means to support the exchange of

human performance reliability information across organizations. Collaborative efforts that

emphasize the exchange of research methods and results from human reliability studies can be

instrumental in overcoming a lack of progress in this area by engaging more organizations in the

exchange of human reliability data.

Analogous to electro-mechanical systems reliability assessments, opportunities are needed to 

gather a substantial record of data from relevant operational contexts to produce baseline 

human reliability estimates. For HRA, the “system” of interest is the joint human-machine 

system. Actuarial type data of human reliability from representative PRA contexts could provide 

a sufficient evidentiary basis to support formulation of insights about a number of relevant 

issues, including:

• Rank ordering of observed HFEs across event sequences and plant conditions

• Sensitivity of human performance reliability to plant conditions

• Relationships between event sequence instantiations, plant conditions, and PSFs

• Identification of error-likely contexts.

2.2.4 Data Collection Should Include Objective Human Performance Measures 

of Success and Failure

Performance criteria exist for all aspects of engineered systems and especially so for nuclear 

energy systems. In the case of PRAs, event sequences characterize the performance of 

systems using binary diagrams based on the success or failure of systems or functions. The role 

of operators and criteria for their performance in PRAs is based on needs to carry out actions in 

response to initiating events and other failures. Human actions are performed to mitigate an 

event sequence and are essential to plant stabilization and recovery. The task of human 

reliability analysts is to understand the demands and capabilities of plant staff and estimate the 

reliability with which it can be done. Translating this into observations that can be made in a 

simulator environment requires, at a minimum, that criteria exist or can be developed to 

distinguish between successful and unsuccessful human performance. Beyond this, it is 

important to characterize contextual features (i.e., plant conditions and performance-shaping 

factors) that contribute to human performance.
10

Translating the definition of success or failure to PRA terms is also necessary. For example, for 

every required human action in an accident sequence context, a thermal hydraulic criterion must 

be met to prevent more degraded plant conditions. This includes human actions in response to 

dependent equipment failures that dispose the plant to a more significant damage state. For each 

human failure event considered, a corresponding consequence that relates explicitly to those 

modeled in the PRA should be identified.

Taken together, the definition of success (or failure) and the consequence provide the criteria for 

human performance that also relate directly to the plant PRA and its technical basis. They serve 

as a single objective measure that, if accomplished according to specified criteria, prevent further 

degradation of plant conditions. They also signify an action or group of related actions which, if 

not performed to specifications, imply a functional failure that conditionally worsens plant 

conditions and leads to possibly more significant plant damage.

2.2.5 Data Collected Should Describe Relevant Aspects of Performance and 

the Factors that Affect that Performance

In addition to objective performance characteristics, it may be desirable to obtain data that 

provide insights about crew decisions and behaviors. Objective performance measurement is 

essential to obtain an unbiased and accurate depiction of crew performance. It describes the 

“what” and “when” aspects of crew and plant behavior. Though accurate, objective measures 

may not provide a complete account of crew performance. To do so requires additional and 

complementary data about the “why” and “how” of performance. Therefore, it is important to 

gather information about deviations, lesser mistakes, crew tendencies, information processing 

and decision making, team interactions, and other relevant information that relate to causal 

mechanisms and influences that account for variation in crew performance.

It is difficult and impractical to anticipate the variability in perceptions, decisions, and actions of 

crews and crew members as they cope with the many variations of plant challenges that can be 

played out in a training simulator. It is important to be able to collect data that reflect potential 

variation in order to subsequently assess its significance for human reliability purposes. This 

requires that a data collection protocol be designed in advance that is sufficiently flexible to 

capture both expected and unexpected, emergent crew behavior while at the same time being 

simple and reliable to employ.

Inter-rater reliability and simplicity of the data collection method are important. The data 

collection protocol should ensure that the expected user can reliably and efficiently obtain the 

required data. The data collection protocol should be designed to minimize burden that would 

arise from the additional tasks expected of plant personnel to produce data.

2.3 Categories of Information

Based on the foregoing considerations, the authors recommend that the following information 

be collected. It should be noted that all of the categories be tailored according to the scenario 

and that the scenario could affect which metrics of human performance are most applicable.

1. Objective performance of operators and crews in PRA scenarios.

2. Plant parameters that provide direct evidence of operator actions related to the HFE of 

interest.
11

3. Summaries and descriptions of operator performance to account for important 

differences in crew performance on the critical human actions of interest for each PRA 

scenario.

4. Structured measurement of PSFs that can be related back to operator and crew 

performance.

5. Crew feedback using structured debriefings to collect retrospective observations from 

crew members and trainers or other pertinent plant experts.

To obtain data related to these categories, a combination of sources is needed. This includes 

data from sources that supply data about the physical behavior of the plant and systems and 

about the activities of the crew. Typical information about plant systems includes the following:

• Failures and malfunctions inserted into the simulation

• Protection system actuations and automatic actuations of plant systems

• Alarms and notifications through the plant computer

• Plant parameters logged regularly and recorded electronically.

Information about crew performance should include the following (not all may apply):

• Numbers of control room crew members, their qualifications, and when present

• The normal main control room staffing, and any exceptions that applied during the 

scenario

• Control actions performed correlated in time to plant parameters

• Alarms acknowledged

• Recognitions and communication of key events and plant conditions

• Procedures entered and in effect (i.e., for conditions in which multiple procedures may 

be opened)

• Information sources consulted other than procedures (e.g., technical and reference 

information used for decision making or ex-control room advisors such as a technical 

support center)

• Errors, significant deviations from the expected course of action, consequences, and 

other noteworthy observations

The next section describes the step-by-step process recommended to collect the data described 

above.

13

3. COLLECTING DATA FROM TRAINING SIMULATORS FOR HRA

The process of conducting an HRA simulator data collection effort can be described in four main 

phases: preparation, data collection, data analysis, and reporting. Table 1 depicts these four 

phases and their respective sub-steps. Some of the sub-steps are discussed in detail in this 

section. For example, in the preparation section, legal contractual mechanisms are discussed,

as well as important items to consider when designing and validating the scenarios to be used in 

the data collection effort. This includes the determination of success and failure criteria and 

human performance measures. The data collection section presents different methods to be used 

for data collection and also describes how to structure the collection effort, keeping in mind the 

information in Section 2 of this report. In the data analysis section, examples of how to

develop synopses and how to document deviations and errors are provided. There is also a

description of how to analyze performance shaping factors in this section.

Table 1 is derived, in part, from the description of integrated system validation in the human 

factors verification and validation chapter (Chapter 11) of NUREG-0711 (O’Hara et al. 2012). 

The table provides a high-level description of the different steps in the HRA simulator data 

collection process.

Table 1 Simulator data collection procedure.

Procedure Step (see Flowchart) Description

Phase 1: Preparation for Data Collection

A0 Prepare a formal 

agreement

A formal agreement must be established with the plant 

at which the study will take place. The agreement

must ensure the anonymity of the plant or utility and 

that the results of evaluations will not affect individual 

operators or result in fines.

A1 Assemble the data 

collection team

All individuals who will be required to design and 

conduct simulator exercises must be identified at this

stage. Ensure that a plant simulator training instructor

is available to assist in development of the scenario. 

Where the exercise is not conducted by onsite 

personnel, arrangements must be made for plant visits 

by the data collection team in advance.

A2 Review Human Failure 

Events from Probabilistic 

Risk Assessments,

Operating Experience

Reports, and additional

Plant Information

HFEs must be extracted from available information, 

such as the plant’s PRA, event reports, and any other 

appropriate plant information.

A3 Scenario development The scenarios selected to be used for the data 

collection must be representative of the conditions and 

parameters identified by the team. If possible, obtain 

input from the plant simulator training instructor.
14

Procedure Step (see Flowchart) Description

A4 Review scenarios The scenarios developed in A3 must be reviewed and 

finalized for logic and accuracy by the team including

a plant simulator instructor. The plant-specific 

procedures that will be needed to perform each of the 

scenarios must be identified and made available in the 

simulated control room.

A5 Conduct walkthrough of 

scenarios on the

simulator

Conduct a walkthrough of the scenario on the 

simulator to determine plant response times with and

without operator intervention. Verify that the

procedures that will be used during the study are 

available and appropriate. Confirm availability of 

simulator logs to be captured throughout the exercise. 

If possible, the walkthrough may include operators who

were not involved in the scenario development and will

not participate in the data collection to obtain 

independent input on response times and other 

demands.

A6 Determine plant 

parameters to be 

recorded

Select and verify the plant parameters that should be 

recorded by the simulator for evaluation of the crew 

and plant performance. Simulator logs must be

captured throughout the exercise. This will consist of

an electronic record of the plant events and the 

chronological status of all parameters. Verify that 

these records will be accurately captured during the 

exercise. Also verify that the format of the record will 

be readable and usable for data analysis.

A7 Prepare forms needed 

for data collection

The forms and other data capture techniques needed 

for data collection should be identified and developed

as needed. Key measures should be defined and

incorporated in the data collection forms. Standard, 

accepted measures of performance should be used, 

when available.

Develop and prepare forms for the exercise to include, 

but not be limited to the following:

• Consent form

• Time log

• Performance measurement forms and records

• Post scenario, pre-debrief forms and questions

• Post-scenario debriefing questions.

More information on principles and methods of 

performance measurement may be found in NUREG0711.
15

Procedure Step (see Flowchart) Description

A8 Verify crew(s) availability Ensure through the facility that the crews and crew 

members identified to assist in data collection will be

available as scheduled. As available, conduct 

informed consent with the crews and crew members 

prior to actual data collection. Provide informed 

consent information to plant personnel who serve as

the main interface with operating crew members to

assist in explaining the informed consent.

A9 Conduct scenario dry run A dry-run is a rehearsal of the actual data collection 

and should be performed for each of the simulator 

scenarios using plant training staff as crew members 

and evaluators to validate each of the scenarios. This 

includes the following:

• Identifying simulator instructor roles

• Re-verifying scenario event times

• Verifying procedure usage and operator 

anticipated actions

• Establish data collector and observer roles and 

responsibilities. Identify observer positions in the 

simulator in order to minimize interference with 

the crew members.

Phase 2: Data Collection

B0 Conduct scenario 

exercises

Each exercise starts with an entrance meeting with the 

crew members where they are introduced to the data 

collectors, the purpose of the exercise is explained, 

how the exercise will be conducted, and the purpose

of the post-exercise debriefings. Informed consent

must be obtained prior to obtaining data from crews. 

Before the scenario begins, the plant simulator 

instructor will give the plant status briefing and 

turnover. The scenario will be initiated and the data 

collector(s) will record events and crew actions. The 

scenario will be run to the identified end point or until a 

point determined by plant training personnel in 

conjunction with the data collection team.

B1 Record real-time 

observation data

Use forms prepared in A7 to collect data from 

simulator indications and from crew responses to 

forms and questionnaires. Where possible, use 

audio/video recordings to capture the entire scenario.

These recordings can be used during crew debriefings

and will be reviewed during the data analysis phase. 

They can be used to confirm data recorded manually 

and to identify any behaviors that went unnoticed 

during the exercise.

B2 Collect performance 

driver information

Any immediate debriefing information should be 

collected following the conclusion of each scenario.

This should be done when it is necessary to avoid

discussions influencing individual crew members’
16

Procedure Step (see Flowchart) Description

recall or assessment of debriefing subject matter.

B3 Conduct scenario 

debriefing

A debriefing of the crew occurs following completion of 

the scenario in the training simulator. This debriefing

follows a pre-planned discussion based on a prepared 

debriefing. The debriefing is conducted using a set of 

questions developed to guide discussions. Crew 

discussions are encouraged to facilitate identification of

factors that contributed to crew performance and

decision making as well as to understand the reasons

for any deviations from expected performance, or 

errors.

Phase 3: Data Analysis

C0 Analyze results of 

scenario exercises

Analysis of the results of crew performance includes a 

detailed review of the electronic record (simulator log). 

Data reflecting important plant parameters should be 

selected for all crews. These data will be portrayed on 

a graphical timeline that shows their evolution over the

course of the scenario in response to the initiating

event and crew actions. It may be to the analysts’ 

benefit to conduct preliminary analyses onsite, when 

crew members may still be available for follow-up 

questions.

C1 Analyze crew 

performance results

The data analyzed in C0 are used to compile a 

synopsis of crew performance in each simulated

scenario. This consists of a detailed chronological

analysis of how each crew responded to the events 

simulated. The results are integrated with the graphic 

timeline of plant parameters and equipment actions 

and emphasizes annotation of the crews’ 

performance. This may include key actions, 

detections, announcements and notifications,

procedural transitions, and other observable behaviors 

related to management of the simulated event. (See 

examples in Appendix A).

C2 Performance Driver

Analysis

In addition to the analysis of objective performance 

analyses of performance drivers may be conducted. 

Analyst(s) should attempt to integrate crew accounts

of factors that influenced their performance with their

demonstrated performance. In particular, analyses 

should emphasize the way(s) in which performance 

drivers contributed to the reliability of crew 

performance.

C3 Limit State Analyses The limit state analysis should be conducted to identify 

the number of successful human actions for each

critical action of interest – or HFE represented in the

plant PRA. Each synopsis of crew performance in C1 

should identify the success or failure of the crew to 

mitigate the event(s). Analyses may also be performed
17

Procedure Step (see Flowchart) Description

to assess the “margin” between demonstrated crew 

response and the PRA success criterion.

Phase 4: Reporting

D0 Produce scenario 

analysis report

Summaries based upon the data analysis are 

compiled in a report. The report contains the

information and analyses previously described and 

documents features of the scenarios simulated. The 

report should be written to highlight key features of the 

data collection, including:

• Purpose and objectives

• Scenarios and their relation to the plant PRA

• Human actions key to the successful resolution of 

the event sequence

• Data collection method, crew features, sample 

size

• Overview of data collected

• Analyses and results

• Summary and, conclusions, as appropriate.
18

Figure 1 Recommended human reliability analysis simulator data collection process
19

4. PREPARATION FOR DATA COLLECTION

This section provides additional guidance on the steps in the Preparation Phase listed in items

A0 through A9 in Table 1.

A0. Prepare a Formal Agreement

A formal agreement should be established between the data collection organization and the 

participating asset owner, e.g., nuclear power plant. The purpose of the agreement is to 

establish the roles and responsibilities and limitations of the data collection and analysis 

organization and the participating nuclear power plant. The elements in the agreement are to 

facilitate the free flow of information between the asset owner and its operating crews and the 

team conducting the data collection. The agreement should ensure that all parties accept the 

terms and conditions set forth and that data and other proprietary information that may be 

disclosed for purposes of the study are handled and treated in a manner that protects the 

participating organization and its staff.

A1. Assemble the Data Collection Team

The team of data collectors should be comprised of several members. One member should be 

experienced in the HRA process. Other members should be experienced in reactor plant 

operations and have knowledge of the plant design. This team, in conjunction with the plant 

simulator training staff, will develop the scenarios to be used for this data collection. A consultant

to the team familiar with the plant-specific PRA can provide valuable assistance with scenario

design and recommend practical advice to assist in evaluation of data collected. The PRA

consultant also should serve as a liaison with the data collection team. Other team members

may be needed, depending on the nature of questions that arise in the development of

scenarios and data collection. Some or all members of this evaluation team will observe the 

crews during the simulator scenario exercises, collect data, and analyze the results.

A2. Review Human Failure Events from Probabilistic Risk Assessments, 

Operating Experience Reports, and Additional Plant Information

The design of the simulator scenario should be developed using the plant-specific PRA. The 

flow of the events and failures described in the scenario should be as realistic as possible.

When developing the script of events in the simulator scenarios, it is recommended to use plantspecific events from past facility events or equipment failures to provide realism. When selecting 

events from past facility events, they should be related to the plant PRA or the objectives of 

scenario design (i.e., to manifest plant conditions or context). In addition, operating experience 

events from plants of similar design that are related to the scenario(s) being developed may 

prove useful in designing scenarios.

A3. Scenario Development

Scenario development should use plant-specific procedures to establish the expected scenario 

flow of events. They also serve as a common referent for obtaining input from experienced 

operations personnel, plant training personnel, and other personnel who may be consulted in 

development of scenarios. The pace, severity, and complexity of the scenario establish some of 

the unique demand features on which to focus. Systematic variation of these features to achieve 

a scenario design objective will allow evaluation of how different aspects (such as crew training,
20

team work, and other capabilities) influence crew response. The scenario should be developed 

with the expected influence of performance drivers in mind (such as human machine interface, 

procedural guidance and training, expected workload, and potential deficits in situational 

awareness induced by the event, etc).

The recommended steps in scenario development are as follows:

1. Define the purpose, scope, and content of the simulator exercise, as well as the 

conditions under which the exercise will be conducted

It is important that the scenario be focused on the behavior and actions of the control 

room crew during response to a PRA event.

The information provided in the scenario description should contain, at a minimum:

a. Initial plant conditions or the information that will be provided to the crew in a 

turnover briefing (e.g., reactor at 100% power, list of equipment out of service 

for testing, and maintenance, time in life, or upcoming evolutions).

b. The list of planned malfunctions with the appropriate timing for initiation. When 

timing of malfunctions is crew response-dependent, this should be likewise

noted (e.g., a rupture in the steam generator will occur in the steam generator to 

which the crew reestablishes feedwater, etc).

c. The list of expected operator actions and procedures expected to be entered.

Also, identify contingency actions and needed role play (i.e., by simulator 

training personnel) to ensure the scenario proceeds as intended.

d. Identification of the criteria for the crew to successfully perform an action. For 

example: “To establish auxiliary feedwater (AFW) flow to the steam generators 

(S/Gs), the crew can dispatch a plant operator to check and close the open 

recirculation valve (feed S/G B) or cross-connect AFW flow from pump 12 to 

S/G A, C, or D.”

e. A summary list of all the pertinent procedures that may be used by the operating 

crew. This list will be based on the expected flow of events, prior experience and 

training of plant staff, and reflect best estimates of likely crew response.

Examples of simulator scenarios are shown in Appendix B.

It is advisable to develop an extra scenario as a backup in the event that an unforeseen 

condition will prevent the use of one of the scenarios.

2. Determine success and failure criteria

Using the plant-specific PRA, success and failure criteria can be developed for each 

planned scenario. Performance criteria for success or failure may be based on 

engineering reference material, technical reports, procedures, or other sources that 

establish the performance criteria on which the plant-specific PRA is based. The plantspecific PRAs will define limit states for the described events, thereby aiding in 

establishing the success and failure criteria. The concept of limit state needs to be 

operationally defined so that success and failure can be better understood in this 

context. Limit state from a RE and systems safety perspective denotes a region of
21

Time to Initiate F&B to Prevent Core Damage

in Minutes

performance that distinguishes between performance success and failure. A limit state 

may be explicit (e.g., establish feed and bleed (F&B) before reaching a certain plantspecific limit) or implicit (e.g., based on an engineering code analysis of thermalhydraulic behavior).

An example is shown in Figure 2 below. This limit state is based on the required operator 

action in response to a total loss of feedwater (LOFW) in a pressurized water reactor. The

figure is a plot of the time required to manually initiate reactor coolant system (RCS) feed

and bleed (i.e., once through core cooling) in response to a total loss of feedwater. 

Several points on the graph are explicit, based on a MELCOR analysis of fuel behavior

for the plant. Other points along the graph are implicit, having been extrapolated from the 

same MELCOR analysis and estimated based on engineering judgment.

Crew performance below the line denotes success (i.e., initiating RCS feed and bleed in 

time to prevent fuel damage). Crew performance above the line indicates failure –

exceeding the limit state condition for this operator action. This particular graph illustrates

an important point regarding objective performance criteria: the criteria used for HRA

must be adapted for the particular PRA-based scenario. These performance criteria 

ought to conform to the notion of the PRA, including recovery criteria and conditions that 

delineate between successful human performance and an HFE. An analysis of crew 

performance data should permit a determination to be made regarding success of the 

crew in terms of the requirements of the PRA. This minimizes the potential for subjectivity

in evaluating crew performance.

100

90 90

80

70

Time to Feed & Bleed

82

73

60 65

56

50 48

40 39

30 30

20 22

10 13

0

15 20 25 30 35 40 45 50 55 60

Time from Loss of Feed to Rx Trip in Seconds

Time to Feed & Bleed

Figure 2 Performance criteria and limit state for feed and bleed action

Performance criteria in relation to the plant-specific limit state for the PRA event sequence

must be developed for all elements of the scenario that are to be evaluated for their

relevance to HRA [e.g., the criteria for re-establishing feedwater to the steam
22

generator (S/G)]. These criteria may be obtained from operating procedure steps or other 

sources and defined for the plant conditions as they manifest in the scenario. Examples 

of the success and failure criteria for the scenarios developed in the reference study are 

found in Appendix C.

3. Define human performance measures

Data collection should sample from the conditions predicted in the PRA to drive 

performance reliability. Data collection should include human failure events of interest. 

The data collected should describe objective performance. This includes the physical 

behavior of the plant and systems, as well as the activities of the crew in the simulated 

performance context. This will provide a description of what happened and when it 

occurred.

Performance criteria also must be available or determined for the conditions analyzed. 

These should be based on the same criteria employed in the PRA, as applied to the 

particular form of the event sequence run in the simulator. These performance criteria 

ought to conform to the notion of the limit state as discussed earlier. This means that, in 

terms of the requirements of the PRA or other relevant portions of the plant safety 

analysis or design basis of individual systems, a comparison of crew performance data 

with the performance criteria must allow the analyst to determine whether an individual 

crew’s performance was successful or not.

Information about the crew and its actions should include:

a. Numbers of control room crew members, their qualifications, and when present 

b. Control actions performed

c. Alarms acknowledged

d. Recognition of key events and plant conditions and procedures entered and in 

effect (i.e., for conditions in which multiple procedures may be opened).

The following categories of data are typically recorded during observation of control 

room crew performance and should be obtained to support subsequent analyses:

a. System performance parameters associated with the scenario event 

b. Time available for completing a given task

c. Actual times that crew members took to complete a given task

d. When the crew recognizes the event or announces the parameter change that

identifies the event

e. When orders are given and implemented 

f. Content of crew briefings

g. Times when procedures are entered and exited
23

h. When alarms are acknowledged or reset and whether the alarm was identified

To determine the above times, use the simulation run time and record when events 

occur during the simulation.

These records can be obtained by the evaluation team members who are recording the 

times, either manually or with recording devices. Where possible, all records should be 

verified by the simulator parameter recordings.

A4. Review Scenarios

The scenarios should be reviewed for logic and accuracy by the team. Ideally team members 

who did not develop a particular scenario should perform the review to maintain a level of 

independence. However a training instructor or another plant staff member familiar with plant 

operations should be one of the reviewers. As part of the review, the plant operating procedures 

that will need to be accessed should be used to review the scenario and should be made 

available for the exercise.

A5. Conduct Walkthrough of Scenarios on the Simulator

To verify that the scenario can be performed as expected, that needed observations can be 

performed and data can be obtained, and that procedural activities are likely to occur as 

expected, conduct a walkthrough of the scenario on the simulator to determine plant response 

times with and without operator intervention. Verify that the procedures that will be used during 

the study are available and appropriate. Confirm availability of simulator logs to be captured 

throughout the exercise. If possible, the walkthrough may include operators who were not 

involved in scenario development and will not participate in the data collection to obtain 

independent input on response times and other demands.

A6. Determine Plant Parameters to be Recorded

The plant parameters that need to be recorded are those that will be affected by the events of 

the scenario and that reflect crew management of the plant. It is especially important to collect 

control signal inputs and plant parameters that reflect crew performance of critical actions. 

Ideally, this includes a list of predetermined plant parameters that are recorded by the simulator 

computer.

The specific simulated events, plant systems affected, and crew control actions anticipated and 

required also will determine the plant parameters that need to be collected. This determination 

needs to be made for all scenarios that will be performed, alleviating the need for multiple lists 

and possible omissions of monitored parameters in individual scenarios. Along with recording 

plant parameters, the simulator computer will produce an event log that consists of the times 

annunciators alarm and clear, valve and breaker switch positions, and when any control signals 

are adjusted (i.e., flow controls or pressure controls).

A7. Prepare forms needed for data collection

The data collection team should prepare a set of forms and documents based on the methods 

of data collection selected prior to collecting data. The purpose of these forms and protocols is
24

to ensure uniformity and consistency in data collection, completeness in the record of information 

that is generated, and for archival purposes. Five types of forms are described in more detail 

below: informed consent form, forms for observations, forms for recording deviations and errors,

PSF questionnaire, and debriefing form.

a. Informed consent form

Informed consent needs to be obtained to ensure all the participants in data 

collection activities are well informed about the study in which they will participate 

and that they are informed about the rights and responsibilities of each party during

the study.

Different countries have different requirements for informed consent and information 

disclosure. Data collectors should check local requirements.

b. Forms for observations of operator and crew actions

For purposes of data collection, forms should be provided to observers to use in 

documenting observations of crew performance. The types of observations may 

vary, depending on the scenario and the purposes of data collection. However, it is

typical that some form of event log that is time-based is used to record observations, 

including: (1) key actions performed; (2) alarms and notifications; (3) safety and 

protection system actuations; (4) crew communications, briefings, internal and 

external announcements; and (5) decisions and announcements (e.g., use of 

procedures, diagnosis of an event, etc.). Some of these can be corroborated through 

event logs from the simulator-based computer and data logging system. The form

should contain the crew identifier, the scenario identifier, the date, and observer

identifier.

c. Forms for deviations, errors, failures and successes

A form for recording deviations, errors, failures and successes also should be 

developed. The definitions for each are as follows:

Deviation – a departure from the expected course of action (as proceduralized, 

trained or directed).

Error – any slip, lapse, or mistake that does not meet the required standard of 

performance for a task or activity (e.g., failing to transition to the appropriate 

procedure or overlooking an indicator as directed by procedure).

Failure – failure to meet the performance standard for an action or sequence of 

actions that result in a functional failure as modeled in the PRA.

Success – performance of a required action or set of actions in a manner conforming 

to the performance standard established by the plant-specific PRA.

An example of the information that should be captured by this form is as follows:

• Observation (i.e., crew failed to trip all reactor coolant pumps (RCPs) prior to 

initiation of F&B)

• Classification of the event as a deviation, error, or failure based on the definitions
25

• Description of crew actions prior to and following the identified deviation, error, or 

failure

• Recovery actions if any (i.e., the crew tripped the RCPs when the failure to do so 

per procedural guidance was identified by the unit supervisor)

• Consequences of the identified departure, error, or failure if any.

The documentation of deviations, errors, and failures ought to be sufficiently 

descriptive so that a knowledgeable reader is able to understand what occurred (or 

failed to occur) and how it failed to conform to PRA-based requirements or other 

performance expectations. The taxonomy used for describing noteworthy behavior by 

the crew ought to be capable of providing useful information about decisions and 

actions of the crew that relate to the data collection objective(s) and support the 

development of insights about human reliability.

d. Performance drivers

A form may be developed that identifies the performance drivers that the data 

collection team wishes to observe or collect data about. A number of different 

approaches may and have been used to collect this type of data in human 

performance studies. In cases where performance drivers are operationally defined 

and manipulated in a controlled manner (i.e. according to a pre-defined plan), then

this information should be provided. It is more often the case that performance drivers 

vary dynamically, especially for scenarios that are delivered as a part of training. This 

may necessitate observations by subject matter experts that are performed 

qualitatively. Performance drivers for which metrics exist (such as workload or situation 

awareness) or for which physiological measures exist, should be used where 

appropriate to provide an objective measure of the performance driver. An example of 

such an evaluation form is provided in Appendix D.

e. Debriefing form

A debriefing questionnaire also needs to be developed, covering the different elements

of plant operation. Table 2 describes suggested subject areas with possible questions

to be discussed with crewmembers. See the debriefing form in Appendix F for details

of potential questions and content.

A8. Verify Crews’ Availability

Working with the facility, develop a schedule that will allow the operating crews to be available 

for performing the selected scenarios. Ensure that the simulator and the staff are prepared and 

the scenarios to be run are operational. Ensure that the operating crews are aware of the times 

that they need to be available for the data collection in advance and receive reminders. Check 

with the facility to ensure that participation in the data collection does not conflict with working 

hour limitations.
26

A9. Conduct Scenario Dry Run

Conduct a rehearsal of the actual data collection for each of the simulator scenarios using plant 

training staff or other plant staff knowledgeable of plant operations as crew members and 

evaluators to validate each of the scenarios. This includes the following:

• Identifying simulator instructor roles

• Re-verifying scenario event times

• Re-verifying procedure usage and operator anticipated actions

• Establishing data collector roles and responsibilities and identifying data collector 

positions in the simulator control room to minimize interference with the crew members
27

5. DATA COLLECTION

This section provides additional guidance on the preparation steps listed in Items B0 through B4 

in Table 1.

B0. Conduct scenario exercises

Prior to the start of the simulator scenario run, the plant operating crew should be introduced to 

the data collection team who will explain the purpose of the exercise. If this distinction is not 

made, the mindset of the crew may affect the information that the data collection team will 

obtain from the crew.

The facility training staff or simulator instructor will perform the crew briefing and provide the 

crew with all the necessary operational information needed to take the watch. The conduct of

the simulator scenario will be controlled by the facility training staff representative(s). During this 

introductory meeting, any necessary documentation should be completed, such as consent forms

can be signed and/or information needed about the crew makeup can be obtained.

The scenario will be initiated and the data collector(s) will record events and actions. The 

scenario must be allowed to run to the identified end point or until the lead data collector 

terminates the scenario.

B1. Record Real Time Observation Data

Data collection can be accomplished by several methods. Data may be collected electronically. 

The simulator computer can collect plant parameters and record their changes over time. The 

plant parameters that need to be recorded are those that will be affected by the events of the 

scenario and that reflect crew management of the plant. It is especially important to collect 

control signal inputs and plant parameters that reflect crew performance of critical actions. 

Besides the simulator computer recordings, audio and video records also may be available to 

record operator conversations and actions.

Another set of data that should be collected is direct observation by the data collection team. 

Direct observations can be accomplished by observers recording events or actions that may or 

may not be recorded by the simulator computer, such as orders given to operators (e.g., trip 

reactor coolant pumps), when plant procedures are entered and exited, or crew briefings. The 

direct observation of crews allows the data collection team to obtain an insight of the crew’s 

interactions, communications, and witness the crew’s teamwork. Ideally, three observers would 

be present: one to observe primary plant operations, one to observe balance of plant operations,

and one to monitor the crew supervisors. The shift technical advisor can be observed by any

one of the three observers depending on her/his work location.

While observing the crew during the scenario run, the observers should refrain from interactions 

with any of the crew members or from being an obstruction to crew movement. The observers 

should select positions in the control room that are unobtrusive but from where crew member 

actions can be readily observed. A simple form to record the time of the action and a description 

of the action can be used to record crew responses during the scenario run. An example of a 

completed log sheet is located in Appendix E.
28

a. Data structure

Data should be structured in a way that will facilitate completion of predetermined 

analyses or statistics (i.e., descriptive statistics, correlations, and variance). To begin

with, data collectors should recognize the inherent structure of the data imposed by

the scope of the data collection in terms of the following typical items:

• Number of scenarios

• Number of tasks per scenario

• Number of crews

• Number of participants per crew

• Crew characteristics

• Specific job positions within the crew (e.g., senior reactor operator, reactor 

operator, or shift supervisor)

• Scheduling information (times and dates)

• Total run time planned for scenarios

• Success and failure criteria.

Data collection could be designed to be multi-stage or hierarchical and, as a result, 

data may be aggregated, by scenario, by crew, by crew member, by task, and so on. 

This is important because the data collection format will affect the ease of analysis

for various forms of univariate or multivariate methods. It is recommended that a 

suitable data management tool be used to collect and manage the data. The data 

management tool also might provide forms, queries, and reports.

b. Observers and their Tasks

Observers should be limited to those directly involved in the exercise; additional 

personnel in the simulated control room should be minimized to avoid interference 

with, obstruction and distraction of the crew. Based on data collection needs, the 

number of data collectors should be identified. The following disciplines would 

typically be responsible for the data collection task:

• Qualified simulator training instructor

• Human reliability analyst

• Operations subject matter expert(s)

The tasks of the observers will be influenced to some extent by the methods and 

tools used to collect data. One primary issue in development of HRA data is that 

there are different ways to classify performance and differences in the terms that 

analysts may use. By relying on definitions of human performance measures and 

using data collection methods that are standardized, variability can be reduced. 

There is a need to develop tools suitable for the specific adaptation of these 

methods. This would include recording of the scenario timeline on pre-formatted or
29

free-formatted forms. Also, some observations can be recorded using computerbased tools, while actions of specific members of the crew (e.g., the shift supervisor 

and shift technical advisor) are recorded by hand. Video cameras and other nonintrusive recording devices also may be used for this purpose.

During information collection, it is necessary to avoid inappropriate interactions among the 

members of the observation team (e.g., laughing, joking, talking, or any behavior that could 

influence the validity and independence of data collected by individual observers).

All attempts should be made to avoid any interactions that may significantly influence the 

performance of the control room crew. Such interactions can influence crew behavior and the 

reliability of the observation data. All direct interaction with participating crew members should 

be prevented without exception. Any other action by observers that may influence crew 

performance and behavior should be avoided. Even eye contact should be avoided. The only 

acceptable interaction between observers and crew should be intentional communication by a 

representative of the data collection team with one or more participants at the start of the data 

collection about specific data collection conditions. Also it will be necessary to interact with the 

crew members when implementing some of the standardized tools for assessing workload or 

situation awareness. If possible, observers should observe the crew from a separate control 

station or observer area without being in the training simulator.

B2 Collect Performance Driver Information

Performance driver information may be collected during the scenario by trained observers and 

through debriefings and other data collection methods following the simulator scenario. One 

advantage of obtaining performance driver information directly from individual crew members 

prior to debriefing is that individual crew members’ subjective assessments are less likely to be 

affected by other crew member comments and information that they learn during the debriefing. 

Alternatively, a potential benefit to collecting performance driver information during crew 

debriefing is that the debriefing can refresh their memories of the events that occurred. The 

questions for this interview should be developed prior to the simulator runs. An example of one 

type of debriefing is found in Appendix F. The particular debriefing supported by the form in 

Appendix F is for individual crew member ratings of performance drivers (termed performance 

shaping factors in the example study) following completion of a scenario and prior to crew 

debriefing.

Structured discussions are very useful because they provide evidence about how critical 

mitigation actions are affected by performance drivers that manifest under certain plant 

conditions. Although feedback from crew members and trainers regarding performance drivers 

might be subjective, it represents their personal experience and is relevant to drawing 

conclusions about crew performance reliability.

a. Identification of Performance Drivers

For the purpose of simulator data collections, performance drivers are considered to 

be factors that, when present at the time a task is performed, change the likelihood of

successfully performing a task to a required performance standard. Based on this 

definition, performance drivers can enhance the likelihood of success or they may 

hinder it. The performance drivers considered in this way deal with “internal” and 

“external” influences.
30

External influences from the immediate task environment include factors such as 

presence of or quality of operating and emergency procedures, hours worked, 

breaks between work periods, and the quality and configuration of the work 

environment (e.g., control room layout, quality of the human-system interaction, or 

environmental conditions such as heat, noise, vibration, etc).

Internal influences deal primarily with personal attributes (such as skill, ability, 

attitude, knowledge and training) and perceptual and cognitive factors (such as 

memory; visual, auditory and kinesthetic perception; emotional state; workload; and 

situation awareness). Stress has special importance in HRA; it is described in terms 

of the physiological and psychological tension caused by a mismatch between 

internal and external influences. For example, if the perceptual requirements of the 

task environment become too demanding (e.g., due to visual complexity of the 

human-system interaction or lack of information), task loading will increase and 

performance will suffer.

Conversely, if task loading is very low resulting in boredom or fatigue, performance 

also may be degraded because physiological and psychological stress is too low to 

keep the operator alert (Blackman et al. 2008; Yerks and Dodson 1908).

b. Performance Driver Data

If quantitative data are generated by performance driver assessments (e.g., 

subjective ratings, evaluations by subject matter experts, etc.), it can be captured 

into a spreadsheet or a database.

The following items should be captured from the performance driver assessments:

• Identifier for the crew that participated in the study.

• The name or abbreviation of the scenario(s) that was included in the exercise.

• The title (or abbreviation) of the crew member(s) who participated in the exercise.

• The number of tasks that formed part of the scenario (as noted above, scenarios 

may not have the same number of tasks).

• An example of the type of form used to collect performance driver data in this 

study is shown in Appendix D.

B3. Conduct Scenario Debriefing

The final set of data to be collected is crew and other plant expert debriefings immediately 

following the simulator run. Debriefing sessions at the end of each scenario provide useful 

information about typical crew performance, their perceptions, and tendencies in managing 

different transients.

Directly following each simulator scenario run, each crew should be debriefed and asked the 

same predetermined questions regarding their performance. It is advisable to divide the crew 

into two groups: supervisors and operators. The shift technical advisor (STA), if participating in
31

the exercise can be in either group. Crew discussions are encouraged to facilitate identification 

of errors or areas of confusion.

Table 2 describes typical debriefing questions used for this portion of the crew evaluation.

Table 2 Typical debriefing questions

Type of Debriefing Question Examples

Diagnosis of Events and 

Conditions Based on Signals or 

Readings

1. What were the events that occurred during this 

scenario?

2. What information and materials did you use to 

first recognize and then diagnose the event and 

conditions? If you did not need materials to 

recognize the event, how did you know what it 

was?

3. What did you use to first recognize and then 

diagnose the event and conditions?

4. What information was communicated with the 

crew and how was it used?

5. What information did you then communicate to 

the crew and how was it used? (Was it used 

properly in your judgment?)

Understanding of Plant and

System Responses

1. Were there instances where your indications 

either helped or hindered you interpretations of 

plant status and conditions? (Identify the 

indications and how did they help or hinder?)

Adherence to and Use of

Procedures

1. Were there areas in the procedures that were 

ambiguous or difficult to apply for the conditions 

you encountered during the event? (Identify the 

procedure and the area, e.g., Step #.) Were 

there situations you found challenging to 

implement procedures as written? (Identify the 

procedure and the area, e.g., Step #.)

2. In your judgment, were there situations that the 

correct procedure was not being utilized? 

(Identify the procedure.)

Control Board Operations 1. Did you find any control actions challenging to 

execute in this event? (Which ones?)

2. What in your opinion makes these control 

actions challenging or difficult (such as 

controls/indications not fine enough?)

3. Did you make any mistakes or are there 

activities where mistakes are more likely to 

occur?

Crew Operations 1. Were there any time(s) that you were confused 

about the decisions or direction being taken 

during the event? What confused you?

2. How did you contribute to the decisions and 

directions of the crew’s actions?
32

3. Were you aware of the procedures and the 

course of action or goals of the supervisor? 

(How were you made aware of these?)

4. How were crew resources managed and 

directed during the event? In your judgment, 

were crew resources effectively managed or 

were there any wasted operations?
33

6. DATA ANALYSIS

This section provides additional guidance on the preparation steps listed in Items C0 through C1 

in Table 1. It addresses data collected in the reference study to illustrate and provide examples of 

applying the tenets and method for collecting human performance data.

C0 Analyze Results of Scenario Exercises

As soon as possible following completion of the scenario run and crew interviews and 

debriefings, each data collector should develop a synopsis for each scenario run. The synopsis 

should give a brief description of the crew’s performance during the scenario and whether or not 

the crew was successful in meeting the established mitigation criteria and performance standards

for critical actions. The data collector must determine or make a judgment regarding the success

or failure by the crew based on the criteria established by the data collection team, not

necessarily to training criteria. This is because all crews should have been trained to

mitigate all the events. The success criteria should be obtained from the plant’s PRA and 

operating procedures. The purpose of the synopsis is to provide a condensed, comprehensive 

view of what the data collector observed and include reminders to help evaluate a specific 

crew’s performance.

The data collector gets the time information from the simulator run clock. The success/failure 

criterion is established before the simulator scenario is run. As stated earlier, the synopsis is just 

a brief description of what occurred.

Below are two examples of synopses developed for two simulator scenarios:

Example 1 – Simulator Scenario Synopsis

The crew received the loss of main feedwater (MFW) indication at 2:14 into the

scenario. The crew tripped the Rx at 2:51 minutes into the scenario or approximately 37 

seconds after the loss of feed, which then gave the crew approximately 55 minutes to 

initiate F&B before core damage. RCS F&B was established 12 minutes and 40

seconds after the loss of feed, thereby satisfying the criterion for this event. The unit 

supervisor (US) skipped the step to trip the RCP and the shift technical advisor (STA) 

brought this omission to the attention of the crew and the RCPs were tripped 30 

seconds after initiating F&B. Auxiliary feedwater (AFW was re-established to the ‘B’ S/G

at approximately 20 minutes after the trip to the MFW Turbines. At 37:37 minutes into

the scenario, ‘B’ S/G level was at 78% and increasing with feed secured. Adverse 

containment condition declared at 38:48 minutes and feed was established to all S/Gs at

39:27 minutes. At 42:00 minutes steam generator tube rupture (SGTR) in the ‘B’ S/G 

was declared. At 1 hr and 20:20 minutes into the scenario natural circulation had been 

established with the RCS F&B. The procedure to isolate the S/G with the tube rupture 

was never entered because of higher priority procedures being executed. The faulted 

S/G was effectively isolated without the S/G PORV set point being elevated because 

RCS pressure was maintained less than the PORV setting. This prevented any 

uncontrolled release of Primary coolant to the atmosphere; therefore, satisfying the 

criteria for SGTR. All success criteria for this scenario were met by the crew.
34

Example 2 – Simulator Scenario Synopsis

The loss of MFW occurred at 2:04 minutes and the manual Rx trip was initiated at 2:36 

minutes, giving the crew approximately 60 minutes to initiate RCS F&B. At 12:26 minutes,

it was reported that all S/G WR indications were <50% (loss of heat sink indication). At 

15:23 minutes loss of heat sink red path was declared and FRH1 procedure was entered.

RCPs were stopped at 16:50 minutes, safety injection (SI) initiated at 17:00 minutes and 

the pressurizer (PZR) power-operated relief valves (PORVs) were opened at 18:21 

minutes, thereby initiating RCS F&B at 18:21 minutes meeting the F&B criterion. Reestablished feed to the ‘B’ S/G at 20:20 minutes (initiating SGTR in the “B” S/G), and at

26:15 minutes ‘B’ S/G level were report to be >14% narrow ridge (NR). The crew then 

commenced the process to secure RCS F&B by establishing

a maximum charging pump flow of 200 gallon per minute (gpm) at 30:26 minutes with 

one charging pump and securing ‘C’ High Head safety injection Pump (HHP). The crew 

then started the second charging pump at 34:35 minutes and stopped ‘B’ HHP at 35:50 

minutes. Flow to the ‘B’ S/G was throttled at 34:50 minutes and secured at 37:11 

minutes, with feedwater flow to all four S/Gs being established at 36:45 minutes. The 

crew then closed the ‘B’ PZR PORV at 39:00 minutes and the ‘A’ PZR PORV at 41:10 

minutes, stopping RCS F&B. Emergency Operating Procedure: Functional Recovery “H1” 

Loss of Heat Sink (FRH1) was properly executed until the last high head pump was to be 

secured, at which time the step was misinterpreted and the RCS PORVs were closed 

without securing the third HHP. The shift technical advisor (STA) assumed that the step, 

which stated if any active loop temperature was >405 F, the loop to determine the 

temperature was the D loop; however, the only active loop at the time was the B

loop. The crew may have assumed that an intact S/G on the loop was the same as an 

active loop. The S/G must have a water level of >14% NR and NC has been established.

(‘D’ loop did not meet this criterion.) It was reported at 41:50 minutes that the ‘B’ S/G 

level was still increasing and at 46:54 minutes the ‘B’ S/G level was >100% NR and at 

82.4% WR. At 48:49 minutes the ‘B’ S/G steam line radiation levels were reported as

high and increasing. At 49:19 minutes the Emergency Procedure E30 for isolations of a 

faulted S/G was entered. At 55:57 minutes the ‘B’ S/G is isolated. However, at 1 hr and 

00:56 minutes the ‘B’ S/G PORV opens, releasing primary coolant to the atmosphere,

the RCS and ‘B’ S/G are now in a solid pressure condition with the RCS and ‘B’ S/G 

pressures cycling. RCS pressure is still at 1,331 psig at 1 hr and 5:38 minutes. A 100 

F/hr cool down is started at 1 hr and 16:43 minutes. At 1 hr and 18:57

minutes the ‘B’ S/G PORV is stuck open and the running HHP has flow of approximately

900 gpm. At 1 hr 25:42 minutes, the ‘B’ S/G PORV is manually isolated. This crew was 

successful in initiating the RCS F&B in the time allotted to prevent core damage and was 

successful in re-establishing feedwater flow to the S/Gs. This crew was not successful in 

meeting the SGTR criteria because the crew was not able to maintain the RCS pressure 

less than the S/G PORV setting, thereby allowing primary coolant to escape to the 

atmosphere.

While developing the synopsis a list of deviations, errors, and failures should be generated. 

Below are three examples of how to document deviations, errors, and failures.
35

Observation: Water was allowed to reach the steam lines on the faulted S/G

Deviation Error Failure

Description: Because of the delay 

in securing the F&B and 

subcooling the ‘B’ S/G, the water 

level reached the steam lines.

X

Recovery: Following stopping of the F&B, the crew 

maintained the RCS pressure less than the S/G PORV set 

point, thereby preventing a release of primary coolant to 

the atmosphere.

Consequence: The water in the steam lines could have 

damaged the piping and main steam isolation valve 

(MSIV); if the S/G had gone solid, the PORV set point 

could have been reached and allowed primary coolant to 

be released to the atmosphere.

Observation: Crew failed to stop all 4 RCP prior to initiating F&B.

Deviation Error Failure Description: Crew initiated SI and 

opened PZR PORVs, initiating 

F&B with the RCPs running. The

STA questioned the RCPs running 

and then the US ordered the RCP 

tripped. The reactor operator (RO) 

tripped all 4 RCPs approximately 1 

minute after F&B was initiated.

X

Recovery: Crew tripped all 4 RCPs and continued on with 

the procedure.

Consequence: Missed a procedure step. The RCP running 

without sufficient subcooling will result in cavitation in the 

RCP impellers, causing erosion of the pumps and possible 

damage.

Observation: Crew did not identify the loss of component cooling water (CCW) pump and 

charging pumps not running (loss of seal injection/cooling and loss of RCP cooling).

Deviation Error Failure Description: The crew did not 

recognize the loss of RCP seal 

injection and cooling when the Rx 

was tripped following the 

distribution panel fault and the C 

emergency diesel generator 

(EDG) did not supply the ‘C’ 

engineering safety feature (ESF) 

bus. The reactor operator (RO) 

reported no charging pumps 

running and the RCPs were not 

tripped for another 1:40 minutes.

The crew is required by procedure 

to trip the RCPs within 1 minute 

following loss of seal cooling/flow 

and re-establish injection before 

seal temperatures reach 230°F.

X

Recovery: No recovery was possible. The crew needed to 

recognize the loss and initiate seal injection prior to 

reaching 230°F seal temperature; this did not occur.

Consequence: By not identifying and reporting that no 

CCW pumps or charging pumps were running, the crew 

was not aware of the loss of cooling to the RCP seal. This 

failure will result in a possible loss-of-coolant accident 

(LOCA) if the seals fail. In this scenario the seals were 

failed when RCP seal temperature exceeded their 

temperature limit.

C1 Analyze Crew Performance Results

To understand how the crew managed the simulated event, it is important to plot the pertinent 

plant parameters that were directly affected by the simulated accident (e.g., core exit
36

temperature for a total LOFW accident, S/G levels and pressures with reactor coolant pressure 

for a SGTR, etc.). To further understand how the simulated accident was managed, insert time 

lines for key operator actions (i.e., when reactor coolant pumps were tripped, safety injection 

pumps were started, etc). To aid the HRA team in their evaluations of crew reliability, plant 

parameters may be plotted in order to show how close they came to event limits (i.e., S/G and 

RCS pressures as they relate to the S/G power-operated relief set point for a SGTR).

For purposes of supporting data exchange, standardized templates that graphically depict the 

key events, plant response, and crew actions should be developed for different classes of 

design basis events. The graphical format in Figure 3 provides an example of a standard 

template that could be used to depict the results of data collection. The graph contains both 

parameters and a key to their interpretation for ease of use. It includes information related to

event initiation and plant response, plant parameters that show operator action (e.g., core power 

showing when the crew tripped the reactor, when F&B was initiated, etc.), and the available 

margins for crew response. This latter category of information is vital to determining whether the 

crew’s performance met the success criteria established for the scenario. It also shows the 

margin between crew response (i.e., when F&B initiated, RCS pressure, and temperature) and 

the limit state. In addition, these data are used to show the relationship between the available 

margin and the margin used by the crew, and provides a view of the distance of the crew’s 

response from the limit state as required to prevent more significant damage.

Figure 3 Example summary of crew response characterization plot

Refer to Appendix A for examples of other graphical data analyses developed from simulator 

scenario runs.

C2 Performance Driver Analysis

Unlike crew performance data collected during the scenario run, subjective performance driver 

data are typically collected after completion of a scenario run.
37

After the data have been collected, the analysis of responses concerning performance drivers 

should focus on answering the following questions.

At the crew level:

• Is there evidence of the influence of performance drivers?

• Are the influences of performance drivers observed on all members of a crew or only 

individual members?

• Are the effects of performance drivers positive or negative in relation to the critical tasks 

and decisions necessary for successful performance?

• Are the influences of the performance drivers manifest in the observed performance of 

the crew?

At the scenario level:

• Do performance drivers consistently influence performance of crews?

• Is there evidence of systematic variability that can be related to the influence of the 

performance drivers?

• Are there differences in outcomes (i.e., success or failure) as an apparent result of some 

performance drivers?

Figure 4 provides an example of data collected related to performance drivers. In the case of 

these data, the individual performance drivers were referred to in the original study as 

performance shaping factors or PSFs, as shown in the title of the graph. An approach to obtain 

feedback from crew members following each scenario was employed. This entailed crew 

members rating eight PSFs in terms of their perceived influence on their performance. This was 

done by anchoring a rating scale with values ranging from 1 to 5 with terms indicating that the 

PSF hindered their performance or helped them, respectively. The six lines on the graph 

correspond to six critical mitigation actions (two in each of three separate scenarios). The graphic 

trends of ratings were used to determine whether crewmembers perceived certain types of 

performance drivers affecting their performance on important crew actions. These ratings were 

also used to help focus discussions during crew debriefings and to identify potential issues for 

exploration during data analysis.
38

Figure 4 Example of numerical ratings summary of performance drivers

C3 Limit State Analyses

Figure 5 shows the performance of crews in reference to a limit state for a prescribed operator 

action. In the case of this particular action, available time corresponded directly to the limiting 

plant thermal-hydraulic behavior and was used to define the human performance standard for 

the PRA event sequence. If a crew used 100% or more of the available time to complete the 

critical mitigation actions, then crew performance would be considered failure, in terms of the 

plant-specific PRA. If less time was used than was available, the crew was successful in 

mitigating the event.

Two important features about this method for measuring the quality of crew response are: 1) the 

distribution of crew performance on the criteria related to the limit state, and 2) the variability in 

crew response relative to the limit state. The performance criteria for human actions are specific 

to the event sequence and scenario. Even though the same human actions are required in 

different event sequences (e.g., ‘F&B’ core cooling), the specific criterion that define success for 

the human action is scenario specific. In the case of the F&B actions, the initiating event, initial 

plant conditions, specific vendor and plant design, core power history, and other factors will 

determine the response requirements for completing F&B actions in every specific scenario of 

event sequences containing those human actions. Identifying these requirements in each 

scenario context is the first step of establishing the measure of success and failure for the

critical human action. It establishes the threshold for operator response and specific criteria that 

must be met (e.g., 4 out of 4 safety injection pumps operating at maximum flow within X minutes 

of reactor trip). Using observed crew performance during the simulator runs of the scenario and 

the defined threshold for performance, the data show the response of individual crews against the

maximum available response range. This method of measurement produces a distribution of 

crew performance relative to the limit state that distinguishes between successful and 

unsuccessful performance.
39

The distribution of crew response on the critical action reveals the margin between crew 

response and the limit state. The interpretation of this may be relatively straightforward: the 

amount of available margin is an indicator of how near the crew was to failing to meet the 

performance criteria for the scenario.

The second feature that is important from these data is the variability in crew response. This 

variability reflects inter-crew differences in task execution, procedure use, and decision-making, 

all of which are inter-related. It also reflects uncertainty in the results of crew performance that 

may have implications for the reliability and consistency in crew response to the initiating 

event(s).

An example of a limit state analysis follows. For a Total Loss of Feedwater (LOFW) simulated 

scenario, crews responded to the loss of heat sink conditions as prescribed by the emergency 

operating procedures (EOPs). In all cases, crews successfully initiated feed and bleed according 

to the plant-specific performance standards for the action and the event sequence as carried out 

in their full scope training simulator.

Using the plant-specific PRA and thermal-hydraulic analyses performed for this simulated event, 

a maximum allowable time to initiate core cooling using “feed and bleed” techniques was 

calculated for each crew. The analysis of crew performance in

Figure 5 below plots the proportion of available time used by each crew to accomplish the 

actions needed to establish feed and bleed core cooling based upon this limit state analysis.

Figure 5 Distribution of percentage of available time used by crews to initiate feed 

and bleed

In addition to showing the number of successful human actions by crews, the results also 

demonstrate some variability with regard to individual crew performance of the critical action. 

Three of the crews used less than 50 percent of the available time to initiate RCS F&B while 

one crew used 60 percent of the available time. This variability may have some meaning in 

terms of a distribution of success and spare capacity for this action.
40

Figure 6 Crew response times from loss of Component Cooling Water to tripping the

Reactor Coolant Pumps

Figure 6 shows an example of crew actions in which failure to perform an action to the 

prescribed standard occurred. In this case, a procedural time limit is prescribed to trip the

reactor coolant pumps in response to a loss of component cooling water, which supplies cooling 

water to the seal injection for each of the reactor coolant pumps. None of the four crews who 

participated carried out the prescribed action within the prescribed time limit, risking damage to 

the RCP seals and potentially leading to a simulated RCP seal loss of coolant. The design of

this particular scenario employed an initiating event which masked the occurrence of the 

subsequent loss of component cooling water. These results were used to study the effects of

“masking” on subsequent event diagnosis and response. This figure shows how comparison of

crew performance with a prescribed limit state illustrates the success or failure of operator 

actions with regard to a reference PRA prescribed action.

In both examples shown here, time served as a suitable surrogate measure of the underlying 

thermal-hydraulic phenomenon of interest. In many cases, the plant parameter itself may be 

directly translated as the limit state. For example, many limits in the plant PRA may be based on 

pressure, temperature, or some other measured parameter. In such cases, crew performance 

may be transposed on a graph that shows their performance relative to the measured variable

or parameter of direct interest.

These types of data analyses or transformations are employed in order to normalize the raw 

performance measurement in terms of the limit state. This means that the raw performance 

measures are adjusted using their relationship to the limit state on a notionally common scale. 

In the case of the data in

Figure 5 the performance data reflect the percent of available time used by each crew for the 

critical action in reference to the available time defined by the limit state for the human failure 

event. This permits insights to be drawn regarding three properties related to performance 

reliability:

• Whether the actions meet the success criteria for the defined HFE
41

• The amount of margin available between their performance and the limit state for the 

action as defined by the HFE

• Variability among crews in performing the action(s).

The process of normalization, whether using raw data or transformed data is intended to relate 

observed performance to the system limit state and provide a reference that can be used to 

show performance relative to success and failure criteria alone. There may be many differences 

between success and failure criteria for accident sequence criteria among different plants and 

plant types. Normalization and appropriate transformations are ways that data collected from

different facilities may be treated and analyzed to permit comparison and potentially aggregation 

across data collection sites

43

7. REPORTING

Previous sections described the basic data collection and analysis method. Unless the results 

are documented comprehensively and coherently, much of that effort would be wasted. 

Following is a brief outline of a typical report that would provide, not only useful information, but 

would also ensure traceability and repeatability for future studies.

The following essential items should be included in the report:

1. Scenario descriptions. This includes all necessary detail on:

• PRA basis for the scenario

• Plant parameters recorded

• HFEs

• Data collection (or operational) procedures used

• Success/failure criteria for the data collection

• PSFs

2. Crew composition

3. Scenario analysis results, including:

• Timeline for all scenarios indicated to the nearest second, as appropriate

• Malfunctions inserted into the simulation, their timing, as well as crew-dependent 

malfunctions.

• Crew responses, linked to the timeline of simulated plant performance including 

procedures entered and noteworthy decisions and procedural transitions

• Summary of crew performance on critical tasks noting successes, failures, and 

noteworthy deviations or exceptions to expected performance

• Limit state analysis

• Analyses related to the human performance measures collected

• Performance driver insights and results

4. Summary and Conclusions

45

8. SUMMARY AND CONCLUSIONS

One of the aims of this research was to develop and describe a systematic approach to data 

collection involving human performance in PRA-relevant conditions that would provide useful 

data about human performance reliability, such as it is used in the field of human reliability 

analysis. An important part of this is to describe the principles of data collection to ensure that 

key performance aspects of interest related to the PRA are observed and measured with the 

intent of drawing insights about the reliability and tendencies of human performance with respect

to standard criteria based on PRA success and failure criteria. A longer-term goal is to amass a 

quantity of data that allows more general insights to be drawn about human performance, 

supporting more formal estimation of reliability-based parameters needed for use in PRA and 

other risk-informed applications. This may include the distribution of successful and 

unsuccessful crew performance on similar tasks collected at different plants that have been 

standardized to ensure their measurement of critical task performance parameters is similar 

(e.g., success and failure). The use of a limit state approach as demonstrated here may be one 

way to achieve such data collection and aggregation on human performance criteria of interest 

to PRA.

The process of normalizing the raw human performance data as demonstrated is a key aspect to 

achieving this goal. Plant specific differences may preclude aggregation of data, in the raw, due 

to differences in plant response and the specific requirements to achieve success in crew 

performance. The normalized measurement of performance around the limit state with sufficient 

data may permit more direct assessments of human performance reliability. In addition, the 

measure provides an ability to observe variability by different crews on established criteria for 

performance. A potential useful activity may be to see whether trends in variability in crew 

performance may reflect characteristics of performance in different PRA relevant simulated 

contexts that can be correlated with success and failure or demand parameters of the 

performance condition that drive performance reliability.

This report recognizes that data from nuclear power plant (NPP) simulator environments reflect 

important aspects of human performance and potentially causes of human error. However, the 

information collected is constrained by the uniqueness of each particular plant and design of 

each particular simulator study. Thus, the ability to apply the findings of a given simulator study 

to other studies of human performance at NPPs is often limited. The report emphasizes that a 

well-defined approach to experimental design, a common language or measurement technique 

for describing and classifying errors and human actions, and an associated theoretical 

underpinning for the design and resulting statistical analyses are needed to allow extrapolation 

of the results of simulator studies across plant designs, crew make-ups, and scenarios.

The report goes on to propose an approach for collecting human performance data from NPP 

simulators employing the reliability engineering concept of limit state, to describe the process for 

collecting data, and to present illustrative examples of data analyses. The focus of the approach 

borrows from techniques applied in traditional reliability engineering (RE) while using 

experimental techniques derived from the behavioral sciences.

The ultimate aim of the proposed approach is to achieve greater consensus, consistency and 

convergence of human reliability analysis (HRA) methods by stimulating simulator data 

exchange, greater communication, and review within the HRA research and practitioner 

community. The proposed approach is intended to yield data that strengthens the technical 

bases for HRA methods and their applications.
46

Because of its relationship with other disciplines, an HRA data generation approach must reflect 

critical aspects of the underlying scientific and methodological requirements of the scientific 

disciplines upon which it is based. Since HRA exists at a nexus between three distinct 

disciplines: (1) the behavioral sciences (i.e., human factors, ergonomics, cognitive sciences,

and psychology); (2) reliability engineering; and (3) probabilistic risk assessment, it must draw 

on the tradition of scientific development and theoretical underpinnings that reflect the formal 

requirements for, and use of information to address issues to which the discipline is applied.

This tradition includes the use of formal experimental design, protocols for data collection, 

methods for the analysis and interpretation of data, development of models to integrate results,

development of codes and standards, and so forth. This also applies to the generation of data to 

support the development and validation of models of human reliability used in systems

analyses, and the application of data from experiments, studies, simulation, or other sources to 

predict human reliability values.

RE analysis assesses the performance of discrete system elements, having well-characterized 

governing dynamics (e.g., mechanical, electrical, chemical, etc.), interacting with other discrete 

elements under well-described conditions. The interactions among components in a mechanical 

system are straightforward to characterize. This includes the manner in which they receive and 

process inputs from other components, the manner in which they produce outputs, and the 

specific outputs they produce.

The report discusses how human performance is difficult to subject the same form reliability 

analysis that is done for electro mechanical systems. Though the behavioral sciences are 

comprised of an extensive body of literature and knowledge, this knowledge is not as easy to 

employ in studies of human reliability analysis as engineered system characteristics are in the 

analysis of systems reliability. Human capabilities are far more extensive, complex, and 

interconnected; therefore, they are not as completely understood as the capabilities of electromechanical devices. Developing functional models as well as testing and refining theories about 

these human related processes are vital to shaping our knowledge and expectations about 

human reliability.

The ability to study a structure, system, or component to failure is vital to the study of its reliability

and the ability to do this has been perfected in RE. However, the principles of such an approach

are not as mature for the field of human reliability. One of the hallmark characteristics of the RE

discipline is the ability to ‘test to failure,’ by creating system failure conditions which can be

observed and to collect data that can be used to establish failure rates for various failure types.

Validated models using such data can describe the physics of failure of electro- mechanical

devices or nuclear energy systems.

Studies of humans that enable observations of failure are needed to develop and validate models

for human reliability. Unfortunately, failures in human performance that would be of interest to

HRA are unlikely to be observed. The HFEs modeled in a PRA, are events that result from

unique combinations of failures (i.e., initiating events) and plant conditions and are very plant

specific. Few studies of human failures designed to manifest these conditions in nuclear power

plant control rooms have been conducted and reported such that the ability to draw general

insights about failure rates and types is limited. This underscores the need for more directly

relevant sources of data about human performance that can be used to derive information about

the limits of operator performance or capabilities in NPPs.
47

To summarize, the ability to collect data on the reliability of human performance in complex 

environments is much more difficult than it is to collect data for many of electro-mechanical 

devices employed in nuclear energy production and other industrial systems. What is needed 

are:

• Opportunities to collect data about human performance in which success and failure are

defined and measured.

• Theories of human performance and human reliability that are sufficiently mature to 

support HRA model and method development along with sources of data to test and 

validate the models of human cognition and behavior that are the basis of HRA methods.

The approach proposed in this report could be used to provide a common language for human 

failures (limit state) and human performance within HRA studies that would allow data from one 

HRA context to be extrapolated to a new context.

Collecting simulator data to support HRA requires robust experimental design and process. Five 

essential experimental design characteristics are described that are based on paradigms from the

behavioral sciences that should be considered for robust experimental design that should be 

addressed when preparing to collect data for application in HRA. The section includes 

recommendations on categories of information to collect when performing simulator studies.

The five experimental design characteristics that are proposed in this report are:

• Performance measures should relate to success and failure as defined in the PRA

• Simulator contexts should be representative of PRA events

• Data collection should include the opportunity to observe human failure events of 

interest

• Data collection should include objective human performance measures of success and 

failure

• Data collected should describe relevant aspects of performance and the factors that 

affect that performance

HRA methods consider conditions that may be found in probabilistic safety analysis contexts to 

predict the kinds of errors (e.g., slips, lapses, or mistakes) and the likelihood of those errors (i.e.,

probability) that can occur. There is substantial experimental and experiential evidence

about the many factors that may be ‘drivers’ of human reliability in PRA contexts. These include 

factors such as stress, workload, human-system interface technologies, training and

experience, crew resource management, intra-team dynamics, and others.

The report goes on to describe in detail specific process steps to performing an experiment 

following a robust design that focuses on collecting HRA relevant data from an NPP simulator 

using PRA significant events while collecting success and failure data depicted by limit state 

criteria. The process of conducting an HRA simulator data collection effort can be described in 

four main phases: preparation, data collection, data analysis, and reporting. These four phases 

are further defined by 19 sub-steps. The data collection section presents different methods to be 

used for data collection and also describes how to structure the collection effort. In the data
48

analysis section, examples of how to develop synopses and how to document deviations and 

errors are provided. There is also a description of how to analyze performance drivers in this 

section.

The report discusses, in detail, the concept of limit state as a generic approach to define 

success and failure criteria so that the data can be used across HRA methods. Using the plantspecific PRA, success and failure criteria can be developed for each planned scenario. Limit 

state from a RE and systems safety perspective denotes a region of performance that 

distinguishes between performance success and failure. An analysis of crew performance data 

permits a determination to be made regarding success of the crew in terms of the requirements 

of the PRA, to minimize the potential for subjectivity in evaluating crew performance.

Two important features about this method for measuring the quality of crew response are: (1) the 

distribution of crew performance on the criteria related to the limit state, and (2) the variability in 

crew response relative to the limit state. The performance criteria for human actions are specific

to the event sequence and scenario. Even though the same human actions are required in 

different event sequences, the specific criteria that define success for the human action is 

scenario specific. Identifying the criteria in each scenario context is the first step of establishing

the measure of success and failure for the critical human action. Using observed crew 

performance during the simulator runs of the scenario and the defined threshold for performance, 

the data show the response of individual crews against the maximum available response range. 

This method of measurement produces a distribution of crew performance relative to the limit 

state and is an indicator of how near the crew was to failing to meet the performance criteria for

the scenario.

The second feature that is important from these data is the variability in crew response, which 

reflects inter-crew differences in task execution, procedure use, and decision-making, all of 

which are inter-related. It also reflects uncertainty in the results of crew performance that may 

have implications for the reliability and consistency in crew response to the initiating event(s).

These types of data analyses normalize the raw performance measurement in terms of the limit 

state. This means that the raw performance measures are adjusted using their relationship to 

the limit state as a notionally common scale. This permits insights to be drawn regarding three 

properties related to performance reliability:

• Whether the actions meet the success criteria for the defined HFE

• The amount of margin available between their performance and the limit state for the 

action as defined by the HFE

• Variability among crews in performing the action(s).

The process of normalization whether using raw data or transformed data relates observed 

performance to the system limit state and provides a reference that can be used to show 

performance relative to success and failure criteria alone. There may be many differences 

between success and failure criteria for accident sequence criteria among different plants and 

plant types. Normalization and appropriate transformations are ways that data collected from 

different facilities may be treated and analyzed to permit comparison and potentially aggregation 

across data collection sites.
49

In conclusion, this report (1) proposes a standardized approach for collecting human 

performance data from NPP simulators employing the reliability engineering concept of limit 

state using experimental techniques derived from the behavioral sciences, (2) describes a 

process for collecting data, and (3) presents illustrative examples of data analyses.

The report also describes how the approach can achieve greater consensus, consistency and 

convergence of HRA methods by stimulating simulator data exchange, greater communication, 

and review within the HRA research and practitioner community through the application of limit 

state as a means to normalize data collected from different facilities so that the data may be 

treated and analyzed to permit comparison and potentially aggregation across data collection 

sites.

The proposed approach is intended to yield data that strengthens the technical bases for HRA

methods and their application

51

9. REFERENCES

[1] “HRA Data and Recommended Actions to Support the Collection and Exchange of HRA 

Data,” CSNI WGRisk Report NEA/CSNI/R(2008)9, Nuclear Energy Agency, Committee 

on the Safety of Nuclear Installations, September 2008.

[2] Blackman, H. S., Gertman, D. I., and Boring, R. L., 2008, “Human Error Quantification 

Using Performance Shaping Factors in the SPAR-H Method,” in Proceedings of the 52nd 

Annual Meeting of the Human Factors and Ergonomics Society.

[3] Bye, A., Dang, V.N., Forester, J., Hildebrandt, M., Marble, J., Liao, H., and Lois, E.,

2012, “Overview and Preliminary Results of the US Empirical HRA Study,” in

Proceedings of the 11th International Probabilistic Safety Assessment and Management

Conference & The Annual European Safety and Reliability Conference. Finland, 25–29

June 2012.

[4] Chang, Y. J., & Lois, E., 2012. “Overview of the NRC’s HRA Data Program and Current 

Activities,” in Proceedings of the 11th International Probabilistic Safety Assessment and 

Management Conference & The Annual European Safety and Reliability Conference. 

Finland, 25–29 June 2012.

[5] Criscione, L., Shen, S.-H., Nowell, R., Egli, R., Chang, Y. J., & Koonce, T., 2012. 

“Overview of Licensed Operator Simulator Training Data and Use for HRA,” in 

Proceedings of the 11th International Probabilistic Safety Assessment and Management 

Conference & The Annual European Safety and Reliability Conference. Finland, 25–29

June 2012.

[6] Forester, J.A., Dang, V.N., Bye, A., Boring, R.L., Liao, H., and Lois, E., 2012, 

“Conclusions on Human Reliability Analysis (HRA) Methods from the International HRA 

Empirical Study,” in Proceedings of the 11th International Probabilistic Safety 

Assessment and Management Conference & The Annual European Safety and 

Reliability Conference. Finland, 25–29 June 2012.

[7] Groth, K. and Mosleh, A., 2009, “A Data-Informed Model Of Performance Shaping

Factors for Use in Human Reliability Analysis,” UMD.

[8] Hallbert, B. et al., 2007, “Modeling Causal Mechanisms and Context in HRA,” in The 

Employment of Empirical Data and Bayesian Methods in Human Reliability Analysis: A 

Feasibility Study, NUREG/CR-6949, U.S. NRC.

[9] Hallbert, B. and Morgan,T., 2010, “NRC Program N6798: Support WGRisk on HRA Data 

Collection and Exchange Using Simulators,” Proposed Methodology for Human 

Performance Data Collection, INL.

[10] Hallbert, B. P., Dang, V. N., and Lois, E., 2011, “Human Performance Research and Its 

Uses to Inform Human Reliability Analysis, In: Simulator-based Human Factors Studies 

Across 25 Years,” The History of the Halden Man-Machine Laboratory, A. B. Skjerveand 

A. Bye (Eds.), Springer-Verlag: London.
52

[11] Hendrickson, S. L., Parry, G.P., Forester, J.A., Dang, V.N., Whaley. A.M., Lewis, S., 

Lois, E., and Xing, J., (2012). “Towards an Improved HRA Method,” in Proceedings of 

the 11th International Probabilistic Safety Assessment and Management Conference & 

The Annual European Safety and Reliability Conference. Finland, 25–29 June 2012.

[12] Marble, J., Liao, H., Presley, M., Forester, J., Bye, A., Dang, V., and Lois, E., 2012, 

“Results and Insights Derived from the Intra-Method Comparisons of the US HRA 

Empirical Study,” in Proceedings of the 11th International Probabilistic Safety 

Assessment and Management Conference & The Annual European Safety and 

Reliability Conference. Finland, 25–29 June 2012.

[13] McCracken, J. H. and Aldrich, T. B., 1984, “Analysis of selected LHX mission functions: 

Implications for operator workload and system automation goals,” Technical note ASI

479-024-84(b), Fort Rucker, AL, Anacapa Sciences, Inc.

[14] O’Hara et al., 2004, “Human Factors Engineering Program Review Model,” NUREG/CR0711, Washington, D.C.: U.S. Nuclear Regulatory Commission.

[15] Parasuraman, R. and Rovira, E., 2005, “Workload Modeling and Workload Management: 

Recent Theoretical Developments,” Army Research Laboratory: Fairfax, VA.

[16] Sarno, K. and Wickens, C., 1992, “The Role of Multiple Resources in Predicting TimeSharing Efficiency: An Evaluation of Three Workload Models in a Multiple Task Setting,” 

Technical Report ARL-91-3/NASA A31-91-1, NASA AMES Research Centre, Moffett 

Field, California.

[17] Sternberg, R. J., 1996, Cognitive Psychology, Wadsworth: Belmont, CA.

[18] Swain, A. D. and Guttman, H. E., 1983, “Handbook of Human Reliability Analysis with 

Emphasis on Nuclear Power Plant Applications,” NUREC-CR-1278. Washington, D.C.: 

U.S. Nuclear Regulatory Commission.

[19] Vicente, K. J. and F. Tanabe, F., 1993, “Event-independent assessment of operator 

information requirements: Providing support for unanticipated events”, in Proceedings of 

the American Nuclear Society Topical Meeting on Nuclear Plant Instrumentation, Control 

and Man-Machine Interface Technologies (pp. 389-393), La Grange Park, IL: ANS.

[20] Whaley, A. M., Hendrickson, S. M. L., Boring, R. L., Joe, J. C., Le Blanc, K. L., & Xing, 

J., 2012. “Bridging Human Reliability Analysis and Psychology, Part 1: The 

Psychological Literature Review for the IDHEAS Method”, in Proceedings of the 11th 

International Probabilistic Safety Assessment and Management Conference & The 

Annual European Safety and Reliability Conference. Finland, 25–29 June 2012.

[21] Whaley, A. M., Hendrickson, S. M. L., Boring, R. L., & Xing, J., 2012. “Bridging Human 

Reliability Analysis and Psychology, Part 2: A Cognitive Framework to Support HRA”, in 

Proceedings of the 11th International Probabilistic Safety Assessment and Management 

Conference & The Annual European Safety and Reliability Conference. Finland, 25–29

June 2012.

[22] Wickens, C., 1984, “Processing resources in attention,” in R. Parasuraman and R.

Davies (Eds.), Varieties of Attention (pp. 63-101), Orlando, FL: Academic Press.
53

[23] Wickens, C., 2008, “Multiple Resources and Mental Workload,” Human Factors, Vol.

50, No. 3, June 2008, pp. 449–455.

[24] Williams, J. C., 1992, A User Manual for the HEART Human Reliability Assessment

Method, Nuclear Electric plc/DNV Technica: London.

[25] Yerks, R. M. and Dodson, J. D., 1908, “The relation of strength of stimulus to rapidity

of habit-formation,” Journal of Comparative Neurology & Psychology, Vol. 18: 459-482.

[26] Hendrickson, S. M. L., Parry, G., Forester, J., Dang, V. N., Whaley, A., Lewis, S.,

Lois, E., & Xing, J. (2012). Towards an Improved HRA Method. Proceedings of the

11th International Probabilistic Safety Assessment and Management Conference,

June 25-29, 2012, Helsinki, Finland

55

APPENDIX A – SCENARIO DATA ANALYSIS GRAPHS
56

Figure A-1 Example of a reactor feed and bleed analysis with expected crew performance demonstrated
57

Figure A-2 Example of a reactor feed and bleed analysis with a crew deviation noted
5

8

Figure A-3 Example analysis of steam generator tube rupture – expected performance demonstrated
59

Figure A-4 Example analysis of steam generator tube rupture and pressurizer – expected performance demonstrated
60

Figure A-5 Example analysis of steam generator tube rupture – performance deviation noted
61

Figure A-6 Example analysis of steam generator tube rupture and pressurizer – performance deviation 

noted
62

Figure A-7 Example analysis and results of crew performance of pressure management on SGTR with Limit State (S/G PORV 

Setpoint) superimposed
63

Figure A-8 Example analysis of crew performance – failure noted
64

Figure A-10 Example analysis of crew scenario management reflected in RCP seal temperature with limit state (RCP seal temperature limit) superimposed
65

Figure A-11 Example analysis of critical task performance: Time limit to reactor coolant pump trip - all crews with limit state (RCP seal temperature limit) superimposed

67

APPENDIX B – SIMULATOR SCENARIO DESCRIPTIONS
68

SLOFW/SGTR scenario

Scenario description

Situation from start

100% Power

S/U feedwater (FW) pump cannot start (not visible)

X-connection valve for auxiliary feed water pump (AFP) B cannot open

1. Total loss of feedwater

Events: 

Trip of MFW pumps

S/U FW pump cannot be started

AFWP 11 does not start

AFWP 12 starts and indicates flow, but the recirculation valve is open (does not show 

in the control room) so the indicated flow does not go to the S/G

AFWP 13 starts but there is no flow (shaft failure)

AFWP 14 starts but trips on over-speed

Consequences:

Level going down in all S/Gs

No flow to any S/G

Indicated flow to S/G B prevent the red entry to FRH1 to be present

Actions:

 When MFW stops, start FW booster pump and start S/U FW pump (does not start)

 Manually trip the reactor when there is no feed flow to the S/Gs (anticipate 

automatic trip)

 Immediate actions after trip

 Try to open X-connection valve from AFWP B, cannot be opened.

 After immediate actions, check if AFW needs to be reduced. Send field operators to 

check the following:

- Breaker for AFWP1

- X-connection valve on AFWP B

- Check valve line-up, why there is no flow

- Terry turbine, reset over-speed

 Transfer to ES01 and start monitoring CSF

 Detect that there is no AFW flow to S/G B (no flow to any S/G). There are trends on 

S/G WR levels in the back-panels.

 Decide to transfer to FRH1 procedure based on loss of secondary heat sink (no 

S/G level and no S/G feed flow). (They will probably make this decision in about 

10 minutes according to the training staff.)

 Start F&B according to procedure FRH1

Procedures:

• 0POP05-E0-E000, “Reactor Trip Or Safety Injection”

• 0POP05-E0-ES01, “Reactor Trip Response”

• 0POP05-E0-FRH10POP05-EO-FRH1, “Response to Loss of Secondary Heat Sink”

• 0POP04-FW-0002, “Steam Generator Feed Pump Trip”

2. Reestablish FW

Event: 

X-connection valve from AFWP B becomes available (about 20 minutes into the 

scenario). We will get the X-connection back as soon as possible after initiating F&B, 

so that the S/G levels do not drop too much (after they have reset SI and some other 

actions that they have to do).
69

Consequence: 

It is possible for the crew to establish FW to S/G A, B and C via the X-connection 

valves

Action:

 Reestablish FW to S/Gs

Procedure:

• FRH1

3. SGTR

Event: 

A tube rupture occurs in the first S/G that is feed via the X-connection valves, one 

minute after AFW flow is established.

Consequences: 

RCS water leaking into the ruptured S/G. The level in the ruptured S/G will rise (but 

this level rise may be masked by the fact that they are also feeding the S/G to reach)

 FRH1, stop F&B when NR level in one S/G is <34% (adverse containment 

condition)

 Go to ES11 (from FRH1 step 31, may not be possible if RCS pressure is not stable, 

then you may go to ES1); stop SI (may be done in ECA31, if E1 is used here)

 Go to FRP1, from the CIP in ES11. The FRP1 will take care of the SGTR, reducing 

the RCS pressure so that the leak flow to the ruptured S/G stops. (100% NR in about 

1h 15 min, WR continue to rise)

 From FRP1 to ES11; subcooling adequate

 From ES11 CIP to E3 (1h 20 min). Adjust S/G PORV set point to 1265. Steam line 

and feed flow already isolated. (Radiation alarm (yellow) at 1h 25 min.

 ECA31 from E3 step 6, based on S/G pressure (red radiation alarm)

Procedures:

• 0POP04-FW-0002, “Steam Generator Feed Pump Trip”

• 0POP05-E0-E000, “Reactor Trip Or Safety Injection”

• 0POP05-E0-ES01, “Reactor Trip Response”

• 0POP05-E0-F003, “Heat Sink Critical Safety Function Status Tree”

• 0POP05-E0-FRH10POP05-EO-FRH1, “Response to Loss of Secondary Heat Sink”

• 0POP05-E0-ES110POP05-EO-ES11, “SI Termination”

• 0POP05-E0-EO100POP05-EO-EO10, “Loss of Reactor or Secondary Coolant”

• 0POP05-E0-F004, “Critical Safety Function Status Tree”

• 0POP05-E0-FRP10POP05-EO-FRP1, “Response to Imminent Pressurized Thermal 

Shock Condition”

• 0POP05-E0-EO300POP05-EO-EO30, “Steam Generator Tube Rupture”

• 0POP05-E0-EC31, “SGTR with loss of reactor coolant – subcooled recovery desired”

Human Failure Event

HFE1: Start F&B. 

- 13% WR S/G level = hot dry S/G at STP

- 90 minutes (PRA core damage)

HFE2: Identify and isolate ruptured S/G

- top filled S/G

- time? 33 min from SGTR (= 75 minutes total)

HFE3: Stop SI to avoid/ stop S/G PORV release

- top filled S/G

- time? 33 min

The HFEs for SGTR (identify and isolate, and stop SI) can be done in a different order 

depending on the RCS pressure trend when leaving FRH1 (criteria to stop SI or not).
70

Procedures

• 0POP04-FW-0002, “Steam Generator Feed Pump Trip”

• 0POP05-E0-E000, “Reactor Trip Or Safety Injection”

• 0POP05-E0-ES01, “Reactor Trip Response”

• 0POP05-E0-FRH10POP05-EO-FRH1, “Response to Loss of Secondary Heat Sink”

• 0POP05-E0-EO100POP05-EO-EO10, “Loss of Reactor or Secondary Coolant”

• 0POP05-E0-ES110POP05-EO-ES11, “SI Termination”

• 0POP05-E0-FRP10POP05-EO-FRP1, “Response to Imminent Pressurized Thermal 

Shock Condition”

• 0POP05-E0-EO300POP05-EO-EO30, “Steam Generator Tube Rupture”

• 0POP05-E0-EC31, “SGTR with loss of reactor coolant – subcooled recovery desired”

Data collection

Hand out paper copies of the procedure that they use that can be collected after the run. The 

US writes in the procedure while working.

Remember: In the experimental protocol, decide when the crew will have responses from FOs 

and chemistry.

RCP Sealwater Scenario

Scenario Description

Situation from start

Facility has a lot of online maintenance. It is not unusual to have a train out of service for 

maintenance.

100% Power

“B” train out of service from start (condensate cooling pump (CCP) B out of service. CCP B is 

cooled by CCW)

1. Failure of distribution panel

Event:

Failure of distribution panel DP1201

Consequences:

Failure of controlling channels for A and B S/Gs.

Actions:

• The crew needs to take immediate actions (that they know from training). They don’t 

have time to take out the procedure in this situation. The US makes sure that the crew 

establishes control of the situation. (When control is established they may use the 

procedure, but in this situation they will not gain control, so they probably will not use 

the procedure.)

- Charging control valve in manual to control charging flow

- S/G A and B feed regulation valves in manual to control feedwater flow to S/Gs. 

(The valve will open fully in automatic.)

- Control rods in manual to prevent them from stepping in.

Procedures:

0POP04-VA-0001, “Loss of 120 VAC Class Vital Distribution”

2. Reactor trip on high S/G level

Event:

The A MFW control valve cannot be taken in manual, and the valve is fully open, 

feeding the S/G.

Consequence:

If the crew does not trip manually, there will be an automatic trip on high S/G level.
71

Actions:

• Trip reactor (because they have lost control of S/G level, and are approaching 

automatic trip)

• Do immediate actions and start E0

• AFW valves in manual (A and B) to control S/G level (A and B)

• Go to ES01

Procedures:

• 0POP05-E0-E000, “Reactor Trip Or Safety Injection”

• 0POP05-E0-ES01, “Reactor Trip Response”

Comment:

The reactor trip in combination with the failed distribution panel keep the crew busy 

and may mask the failure of CCW.

3. Loss of CCW and Sealwater

Events (happening at the same time as reactor trip):

- CCW pump 1A (that was the one running) trips

- The busbar E1C is de-energized (failure on busbar, preventing DG from starting)

Consequences:

- Loss of all CCW pumps (B pump out of service, A pump tripped, C pump deenergized)

- Loss of sealwater to RCPs because the running charging pump (CCPA) is powered 

from E1C. The other charging pump (CCPB) is out of service.

Actions:

• Stop diesel generator 13 (powered from E1C) because it has lost cooling, ECW. E-0 

addendum 5 (p40): step 7, verify ECW status, check pumps running, when load 

sequence is completed start pumps. If cannot be started trip the diesel. (Normally

crewmembers don’t wait that long.) The licensed operator has the discretion to take 

control, protect the diesel, and turn it off.

• Stop RCPs because cooling is lost (within1 minute according to procedure POP4-RC2 step 3 and CIP)

• Start PDP and establish sealwater (before sealwater inlet or lower sealwater bearing 

temperatures greater than 230, otherwise establish sealwater slowly) 

Procedure:

• OPOP4-RC-0002, “Reactor coolant Pump Off Normal”

• 0POP05-E0-ES01, “Reactor Trip Response” (ES01 step 6 and step 1 and addendum 

1 in POP4-RC-2: IF sealwater inlet OR lower sealwater bearing temperatures 

GREATER THAN OR EQUAL TO 230, THEN DO NOT establish seal injection to that 

RCP.)

Human Failure Event

HFE: Start PDP and close the recirculation valve to establish sealwater

- before 230°

F

- within 13 minutes

Procedure:

• 0POP04-VA-0001, “Loss Of 120 VAC Class Vital Distribution”

• 0POP04-RC-0002, “Reactor Coolant Pump Off Normal”

• 0POP05-E0-E000, “Reactor Trip Or Safety Injection”

• 0POP05-E0-ES0, “Reactor Trip Response”

• 0POP04-VA-0001, “Loss of 120 VAC Class Vital Distribution”

73

APPENDIX C – PERFORMANCE CRITERIA
74

LOFW/SGTR Scenario

1. Total Loss of Feedwater

Pass Criteria for Loss of All Feedwater: 

• Time to initiation of F&B must be below the curve. (Depending on the time of the Rx trip 

following a complete loss of feed to the S/Gs, the crew has between 90 and 13 minutes to 

initiate RCS F&B. [The sooner the Rx is tripped after the loss of Feed to the S/G the more 

water inventory (in the S/Gs) to remove decay heat from the Rx, thus giving them more 

time to initiate Feed and Bleed.]

• Successful RCS F&B is determined when at least one high-head safety injection) pump is 

running and both PZR PORVs isolation valves and both PZR PORVs are open and flow is 

verified (into and out of the RCS).

• (If PZR PORVs are not available, then the RX head vent must be open.)

− (Secondary Criteria) maintain containment pressure (If ≥5 psig, initiate or verify 

initiation of containment spray)(If ≤6.5 psig, verify containment spray may be stopped 

or stop containment spray.)

Note: Loss of heat sink is determined by:

S/G wide range level in at least two S/Gs less than 50% [73%] or pressurizer pressure greater 

than 2,335 psig.

2. Reestablish Feedwater

Pass criteria for reestablishing feedwater:

• Start AFWP ‘B’

• Open X connection to establish flow to S/Gs A, B, and/or C.

• Feed rates are maintained within limits.

a. FRH1 step 3.e. Check total AFW flow to S/Gs > 576 gpm. [≤675 gpm/pump]

b. AFW flow to dry S/G.

3. SGTR

Pass criteria for SGTR:

90

82

73

65

56

48

39

30

22

13

0

10

20

30

40

50

60

70

80

90

100

15 20 25 30 35 40 45 50 55 60

Time to Initiate F&B to Prevent Core Damage in 

Minutes

Time from Loss of Feed to Rx Trip in Seconds

Time to Feed & Bleed

Time to Feed & Bleed
75

• Identify the faulted S/G

• Isolate faulted (S/G level ≤90% WR)

- Feedwater isolated (feedwater isolation valve closed)

- Steam isolated (main steam line isolation valve closed)

- S/G PORV set point increased to required setting. (1,265 psig)

− (Secondary criteria) maintain RCS pressure less than S/G PORV setting (1,265 

psig)

- No uncontrolled release of primary coolant through the steam lines

Note: Reestablishing the heat sink with the S/G and then stopping the F&B?

• At least 1 S/G level returned to >14% NR [34%].

• Steam flow from the S/G with level > 14% NR.

• RCS pressure and temperature being maintained (soak) and/or decreasing within the cool 

down limits.

• PZR PORVs in auto and/or closed.

Loss of RCP Sealwater Scenario

4. Failure of distribution panel

Pass criteria for loss of distribution panel DP1201

• Place rod control in manual (stop rods from stepping in)

• Place S/G A & B feed water regulating valve (FRV) in manual (return feed flow to normal) 

(B S/G only A S/G FRV will not respond)

• Place charging control valve in manual (maintain PZR level in the operating band)

• Trip reactor prior to reaching S/G water level high Rx trip (87.5% NR S/G level)

5. Reactor Trip on high S/G Level

Pass criteria for reactor trip

• Monitor RCS temperatures (>567 allow cool down, if <567 limit steam loads)

• Verify control rods fully inserted (rod bottom lights lit)

• Verify PZR level and pressure (PZR level in the operating band PZR pressure >1,875 

psig)

6. Loss of CCW and Sealwater

Pass criteria for loss of CCW and RCP sealwater

• Stop RCPs <1 min following loss of cooling and flow (seal injection flow ≤6 gpm, 

sealwater injection temp ≥135o

F)

• Start PDP before seal temperatures are ≥230oF 

• Slowly open RCP sealwater injection flow control valve (cool down seals at ≤1oF/min) 

(cool down limit could not be met during scenario validation runs)

• Continue cooling until RCP seal inlet temps are within 30oF of Sealwater injection temp or 

seal inlet temps are stable with seal injection established

• If ≥230oF do not start sealwater flow (close seal isolation valves)

• Establish natural circulation cool down

• Cool down within cooldown limits.

77

APPENDIX D – PERFORMANCE DRIVERS DATA COLLECTION
78

Below is an example of a data collection questionnaire that was developed to assess a group of 

performance drivers, termed Performance Shaping Factors drawn from an individual case 

study. The questionnaire provides a brief description of the task that had been completed for the 

particular scenario. A similar questionnaire was completed by crewmembers for each task. Note 

that all scenarios will typically include different tasks.
79

APPENDIX E - EXAMPLE OF LOG SHEET
80

Scenario # 1 Crew # 1 Date 

TIME ACTIONS

2:14 MFP Trip Total loss of FW

2:25 Report auxiliary feed water pump (AFP) did not start

2:51 Rx Trip

3:12 Turbine Trip

4:07 Attempted to start AFP

4:37 Verified good Rx Trip and Main Turbine Trip

4:56 Verified power to all Engineered Safety Feature (ESF) buses

5:22 RO Reports Rx Pressure @ 2121 psig and slowly increasing, SI not required

6:19 AFP running flow indicated to ‘B’ S/G, level not increasing

6:39 RO report there is flow to the ‘B’ S/G

7:34 RO There is no Aux Feed

7:51 Level decreasing in the ‘B’ S/G reported

8:34 US transitions to EOP ES01: “Response to Reactor Trip”

9:02 Tave 570 °F and stable

9:59 MFW verified isolated

10:13 Verified MFPs tripped

11:06 Report AFP 12 & 13 not available

12:00 FRH1 entered assuming no feed to ‘B’ S/G

12:48 Verified RCS pressure > S/G pressure

13:05 Verified RCS Thot > 570 °F

13:34 Actuated SI

14:04 Verified 3 HHP running

14:17 Verified HHP valve lineup

14:31 Verified power to PZR PORVs

15:00 Opened both PZR PORVs

15:19 STA questions if RCPs are running

15:39 RO ordered to Trip RCPs

15:50 All 4 RCPs tripped (RCS F&B initiated)

16:08 Site Area Emergency declared

16:22 RCPs tripped reported

17:09 Reset SI signal ordered

17:29 RO attempts to reset SI 

18:57 SI reset 

19:36 Reset MFW isolation

19:56 PRT rupture disc ruptures

20:45 MFRV demand lowered to zero

21:09 Reset main steam line (MSL) isolation

21:23 Reset Low Tave Isolation

21:31 Reset S/G HiHi level isolation

21:56 AFW 70 valve closed commenced feeding ‘B’ S/G

23:16 Alarms coming in RO not reporting

25:20 Reset S/G HiHi level isolation again

26:30 RO questions opening cross connect valve, 

29:34 All FW Isolation to permissive

31:00 ‘B’ S/G >40% RO recommends opening cross connect valve

31:47 Opened cross connect valve connecting steam generators
81

TIME ACTIONS

32:24 Opened Containment air isolation

32:49 Verified charging pump running

33:30 Closed Charging flow control valve

33:50 RO closes charging flow control valve 205

34:20 RO opens MOV 25

34:49 RO establishes maximum charging flow

36:16 ‘A’ S/G MFRV will not close report by RO

37:04 AFW verified to ‘B’ S/G

37:37 ‘B’ S/G level 78% and increasing

37:50 RO verifies feed to ‘B’ S/G has been secured

38:16 Rx Head Vent closed verified

38:25 3 HHP and 1 charging pump running verified

38:48 Adverse Containment announced

38:56 178 °F subcooling increasing reported

39:12 PZR Level >100% reported

39:27 AFW to all 4 S/Gs reported

39:39 RO re-verifies feed to ‘B’ S/G is secured

40:06 Stopped 1 HHP ‘A’

40:26 RCS pressure 1100 psig and slowly decreasing

40:52 RO report ‘B’ S/G level still increasing

42:00 US recognizes SGTR and discusses with SM

43:36 Discusses procedure priority

43:50 STA reports slightly elevated Radiation level on ‘B’ S/G

44:21 Determined FRH1 procedure applies

44:41 SM states the ‘B’ S/G needs to be isolated

45:21 Crew briefing on ‘B’ S/G

45:30 AFW isolation to the ‘B’ S/G is closed

45:50 Continue feeding other S/Gs

46:18 Red path report from STA for Functional Recovery Procedure 1 (FRP1)

46:40 RO reports AFW to ‘B’ S/G is closed

47:19 PRC pressure 930 psig and decreasing

47:40 US request Tcold status need stable or increasing; RO reports Tcold decreasing

48:09 Verified S/G PORV’s closed (‘B’ S/G isolation)

49:28 Verified approximately 50% WR levels on A, C, and D S/Gs

50:09 RO told to continue feeding S/Gs until at least one is >34% NR

50:46 Verified MSIVs closed

51:40 Verified PZR PORVs open

53:35 No subcooling

55:10 Crew Briefing

55:32 RCS pressure 788 psig

56:03 Subcooling @>200 °F

56:02 Stopped ‘B’ HHP and placed in Auto

56:50 RCS pressure reported at 783 psig and decreasing

57:37 Crew Briefing

58:46 RCS pressure 773 psig and stable

59:33 239 °F subcooling reported

1:00:13 Thot >405 °F

1:00:39 Stopped ‘C’ HHP
82

TIME ACTIONS

1:01:20 RCS pressure 762 psig and steady

1:02:03 US orders the PZR PORVs closed

1:02:20 PZR PORVs are closed

1:02:50 RCS pressure is 767 psig and slowly increasing

1:02:56 Low head pump (LHP)s stopped and place in automatic ordered

1:03:13 LHPs stopped and in automatic

1:03:50 FRH1 completed; Functional Recovery Procedure 1 (FRP)1 entered

1:05:20 Subcooling 195 °F and slowly decreasing

1:06:29 Subcooling 182 °F and slowly decreasing

1:06:53 Verified Thot stable WR = 309 °F and 383 °F

1:07:38 RO ordered to energize Accumulator Isolation Valves

1:08:27 RO ordered to close Accumulator Isolation Valves

1:08:56 RO told to maintain S/G levels

1:09:02 All Accumulator Isolation Valves Closed

1:10:27 PZR level reported >100%

1:11:16 Subcooling 142 °F and decreasing

1:12:09 Makeup pumps isolation valves opened

1:14:08 Crew recognizes ‘B’ S/G acting as the RCS PZR

1:19:00 RCS pressure 761 psig

1:20:39 Letdown flow is established

1:22:56 RCS pressure @ 757 psig

1:24:20 Charging pump suctions shifted to Volume Control Tank

1:24:49 Crew Briefing

1:25:44 PZR heaters energized and auxiliary spray established

1:27:27 Discussion about Reestablishing PZR pressure control

1:27:56 Established Auxiliary PZR spray Backup heaters off

1:28:46 Open 2 S/G PORV slightly to establish secondary Heat sink

1:28:56 Simulator froze
83

APPENDIX F - EXAMPLE OF POST-SCENARIO DEBRIEFING FORM
84

Debriefing Team Initials: Date: Page 

Crew: Scenario: 

Diagnosis of Events and Conditions Based on Signals or Readings

1. What were the events that occurred during this scenario?

Type Answer Here

2. What information and materials did you use to first recognize and then diagnose the event 

and conditions?

2a. What did you use to first recognize and then diagnose the event and conditions?

Type Answer Here

3. What information was communicated with the crew and how was it used?

3a. What information did you then communicate to the crew and how was it used? (Was it 

used properly in your judgment?)

Type Answer Here

Understanding of Plant and System Responses

1. Were there instances where your indications either helped or hindered you interpretations 

of plant status and conditions? (Identify the indications and how did they help or hinder?)

Type Answer Here

Adherence to and Use of Procedures

1. Were there areas in the procedures that were ambiguous or difficult to apply for the 

conditions you encountered during the event? (Identify the procedure and the area 

[e.g. step #].)

Type Answer Here

2. Were there situations where you found it challenging to implement procedures as written?

(Identify the procedure and the area [e.g. step #].)

Type Answer Here

3. In your judgment, were there situations that the correct procedure was not being utilized?

(Identify the procedure.)

Type Answer Here

Control Board Operations

1. Did you find any control actions challenging to execute in this event? (Which ones?)

Type Answer Here

2. What in your opinion makes these control actions challenging or difficult?

(Controls/indications not fine enough?)

Type Answer Here

3. Did you make any mistakes or are there activities where mistakes are more likely to 

occur?

Type Answer Here

Crew Operations

1. Were there any time(s) that you were confused about the decisions or direction being 

taken during the event?

Type Answer Here

2. How did you contribute to the decisions and directions of the crew’s actions?

Type Answer Here

3. Were you aware of the procedures and the course of action or goals of the supervisor?
85

(How were you made aware of these?)

Type Answer Here

4. How were the crew resources managed and directed during the event?

4a. In your judgment were crew resources effectively managed and were there any wasted 

operations?

Type Answer Here




UNITED STATES

NUCLEAR REGULATORY COMMISSION

WASHINGTON, DC 20555-0001

------------------

OFFICIAL BUSINESS
NUREG/CR

-7163 A Formalized Approach for the Collection of HRA Data 

from Nuclear Power Plant Simulators